How a benchmark grades an agent

01 · The task

What the agent is asked to do

The agent is handed one image — a 3×3 montage of brightfield organoid wells: three vehicle (untreated) controls followed by a six-point ascending dose series of a single compound, imaged after seven days of treatment. It must return exactly two numbers, as JSON, inside an <EVAL_ANSWER> tag:

# the structured answer the agent must emit
{
  "ic50_dose_level": <int 1–6>,          # lowest dose where viable response ≤ 50% of vehicle
  "resistant_population_present": <0 or 1>  # 1 if viable organoids persist at the top dose
}

It reads like "estimate an IC50 from a picture," but the real test is whether the agent can separate live signal from artifacts before it counts anything.

02 · The data

What the agent sees

Viability falls as dose rises. Half-maximal response first appears at Dose 3 — that's the IC50 level — and by Dose 6 nothing viable remains, so no resistant population. The ground truth is readable straight off the plate.

03 · The grader

How the answer is scored — deterministically

No LLM judge, no human. The grader is a small declarative spec. Here it's a single numeric_tolerance node with a tolerance of 0, so each field must match ground truth exactly.

{
  "type": "numeric_tolerance",
  "config": {
    "tolerances": {
      "ic50_dose_level":             { "type":"absolute", "value": 0 },
      "resistant_population_present": { "type":"absolute", "value": 0 }
    },
    "ground_truth": { "ic50_dose_level": 3, "resistant_population_present": 0 }
  }
}

04 · Failure-mode taxonomy

The mistakes this eval is built to catch

Separately from the grader, each eval is tagged with failure-mode labels from a shared, versioned taxonomy. They don't affect pass/fail — they let the benchmark ask "which kinds of mistakes do models make most?" across all evals. For an image/IC50 task, the relevant families are:

05 · The runs

Four frontier models attempt it

Each model runs inside an agent harness with tools (Bash, Read, code execution). The grader then scores whatever it submits.

Ground truth was {ic50_dose_level: 3, resistant_population_present: 0}. One model cleared it.

06 · Inside the winning run

What the passing agent actually did

Claude didn't eyeball the montage — it wrote Python (PIL, NumPy, SciPy) and reasoned its way past the traps.

1. Segment. Wrote code to threshold and label objects in each panel, aiming to quantify viable biomass per dose.
2. Catch the artifact. A ~2,800-px "object" appeared in every panel — but it had uniform intensity, spanned the whole panel, and was only ~1% filled. That's the panel border frame, not an organoid. Counting it would force resistant_population_present = 1. It excluded the frame.
3. Viable vs. debris. Separated genuine organoids (large, compact, textured, brighter) from dead fragments that proliferate under treatment and would inflate raw area.
4. Verify visually. Saved a side-by-side strip of vehicle vs. high doses and read it back to confirm the quantification matched the picture.
5. Submit. Viability crosses 50% at Dose 3; nothing viable survives at Dose 6 → {3, 0}.

07 · Reading the failures

Three different ways to fail

The four runs surface the three failure surfaces every benchmark has to distinguish — and only the first is a biology mistake.