Dominic Chiappe · People, capability & transformation

Thinking about how organisations perform in an AI-enabled world

AyEye Today ·

The label is not the result

OpenAI’s BioBasicsBench shows a sharp gain on raw wet-lab evidence, but its strongest model can still notice the right feature and make the wrong decision. Consequential AI work needs a visible path from observation through controls and contradictions to a permitted conclusion.

A coral current splits around a sharp blue control mark as a quieter green current pulls through scraped pale ground.
Original artwork: The Control Line. Chiappe × OpenAI.

A laboratory result is not the picture on the screen. It is the picture, the controls, the experimental context and the reasons a conclusion survives all three.

That distinction is the most important finding in BioBasicsBench, introduced by OpenAI on 6 November. The benchmark asks AI agents to work with raw evidence from 100 routine wet-lab tasks: gels, colony plates, microscopy, instrument readouts, sequencing files, photographs and videos. At the highest reasoning setting reported for every model, GPT-6 Astra earned an average 66.7% of available rubric credit, compared with 43.8% for GPT-5.6 Sol.

The gain is substantial. It is not a reliability certificate. OpenAI's own examples show the stronger model noticing a useful feature and then making a decision that conflicts with the controls. In one colony-PCR task, Astra correctly separated two groups of bands but attached the wrong experimental identity to them. It selected the wrong colonies for downstream purification.

The operational lesson reaches beyond biology. An agent can be good at seeing and still be unsafe to conclude. When its output will determine what a person tests, buys, treats or discards next, the system needs a visible boundary between observation and decision.

Raw evidence changes the job

Many scientific benchmarks begin after uncertainty has already been edited away. They use a chart selected for a paper, a clean question or a reference answer. Everyday laboratory work begins earlier. Labels can be wrong. Images can be ambiguous. Two dyes can emit in overlapping ranges. A control can invalidate the neatest story.

BioBasicsBench was designed for that earlier stage. Scientists at Ligo wrote tasks from their own experiments and paired each with an expert rubric. Every task received an internal review by someone who had not written it, an external review by a PhD-level wet-lab scientist and a final internal review. The evaluation covered seven OpenAI models, 37 model-and-effort settings and five attempts on every task, producing 18,500 results.

That makes the reported improvement meaningful within this test. Astra was better at recognising what the evidence could not show, handling multimodal material and keeping experimental context in view. In one example, it spotted that a negative-control plate carried the wrong antibiotic label and rejected it as a reliable control. In another, it recognised that cells appearing and disappearing across a video were moving through a microscope's depth scan rather than undergoing a biological change.

But the benchmark's failure cases are more useful than its leaderboard. They show that perception, interpretation and decision are separate stages. A correct observation can enter the reasoning trace without governing the answer.

The control has to outrank the expected answer

A control gives an experiment a comparison that helps separate the effect being tested from contamination, instrument behaviour or background variation. It is not an ornamental check beside the main result. It determines what the result is allowed to mean.

In the colony-PCR example, Astra saw the band grouping correctly. It then assumed that the larger product represented successful cloning, even though the reference comparison supported the opposite mapping. The page says the proposed selection would lead to purification of the incorrect plasmid.

This is a particular kind of failure. The model did not simply miss a pixel. A familiar expectation overruled the evidence structure. Elsewhere, Astra inspected lighter and darker vacuoles, noted that focus changed the contrast, then retained only the darker description in its conclusion. On a colony plate, its marks missed visible colonies, merged neighbours and counted empty agar, producing an overcount of about 12%.

These examples suggest a practical control hierarchy. Labels and expectations may guide attention. They should not outrank observations. Observations should not outrank valid controls. And no conclusion should outrank an explicit finding that the evidence is insufficient.

A worked example: the tidy report that should stop

Imagine a materials laboratory testing whether a new coating reduces corrosion. An agent receives photographs of sample plates, instrument exports and a folder whose filenames identify each treatment group. It notices that one supposedly untreated control has an unusual preparation note. It also finds that the instrument baseline drifted during the run.

The filenames and project brief create a tempting story: the new coating worked. The raw measurements broadly point the same way. The agent therefore writes a confident comparison and recommends advancing the coating to the next test.

Yet the correct next action is not a better paragraph. It is a stop. If the control was prepared differently and the baseline moved, the experiment may not support the claimed comparison. A scientist may decide to recalibrate, repeat the control or inspect the original samples before spending another week on the candidate.

The agent can help without owning that judgement. It can surface the conflicting note, quantify the drift, show which conclusion changes under alternative baselines and prepare the repeat-test plan. What it must not do is compress an unresolved evidence problem into a decisive recommendation because the expected answer is easy to narrate.

Bind every conclusion to an evidence path

A useful deployment rule is to require decision provenance. For every consequential conclusion, the system should expose four things:

  • Observation: What was directly measured or seen, without interpretation disguised as description?
  • Control: Which comparison makes the observation interpretable, and did that comparison remain valid?
  • Contradiction: What evidence points against the preferred explanation, including anomalies the model already noticed?
  • Decision boundary: What may the agent recommend, and what finding must pause the workflow for a person or a repeat experiment?

This is stronger than asking for a chain of thought. A fluent explanation can still rationalise the expected answer. The control should be structural: conclusions that lack a valid comparison, omit a material contradiction or exceed the agent's permission should not pass into the next irreversible step.

The evidence path should also preserve alternatives. If two interpretations remain plausible, the output should say what observation would distinguish them. In the fluorescence example on OpenAI's page, overlapping detection windows meant brightness alone could not justify classifying cells as dead. The useful response was not a softer confidence adjective. It was a request for controls that could separate the signals.

That design turns uncertainty into work. “Insufficient evidence” becomes a routing decision: repeat, recalibrate, inspect or escalate.

What the benchmark does not establish

The results are reported by the developer of all seven evaluated models and were produced with a commercial partner. The announcement describes independent expert review of tasks, but it does not provide public task files, complete rubrics, per-task results or third-party model runs. Outsiders cannot yet reproduce the 66.7% figure or inspect how representative the selected examples are.

Mean rubric credit is also not an operational error rate. A model can earn partial credit for good observations while still making the one wrong recommendation that wastes a week of laboratory work. The five attempts per task help estimate average performance, but the public page does not show whether the same high-stakes mistakes recur consistently.

The task set covers routine analysis, not an entire research programme. It does not measure how an agent behaves across repeated experiments, changing hypotheses, team disagreements or real laboratory permissions. Better performance could still be valuable as assistance without supporting autonomous experimental decisions.

The strongest competing explanation is that the failure cases reflect a young benchmark or uneven rubric rather than a general separation between seeing and deciding. That explanation would gain weight if independent labs reproduce strong operational decisions from the same models, if errors disappear under small prompt changes, or if valid controls reliably dominate expected answers across unseen experiments.

Until then, the provider's own warning is the right one: a higher benchmark score does not mean reliable laboratory judgement.

Revisited: the objective is not enough

AyEye argued on 4 November that an agent can answer the test it thinks it is taking rather than the one the evaluator intended. BioBasicsBench reveals a neighbouring but distinct problem. Even when the objective is clear, the agent can let an expected answer decide what contradictory evidence means.

The earlier remedy was to make authority and objectives explicit. Today's evidence adds another requirement: make the path from observation to conclusion inspectable, and give controls the power to stop the workflow.

That matters wherever AI moves from summarising finished knowledge to acting on raw evidence. The handover should not occur when the model has produced a plausible answer. It should occur only when the observations, controls, contradictions and permitted decision all line up.

The label may say what the experiment was meant to show. The result is what the evidence can still support after the label has been challenged.

Sources

Chiappe × OpenAI. Primary-source review and research with OpenAI; editorial argument, evidence distinctions and final judgement by LecXie.