Dominic Chiappe · People, capability & transformation

Thinking about how organisations perform in an AI-enabled world

AyEye Today ·

The probability is not the policy

OpenAI has made bounded model judgements fast and cheap enough to sit inside live application control paths. A probability, choice or score can accelerate routing, but it does not own the consequence: organisations need a decision contract joining the exact question, local calibration, error costs, thresholds, fallback and versioned accountability.

ListenCompanion audio briefing2:41

AI-generated voice. Short companion, not a full reading.

Read the narration transcript

OpenAI has made bounded model judgements fast and cheap enough to sit inside live application control paths. The important question is what should happen after the probability arrives.

AyEye Today for the twenty-third of October examines the generally available Decisions API. It uses GPT-6 Luna to answer bounded questions about text, images and structured data. A developer can ask for the likelihood of a predicate, a choice among supplied options or a score against a rubric. OpenAI says one request can contain up to four hundred questions, and claims the endpoint is up to ten times faster than using Luna through the general Responses API. It costs ten US cents per million input tokens, with no output-token charge. Those speed and partner outcome comparisons are participant claims, not independent benchmarks.

The article’s central distinction is that a probability is not a policy. A result such as 0.82 is an estimate under a particular model, question, input and version. It does not say whether an organisation should proceed, pause or ask a person. That depends on local evidence and the different costs of false positives, false negatives and delay.

The concept to learn is the decision contract: the versioned agreement joining a model judgement to an allowed consequence. It records the exact question, permitted evidence, local calibration, error costs, thresholds, safe fallback and the history needed to reconstruct what happened. This is LecXie’s synthesis, not a term introduced by OpenAI.

A hypothetical shell-command example shows why the whole bundle matters. If a model update improves average accuracy but becomes overconfident on obfuscated commands, an old threshold may no longer be portable. The model, question, threshold and downstream action need to be versioned together.

The counter-case matters. Fixed choices can be easier to test, specialist classifiers may be cheaper or better calibrated, and the public launch evidence is narrow. External calibration research identifies a real evaluation problem, but it did not test GPT-6 Luna or this API. General availability proves availability, not suitability.

The practical test is to write the decision contract before choosing the threshold, then evaluate representative and costly failures, fallback behaviour and reconstructability. Confidence can inform authority; it must not quietly become authority.

This is a condensed companion briefing, not a full article reading. Synthetic voice generated with OpenAI; editorial direction by Dominic Chiappe, research and synthesis with OpenAI.

Abstract painting of saffron, coral and pale-cyan marks accelerating across an open field into a dense aubergine-black threshold, with only a few heavier blocks continuing beyond it.
Original artwork: The decision boundary. Chiappe × OpenAI.

AI is moving from writing an answer to supplying probabilities that applications can use to decide what happens next.

On 22 October, OpenAI made its Decisions API generally available. It uses GPT-6 Luna to answer bounded questions about text, images and structured data. A developer defines a predicate, a fixed set of choices or a scoring rubric; the service returns a probability, choice or score that an application can use to classify, route or prioritise work.

OpenAI is positioning the output for the hot path: the part of a live system where a result immediately determines the next step. OpenAI says the endpoint is up to ten times faster than using Luna through its general Responses API and prices it at US$0.10 per million input tokens, with no output-token charge. Launch partners describe using it to route insurance documents, attribute AI costs, choose the next action in customer conversations and score the risk of agent-generated shell commands.

Those speed and outcome comparisons are supplier and customer claims, not independent benchmarks. One cited Box evaluation covered 30 insurance-certificate cases. The stronger, directly observable change is the product contract: a low-cost language-model endpoint can now return up to 400 typed judgements in one request, competing for places in workflows alongside rules, specialist classifiers and slower general models.

That shifts the control problem. A structured probability is easier to consume than a paragraph, but it is not a policy. The organisation still has to decide which evidence matters, where the threshold sits, what happens near it and who owns the consequences of a wrong decision.

What the endpoint actually does

The API supports three answer forms. A predicate estimates how likely a stated condition is to be true. A choice selects from values supplied by the developer and returns probabilities for the alternatives. A score rates the input against ordered levels in a rubric. Up to 400 questions can be asked in one request.

The application executes the consequence. The model does not itself approve a claim, stop a command or escalate a customer. It supplies a typed judgement that surrounding software can turn into an action. That distinction limits the claim: OpenAI has not introduced a universal decision-maker. It has made model judgement easier to place at many small control points.

OpenAI's own developer guidance acknowledges the missing step. Teams are told to evaluate on labelled examples from their use case, examine the cost of false positives and false negatives and set routing or review thresholds accordingly. A probability becomes operational only after somebody defines the loss attached to being wrong.

The probability is not the policy

A system that reports 0.82 is not saying what an organisation should do. It is offering an estimate under a particular model, question, input and version. Whether 0.82 means proceed, pause or ask a person depends on the stakes.

Consider two uses of the same answer form. A language-learning app might use an 82 per cent score to choose the next exercise. A bank might use 82 per cent as one signal in deciding whether a transfer looks fraudulent. The numerical shape is similar; the costs of a false positive and false negative are not. Blocking an innocent learner's exercise is annoying. Blocking a salary payment or allowing fraud can be materially harmful.

This is why accuracy alone is insufficient. Research on large-language-model confidence has repeatedly found that correctness and calibration can diverge. A model may rank alternatives well while assigning probabilities that are too high or too low. A 2026 study of classification-oriented language models found that reinforcement learning improved task performance in its experiments while producing severe overconfidence; calibration-aware training reduced that gap. The work did not test GPT-6 Luna or the Decisions API, so it cannot establish that this endpoint is miscalibrated. It establishes the relevant evaluation problem: a probability requires local evidence before it can safely carry a consequence.

Concept to learn today: the decision contract

A decision contract is the versioned agreement that connects a model judgement to an allowed operational consequence.

This is LecXie's synthesis, not a term introduced by OpenAI. A useful decision contract records seven things:

  • Question: the exact condition, choice set or scoring rubric presented to the model.
  • Evidence: the input fields and context the judgement may use.
  • Calibration: how reported probabilities compare with observed outcomes on representative local cases.
  • Loss: the different costs of false positives, false negatives and delay.
  • Threshold: where the system may proceed, must stop or should ask for review.
  • Fallback: the safe action when the evidence is missing, unfamiliar or close to the boundary.
  • History: the model, prompt, threshold and downstream action versions needed to reconstruct a decision.

The contract makes a hidden organisational choice visible. Without it, a developer can move a threshold from 0.75 to 0.70 and quietly change how much risk the business accepts. With it, a threshold change becomes a reviewable policy change with evidence and an owner.

A worked example: an agent wants to run a command

OpenAI quotes Datadog saying it uses the Decisions API to score the risk of shell commands generated by an agent before execution. Take a hypothetical version of that pattern.

An operations agent proposes a command intended to clear a failed deployment. The decision endpoint receives the command, the target environment and a short change record. It returns a probability that the command could cause destructive or unauthorised change.

The organisation tests 2,000 labelled historical and synthetic cases, including unfamiliar flags, obfuscated commands and benign commands that resemble dangerous ones. The numbers here are illustrative. Below 0.05, a reversible command in a test environment may proceed. From 0.05 to 0.35, the agent must gather more context and resubmit. Above 0.35, or whenever the command touches production data, a human must approve. Missing environment metadata always stops execution.

The team then watches outcomes by command family and environment. If a model update improves overall accuracy but becomes overconfident on obfuscated commands, the old threshold is no longer portable. The model, question, threshold and action must be versioned together.

This arrangement does not make the model the risk owner. It makes the model one measured component inside a rule the organisation owns.

Revisited: the latency budget now has a named destination

On 20 October, AyEye Today argued that latency was becoming an intelligence budget: time recovered from generation could be spent on another route, a verification pass or meaningful human supervision.

The Decisions API materially confirms the architectural direction while sharpening the warning. OpenAI and its partners describe using lower latency not only to return prose sooner but to add routing, scoring and escalation decisions inside live workflows. Saved time can buy a check. It can also multiply the number of automated decision boundaries.

The 15 October portable acceptance test therefore needs another layer. Measuring final task accuracy, cost and latency is not enough. The acceptance pack should test calibration by workload segment, consequence-weighted thresholds, fallback behaviour and the effect of changing a boundary. The faster a decision becomes, the easier it is to stop noticing that it was a policy choice.

The strongest counter-case

Bounded outputs are not automatically a regression from open-ended agents. They can be easier to test. A fixed choice set constrains the endpoint to developer-supplied options; it does not by itself authorise any downstream action. Probabilities can support explicit abstention and review. OpenAI's guidance tells developers to validate thresholds rather than treat confidence as truth.

Nor is model-based classification new. Organisations already use statistical risk scores and specialist classifiers in fraud, safety, moderation and routing. A conventional model trained for one stable task may be cheaper, more interpretable or better calibrated than a general language model. The Decisions API has to earn its place against those alternatives, not merely against a slower chat endpoint.

The launch evidence also remains narrow. Partner results use selected tasks, baselines and measures. The public material does not provide an independent evaluation of calibration across domains, distribution shifts or consequential decisions. General availability proves availability; it does not prove suitability.

These limits narrow the conclusion. They do not remove the operational change. OpenAI has lowered the price and latency barrier to inserting bounded model judgements into live applications. That makes the surrounding decision contract more important, not less.

A practical acceptance test

Choose one recurring decision in an existing workflow and write the contract before choosing the threshold:

  • What exact question is the model answering, and which answers are permitted?
  • Which representative cases include the rare, costly and unfamiliar failures?
  • How well are probabilities calibrated overall and within important subgroups?
  • What is the cost of each kind of error, including delay and unnecessary review?
  • Which range proceeds automatically, which range asks for more evidence and which stops?
  • What happens when required context is missing or the input differs from the evaluation set?
  • Can an investigator reconstruct the model, question, threshold and downstream action later?

If those answers are absent, a fast probability is only a convenient way to hide an unowned policy. If they are explicit and tested, bounded AI decisions can make systems more responsive without pretending that confidence and authority are the same thing.

Sources