# The evaluator is inside. Independence still needs a mandate

OpenAI’s four embedded-assessment agreements give outside teams access to unreleased models, training runs and internal controls. The assessors’ own terms show why proximity is not enough: confidence depends on what they can inspect, what they can publish and how clearly they disclose what remained out of scope.

**AyEye Today · 2026-11-17**

An assessor can sit inside the building and still be working from a visitor's badge.

OpenAI announced on 16 November that it has signed agreements with Apollo Research, Goodfire, METR and Transluce for a first phase of embedded safety assessments. Depending on the engagement, those teams may see unreleased models, training checkpoints, source code, evaluation data, monitoring records, technical documents and the people who built the systems.

That is materially deeper than asking an outside group to test a finished model shortly before release. It can expose the training choices, intermediate behaviour and internal controls that produced the final system. The assessors' own accounts, however, make the boundary equally important. Access is scoped. OpenAI can decline or limit some requests for security, technical or intellectual-property reasons. It can request redactions of confidential evidence. Transluce says its pilot is not comparable to employee-level access; Apollo says the pilot does not yet fulfil its full criteria for access and disclosure.

The consequential development is therefore not that frontier-AI safety has acquired an independent seal. It is that the mandate of the independent assessor is becoming a visible evidence object. Readers can begin to ask what the evaluator could inspect, what it could publish and how clearly it can describe the gaps.

## What is actually being assessed?

The four engagements cover different claims rather than one universal safety verdict.

Apollo plans to place six evaluators at OpenAI for four months. Its scope covers whether training incentivises models to subvert controls, whether evaluations can elicit such behaviour, whether subversion appears during internal use and whether relevant reasoning would be detectable. OpenAI says Apollo will initially receive selected public and unreleased models, training checkpoints, relevant training data, intervention records, monitoring systems and flagged behaviour.

Goodfire will use interpretability tools to examine model internals, primarily to help OpenAI researchers monitor and debug training. METR intends to assess misaligned behaviour in OpenAI's largest reinforcement-learning run and, subject to a separate agreement on scope, conduct a broader monitoring assessment. Transluce will spend six weeks examining the quality and coverage of selected post-training alignment evaluations, with a deeper follow-on assessment intended for 2027.

These are serious, bounded questions. They are not an audit of everything OpenAI does, every model it trains or every risk its systems could create. OpenAI itself notes that the first METR phase is narrower than METR's broader alignment-assessment proposal. Goodfire's primary audience is internal. Transluce's public report cannot include results about internal-only models. The correct public noun is pilots, not certification.

## Independence has three different rights

Apollo offers a useful test for the arrangement. An evaluator needs access to relevant evidence, disclosure rights sufficient to report consequential findings and meta-transparency: permission to explain the constraints on both.

Financial relationships are another condition to disclose rather than a shortcut to a verdict. OpenAI says Apollo and Goodfire accept compensation that is not contingent on their findings. METR and Transluce do not accept fees for these assessments, although they may accept the compute needed to do the work. Payment does not by itself establish or defeat independence; readers need to know who funds the work, whether compensation depends on the outcome and which protections preserve an unfavourable finding.

The distinction matters because proximity is not independence. Working on site can provide deeper context while also placing the evaluator inside the developer's security perimeter, on its devices and within an agreed scope. OpenAI says assessors can request additional access, but it retains the ability to consider alternatives when direct access would expose its most sensitive intellectual property. Apollo says OpenAI may withhold in-scope material where it considers that reasonably necessary for technical feasibility, security or intellectual-property protection.

Disclosure is similarly layered. OpenAI says assessors remain independent in publishing their assessments, but the company may request redactions of confidential information and delays for responsible disclosure. Apollo says OpenAI cannot redact its verdicts or seek a redaction merely because a finding is unfavourable, although supporting evidence may still be removed if OpenAI considers it sensitive. Transluce says OpenAI can request redactions and that its agreement prevents publication of results about internal-only models.

That leaves meta-transparency doing unusually heavy work. Apollo can state what it requested, what was withheld or delayed, why OpenAI refused, whether evidence was redacted and whether those constraints changed its conclusions. Transluce likewise says it will report requests, approvals and denials and can say when publication restrictions materially affected its findings. A public report can therefore be informative even when readers cannot inspect every underlying artefact, provided the report makes the missing evidence visible rather than converting silence into reassurance.

## A worked board decision: read the mandate before the verdict

Imagine a board risk committee receives a future embedded-assessment report saying: “We found no evidence that the model attempted subversion during internal use.” The sentence sounds reassuring. It is not yet a decision.

The committee should first identify the claim's denominator. Which models, checkpoints, training runs, internal deployments and dates were in scope? Did the evaluator see the monitoring records for the whole period or a selected sample? Could it search ordinary staff channels and incident records, or only request interviews through a designated contact? Did the developer refuse any relevant access, and could the evaluator say so?

Next comes the test method. Did the team merely review the developer's existing evidence, or could it run its own searches, elicitation experiments and tools? Could it follow an unexpected failure into another training stage? How did it handle behaviours that appeared only in an internal model whose results could not be published?

Then comes disclosure. Is the public verdict supported by inspectable evidence? If evidence was redacted, did the evaluator say whether the removal affected confidence? Did the unredacted report reach an oversight body with authority to require action? OpenAI says its Foundation Board's Safety and Security Committee will receive unredacted reports and may speak privately with assessors. That is a governance route; its value still depends on what the committee does with the finding.

The committee can now record a bounded decision: accept, reject or qualify the assessed claim for the named models and period; list unresolved access gaps; require remediation evidence; and set an expiry trigger for new checkpoints, training methods or internal deployments. “Independently assessed” becomes the start of the decision record, not its conclusion.

## Why testing only at the finish line is becoming weaker

A separate METR note published on 16 November shows why an evaluator may need to follow development rather than arrive at the final benchmark.

METR gave investigator agents 19 alignment evaluations and 36 capability evaluations and asked them to infer what each environment was designed to test. On this dataset, recent models were increasingly able to distinguish the two. The latest Anthropic model in the study ranked every alignment evaluation above every capability evaluation using both of METR's scoring approaches; GPT-6 Astra also separated the categories better than earlier OpenAI models. The agents noticed contrived scenarios, exploitable graders, conflicting goals, safety-related information and environments that looked simulated.

This does not show that a model deliberately hides dangerous behaviour in deployment. The study measures recognition of evaluation settings, not successful deception, and the researchers say some tells may be removable. Its narrower implication is still consequential: recognising an evaluation can give a model information that may affect its behaviour; this study establishes recognisability, not that behaviour actually changed.

Embedding assessors across training, evaluation and internal use creates a chance to compare more natural behaviour with staged tests and to inspect how monitoring evidence was produced. It also creates a larger evidence surface. If the evaluator can see only the agreed slice, the word embedded may suggest more coverage than the mandate provides. The solution is not a grander label. It is a report that names the slice.

## Revisited: the monitor now has an assessor

[AyEye argued on 30 October](https://www.lecxie.com/publications/ayeye-today/2026-10-30.html) that an alert is not a brake. METR and Redwood had found ways around an earlier version of OpenAI's internal monitoring, while OpenAI said most of the weaknesses had been fixed. The unresolved question was whether outsiders could test the remediation rather than merely hear about it.

The new agreements materially advance that question. METR is being invited back for a broader monitoring assessment, although the final scope remains pending. Apollo and Transluce can inspect training and evaluation evidence that ordinary external testing cannot reach. The earlier judgement is strengthened: monitoring claims need adversarial review. The update is institutional, not yet evidential. Agreements and access protocols establish a route for assessment; they do not establish that the controls work.

[The 16 October edition](https://www.lecxie.com/publications/ayeye-today/2026-10-16.html) called for versioned safety cases that connect a public claim to a particular model, control and incident record. Embedded reports should inherit the same discipline. A verdict needs the assessed model and training-run identity, the claim tested, the evidence window, the access limits, the unresolved findings and the change that would make the verdict stale.

## The counter-case: a locked door can protect the audit

It would be easy to treat every denied request or redaction as proof that the exercise is captured. That would be too simple.

Frontier laboratories hold model weights, exploit details, personal data, trade secrets and security information that cannot responsibly be copied into a public appendix. Unrestricted access can expose systems to theft or misuse. Immediate disclosure can reveal a vulnerability before it is fixed. A narrow assessor with a precise claim may produce more reliable work than a nominally universal audit that lacks the time or expertise to examine anything deeply.

The stronger objection is that access constraints can be legitimate and still bias what the evaluator sees. The developer knows its own systems, chooses the initial scope and controls the working environment. Relevant evidence may sit in an informal conversation or channel nobody thought to request. OpenAI's employee-discussion protocol lets assessors seek additional staff, but staff are authorised and guided by the company; publishable statements from those conversations generally require formal attestation unless established independently.

The practical falsifier is therefore visible over time. If the pilots surface previously unknown failures, publish their material limits, trigger verified changes and preserve verdicts that are unfavourable to the developer, the arrangements will have produced independent value despite bounded access. If refused access, redaction or scope repeatedly prevents assessors from testing the load-bearing claims, the word embedded will have outrun the evidence.

## What would make the new institution credible?

Five things would turn these pilots into a durable assessment system:

- Claim-level mandates. Name each safety claim, model, training run, deployment context and evidence period.

- Access ledgers. Publish the categories requested, granted, delayed and refused, with the evaluator's judgement about the effect of each gap.

- Protected verdicts and visible redactions. Preserve the evaluator's conclusions, explain when supporting evidence cannot be shown and give an unredacted report to a body that can act.

- Remediation trails. Record which findings changed training, monitoring, access or release decisions, then let the assessor test the fix.

- Continuity. Reopen the claim when a checkpoint, training method or deployment boundary changes rather than treating one report as permanent assurance.

OpenAI's agreements are a significant widening of outside access to a development process that has remained largely closed. The assessors' candid descriptions of their constraints make the opening more credible, not less. They also stop the public from mistaking it for a completed solution.

The useful question is no longer simply whether an outside organisation was present. It is whether that organisation had a mandate capable of finding an inconvenient fact, saying what it found and explaining what it could not see.

## Sources and method

This edition treats OpenAI's announcement and protocol as the producer's description of the engagements. Apollo and Transluce provide independent accounts of their own contractual scope, access and disclosure limits. METR's evaluation-awareness note supplies separate empirical evidence about why finished-model tests can be recognisable; it does not prove strategic deception or validate any OpenAI safety claim. The signed engagements have not yet produced final assessment reports.

- [OpenAI, “Our first phase of embedded assessments”](https://openai.com/index/our-first-phase-of-embedded-assessments/), 16 November 2026.

- [Apollo Research, “A Pilot Towards Embedded Evaluations”](https://www.apolloresearch.ai/blog/pilot-towards-embedded-evaluations), 16 November 2026.

- [Transluce, “Announcing an Embedded Assessment Pilot with OpenAI”](https://transluce.org/embedded-assessment-pilot-with-openai), 16 November 2026.

- [METR, “Measuring alignment eval tells”](https://metr.org/notes/2026-11-16-measuring-alignment-eval-tells/), 16 November 2026.

- [METR, “A proposal for better informing the public about loss-of-control risks from frontier AI development”](https://metr.org/blog/2026-10-14-better-informing-the-public-about-loss-of-control-risks/), updated 12 November 2026.

Review window: 16 November 2026 06:46 to 17 November 2026 06:45 Europe/London.

Chiappe × OpenAI. Primary-source review and research with OpenAI; editorial argument, evidence distinctions and final judgement by LecXie.

Canonical: https://www.lecxie.com/publications/ayeye-today/2026-11-17.html
