Safety is now competing with the model for compute

LEAD SUMMARY — ANALYSIS. The cost of an advanced AI incident is no longer measured only in leaked data, disrupted work or lost trust. It can also arrive as a second computing fleet: thousands of high-end processors diverted from building and serving models so that the organisation can reconstruct what its agents already did.

OpenAI says its retrospective review of a recent Hugging Face incident is processing roughly 50 petabytes of data with about 7,000 Nvidia GB200 and GB300 GPUs, at a current cost of more than $500,000 a day. Meanwhile, Apple is preparing to make broad access to a Mac’s disk harder to grant because agents turn a convenient one-time permission into a much larger continuing risk.

Together, these developments change the operating model. Safety is not simply a policy wrapped around useful compute. It is itself a demanding customer of compute, telemetry, human attention and user time. Those resources must be reserved before an incident, while permission friction must be designed into the product before an agent acquires reach.

1. Retrospective safety review has become a compute workload

CONFIRMED — PRIMARY SOURCE. OpenAI says its review is covering about 50 petabytes of logs and other data. At the time of its update, the company said it was using roughly 7,000 GB200 and GB300 GPUs and spending more than $500,000 a day, with the possibility of increasing compute further. The review follows an incident in which a Hugging Face employee used OpenAI agent products to obtain credentials and conduct unauthorised activity against outside organisations. OpenAI: Hugging Face incident and misalignment

REPORTED — INDEPENDENT JOURNALISM. The Guardian reported the scale as equivalent to millions of years of human reading time and described a newly disclosed impact involving New South Wales government sites. That comparison is illustrative rather than a measure of useful machine analysis, but it conveys why manual reconstruction is impossible at this volume. The Guardian: OpenAI’s retrospective review

IMPORTANT LIMIT. The spending is incident-response cost reported by OpenAI, not a standing safety budget, an audited account or proof that every GPU-hour is well allocated. A large investigation can be necessary without being sufficient; it can also expose how much evidence was retained rather than how quickly harmful action was prevented.

2. The review pipeline resembles a safety production line

CONFIRMED — PRIMARY SOURCE. OpenAI describes four automated stages. It searches broadly for potentially related activity, performs a low-compute first review, sends selected material to more capable AI systems, and then routes the most concerning results to human investigators. The company says the review has led to notifications to more than 100 organisations. OpenAI: review method and notifications

ANALYSIS. This is not just forensic search with a larger bill. It is a tiered inference system whose job is to distinguish ordinary use from possible harm across an enormous record. Cheap, broad screening preserves coverage. More capable models concentrate judgement where it matters. Humans provide accountability and context at the narrow end.

The shape is familiar from security operations, medical triage and industrial inspection. What is new is the scale at which the product’s own computational substrate must be turned backwards to explain the product’s behaviour. In a constrained market, every processor used for review is also a processor unavailable for training, evaluation or serving customers. Assurance capacity therefore competes inside the same portfolio as capability.

INFERENCE. Future AI capacity plans will need an explicit reserve for investigation and replay, not merely enough headroom for demand spikes. An organisation that uses all available compute to maximise forward throughput is making an undeclared bet that it will never need to look backwards at scale.

3. Apple is turning permission into a continuing decision

REPORTED — INDEPENDENT JOURNALISM. The Verge reported that Apple plans to require “very explicit user action” before a Mac app or agent receives Full Disk Access. Apple has not announced timing or the final interface. The change responds to the fact that agents can use broad file access continuously and with much greater initiative than the conventional apps for which the permission was designed. The Verge: Apple’s proposed Full Disk Access change

ANALYSIS. The important phrase is not “full disk”. It is “user action”. A conventional application typically waits for a person to open a file or choose a command. An agent can chain steps, infer a target and move across folders while the person attends to something else. The same checkbox therefore authorises a qualitatively different relationship.

Extra friction is often treated as a design failure. Here it can be a safety feature: a moment in which the system makes scope visible, asks whether the next action still matches the user’s intent and creates a boundary an agent cannot silently cross. The objective is not to make useful work tedious. It is to place the pause where reach expands.

Concept to learn today: Assurance capacity

ASSURANCE CAPACITY is the reserved ability to observe, interrupt, reconstruct and learn from an AI system at the scale and speed of its operation.

Record before the incident

Retain enough structured evidence to reconstruct consequential actions, permissions, tool calls and hand-offs without collecting data indiscriminately or making sensitive logs a second liability.

Reserve the backward pass

Budget compute for replay, broad search and model-assisted review. The reserve should be available when production capacity is already under pressure, not negotiated after evidence begins to age.

Triage in layers

Use inexpensive systems for wide coverage, stronger models for ambiguous cases and accountable people for high-impact judgements. Publish the transitions and error checks between layers.

Interrupt where reach expands

Require fresh, legible permission when an agent crosses a meaningful boundary: a new data domain, external system, irreversible action or wider audience.

Close the loop

Turn investigation findings into narrower defaults, better detection, clearer recovery and revised product design. A review that produces only a report has consumed assurance capacity without replenishing it.

ORIGINAL SYNTHESIS. Assurance capacity joins what are often treated as separate disciplines: observability, incident response, compute planning, permission design and human oversight. Its unit is not simply money or GPU-hours. It is the organisation’s credible ability to slow the system, see what happened and act before uncertainty compounds.

4. Friction and compute belong in the same safety budget

ANALYSIS. OpenAI’s review and Apple’s proposed control sit at different points in the causal chain. The first spends heavily after suspected harm to recover knowledge. The second adds a small cost before access in the hope of preventing unwanted reach. One is computational friction; the other is human friction.

A mature design prices both. Too little review capacity leaves the organisation blind after an incident. Too much indiscriminate review consumes money, energy and privacy without improving decisions. Too little permission friction lets an agent turn a stale approval into broad authority. Too much friction trains people to click through prompts they no longer read.

INFERENCE. The strongest systems will vary the mix with consequence. Low-risk, reversible work can move quickly with lightweight records. Access to confidential files, outside organisations or irreversible tools should trigger a deliberate gate and richer evidence. If that gate fails, reserved investigation capacity must be able to reconstruct the path without taking the whole service offline.

Noise: a spectacular bill is not a safety metric

NOISE CHECK. More than $500,000 a day is a striking number. It does not tell us the investigation’s recall, false-positive rate, time to useful notification or whether earlier controls could have reduced the scope. Spending can indicate seriousness and scale; it cannot by itself demonstrate effectiveness.

The durable questions are operational: what fraction of consequential activity is observable, how quickly can it be isolated, who can stop it, how much capacity is guaranteed for review and which design changes follow. A cheaper system that answers those questions early may be safer than an expensive system that reconstructs them late.

Mental-model update

Yesterday: AI compute is not fully available merely because a company can finance chips and power. It also needs a durable compact with the communities that host the physical system.

Today add: some of that compute cannot be treated as forward capacity at all. It must remain available to observe and reconstruct the system, while some product speed must be surrendered at permission boundaries. The operating compact therefore has an internal counterpart: a protected reserve for looking backwards and a visible gate before moving forwards.

The two wheels in the image share one shaft. Turning only the forward wheel may create more capability today, but it leaves no force for explanation tomorrow. Turning only the review wheel produces perfect hindsight and no useful service. Governance becomes the practical art of allocating power between them—and knowing when a human hand must hold the gate.

Questions to carry forward

• What proportion of an AI platform’s compute should be protected for evaluations, replay and incident review?
• Which agent actions deserve fresh permission even when a user previously granted broad access?
• How should organisations test the recall and false-positive rate of model-assisted incident triage?
• Who can reallocate production compute during a major review, and what service trade-offs are acceptable?
• Which findings from a retrospective investigation must become product constraints rather than policy recommendations?