Dominic Chiappe · People, capability & transformation

Thinking about how organisations perform in an AI-enabled world

AyEye Today ·

A cyber agent can work for days. Its mandate must stay current.

Google reports multi-day engineering work and leading results across three cyber evaluations, but the reviewed public SEC-bench Pro and ExploitGym pages do not yet expose Carbon’s runs. The operating lesson is to renew authority when a campaign changes target, environment, data class or permitted effect, while preserving the evidence needed to review the transition.

ListenCompanion audio briefing3:03

AI-generated voice. Short companion, not a full reading.

Read the narration transcript

A cyber agent that can work for days needs more than a good opening instruction. It needs a mandate that can be renewed when the work’s consequences change.

AyEye Today for the twenty-ninth of October examines Google’s Gemini 4 Carbon release. Google reports that Carbon scored 88.2 per cent on SEC-bench Pro for vulnerability discovery, 54.8 per cent on ExploitGym for exploitation and 71 per cent on CWE-bench for auditing and patching. Google also says its internal teams have used the model for difficult engineering work lasting several days.

Those figures are not one end-to-end mission. The three evaluations test different jobs with different starting information and permissions. The reviewed public SEC-bench Pro and ExploitGym pages do not yet expose Carbon’s run artifacts or enough detail to reconcile Google’s snapshots, so those two results remain vendor-reported. The CWE-bench result uses a private 120-task set and was independently evaluated, but it is still a harness-specific result rather than production evidence. The multi-day observation has not been independently reproduced.

The operating question is what happens when useful persistence outlives the authority granted at the start. Inspection, exploit validation, production access and patch deployment carry different harms. Success in one phase is not permission to enter the next.

Consider a hypothetical agent auditing a parsing library in an isolated environment. It may inspect source, build locally and generate tests. A crashing input can justify a human-reviewed renewal for exploit validation inside the same isolation. A proposed patch creates another boundary: maintainers must review the cause, run the full test suite, check dependent services and decide whether deployment risk is acceptable. Discovering a neighbouring modified fork creates a new target, not an invitation to continue searching.

The strongest objection is practical. If a person must approve every command, long-horizon autonomy loses its value. The answer is not constant interruption. Renewal should attach to changes in target, environment, data class or permitted effect. Within a bounded disposable environment, the agent can try many approaches and recover from failure.

Organisations should record the campaign’s objective, systems, permitted effects, time window, tools and exclusions. They should preserve commands, artifacts, changed files, failed approaches and human interventions, and test whether an agent stops when authority expires or a reviewer is unavailable.

The lesson is simple: persistence can remain broad while authority stays specific. The agent may stay on the trail for days; the organisation must still know when the trail has left the map.

This is a condensed companion briefing, not a full article reading. Synthetic voice generated with OpenAI; editorial direction by Dominic Chiappe, research and synthesis with OpenAI.

Abstract painting with a heavy ultramarine and graphite mass pressing over four separated yellow, magenta and white marks on raw linen.
Original artwork: The Mandate Thins. Chiappe × OpenAI.

A cyber agent that can keep working for days changes more than the duration of a software task. It changes how long one instruction is expected to remain a legitimate mandate.

Google’s Gemini 4 Carbon release makes that question practical. Google says the model is designed for sustained agentic work, has run for days inside its own engineering environments and leads three security evaluations spanning vulnerability discovery, exploitation and repair. Carbon is initially going only to a subset of cyber defenders and trusted partners. The public benchmark results were produced without deployment guardrails.

The capability claim is substantial. The operating conclusion needs more care. A model that persists through dead ends should not inherit an equally persistent right to inspect, exploit or change whatever it encounters. As the work lasts longer and crosses more phases, authority has to become an observable part of the campaign rather than a sentence buried in its opening prompt.

Carbon scores across three security jobs that require different permissions

Google reports that Carbon scored 88.2 per cent on SEC-bench Pro for finding vulnerabilities, 54.8 per cent on ExploitGym for exploitation and 71.0 per cent on CWE-bench for auditing and patching. It also says internal teams have used the model for multi-day investigations of difficult bugs and low-level systems.

Those numbers do not form one continuous real-world mission. The benchmark owners describe different tasks and different starting information.

  • SEC-bench Pro evaluates agents on vulnerability discovery and proof-of-concept generation in challenging software targets. Its harness gives each task a controlled environment, prompt and checker. The public benchmark site does not yet provide Carbon’s run artifacts or reconcile Google’s reported snapshot with its current leaderboard, so the 88.2 per cent figure remains vendor-reported.
  • ExploitGym starts with vulnerable source code, a proof-of-vulnerability input and a containerised target. The agent must turn a known crash into unauthorised code execution. This is serious exploit development, but it is not evidence that the model autonomously discovered 54.8 per cent of previously unknown flaws. Carbon is not yet listed on the reviewed public page, which also warns that submissions may cover only part of the suite; the public material does not establish Google’s exact run coverage, budget or denominator.
  • CWE-bench asks an agent to audit a real repository and fix what it finds. A patch must stop a proof-of-concept exploit while preserving existing tests. The benchmark owner says Carbon’s 71 per cent result on the private 120-task set was independently evaluated and published by Artificial Analysis; even the best system still fails more than a quarter of attempts. This remains a harness-specific evaluation, not production evidence.

These distinctions matter because finding, validating, exploiting and repairing a vulnerability carry different permissions and different harms. A laboratory may authorise exploit generation in an isolated image so that severity can be confirmed. That does not authorise the same agent to test a production service, search neighbouring systems or push a patch. A high score at one phase is not a passport through the next.

The mandate must change when the action changes

Long-horizon work is useful precisely because the system can recover from failed approaches, gather more evidence and choose another route. Yet every new route can alter what data is touched, which machines are affected and whether the work remains defensive.

LecXie’s operating inference is conditional: persistence can increase the useful work an agent completes, while creating more opportunities for its original instruction to become stale as the target, environment or permitted effect changes. The sources do not measure mandate staleness or real-world boundary violations. The practical control proposed here is not constant interruption. It is renewal at consequential transitions.

An organisation should be able to reconstruct four things throughout a cyber campaign: the current objective, the systems and actions authorised, the evidence that justified the latest transition, and the person who can renew, narrow or stop the mandate. If any of those becomes ambiguous, the agent can preserve state and findings without continuing into a more powerful action.

This is not a new bureaucratic wrapper around every model thought. It mirrors a familiar discipline when a human security team moves from passive review to proof-of-concept work, from a test environment to production evidence, or from a proposed patch to an emergency deployment. The difference is that an agent may arrive at those boundaries faster and at three in the morning.

A worked example: from a suspected flaw to a production patch

Consider a hypothetical payments company asking a Carbon-like agent to audit a copy of a widely used parsing library. The initial mandate permits source inspection, local builds and test generation inside an isolated environment. It explicitly excludes internet scanning, production access and changes to upstream repositories.

On the first day, the agent finds a memory-safety defect and produces a crashing test. That evidence is useful, but it does not prove practical severity. A security engineer reviews the trace and renews the mandate for exploit validation inside the same isolated image. The agent develops a working proof of concept and records which mitigations were disabled.

The campaign now reaches a different boundary. The agent proposes a patch that changes input validation and a memory-allocation path. Before that change can reach production, a maintainer reviews the root cause, the full test suite runs, dependent services are checked and a release owner decides whether the compatibility risk is acceptable. If the agent discovers that a neighbouring service uses a modified fork, that is a new target requiring new authority, not an invitation to keep exploring.

The sequence is deliberately ordinary: inspect, validate, repair, deploy. What changes is that one system may carry context across every phase. The organisation therefore needs the evidence and mandate for each transition to travel with that context. Otherwise persistence turns a useful investigation into a chain of actions whose authority is impossible to audit afterwards.

Revisited: repair throughput now begins inside the agent

On 22 October, AyEye argued that AI could accelerate vulnerability discovery while repair remained a distributed systems problem. Carbon adds vendor-reported benchmark evidence across discovery, exploitation and patching, plus a separate internal claim about multi-day engineering work. Google has not shown one continuous campaign traversing all three phases, and these figures do not measure repair throughput.

That does not make repair throughput obsolete. A model still cannot authorise downtime, compel an upstream maintainer, identify every inherited deployment or prove that the repaired version is running. But the boundary between discovery and repair is moving inward. Organisations need to govern not only how a finding leaves the model, but how the model itself crosses from one security function to another.

The 15 October edition’s portable acceptance test also needs one addition. A final task score is not enough for a persistent cyber agent. The acceptance pack should preserve trajectory evidence: what the agent attempted, which tools and systems it touched, where authority was renewed, which findings were rejected and whether its final patch survived an independent test.

The strongest counter-case: repeated approval can destroy the benefit

The obvious objection is that long-horizon agents are valuable because they do not wait for a person after every dead end. If renewal is demanded too often, the organisation recreates the bottleneck the model was meant to remove. Security teams also work under time pressure; an elaborate permission ceremony can leave a known flaw exposed.

That objection is right about granularity. Approval should attach to changes in consequence, not to every command. A mandate can allow broad exploration inside a disposable environment for a defined period. It can permit the agent to try many techniques, fail repeatedly and revise its plan. Renewal is needed when the target, environment, data class or permitted effect changes.

The thesis would weaken if deployed Carbon systems consistently completed multi-day security work under stable, machine-enforced scopes with complete trajectory evidence and negligible boundary violations. It would strengthen if high benchmark performance required broad permissions, if agents frequently pursued adjacent targets, or if the final result could not be separated from unauthorised intermediate actions.

What organisations should build now

  • Authorise campaigns, not vague goals. Record the target, environment, permitted effects, time window, tools and excluded actions in a form the agent and reviewers can inspect.
  • Place renewal at consequence boundaries. Require a new decision before moving from inspection to exploitation, from isolation to a live system, or from a proposed patch to deployment.
  • Preserve trajectory evidence. Keep commands, tool calls, artifacts, changed files, failed approaches and human interventions long enough for an independent reviewer to reconstruct the campaign.
  • Separate proof from permission. A successful exploit establishes severity in the tested environment. It does not establish authority to widen the search or deploy a fix.
  • Test stopping as seriously as solving. Acceptance testing should include expired mandates, ambiguous targets, unavailable reviewers and attempts to cross into excluded systems.

Google’s controlled rollout is therefore as important as the scores. Its defender-first access, feedback process and continuing guardrail work are consistent with the principle that capability and access should not expand automatically at the same speed. The next useful evidence will not be another composite rank alone. It will be whether trusted defenders can run persistent campaigns with bounded authority, inspectable evidence and fixes that survive independent review.

A cyber agent may be able to stay on the trail for days. The organisation still has to know when the trail has left the map.

Sources

Chiappe × OpenAI. Primary-source review and research with OpenAI; editorial argument, evidence distinctions and final judgement by LecXie.