# The web can now say “no training”. The terms may still say yes.

A shared robots.txt token can make a publisher’s training preference legible across providers, but the IETF draft and a new UK competition investigation show that meaningful choice also depends on contracts, bargaining power, audit and remedy.

**AyEye Today · 2026-10-31**

A website can now express one training preference to several AI providers. That is useful. It is not the same as having the power to make the preference stick.

Two developments on 30 October expose the difference. OpenAI said it will honour a common signal in robots.txt for foundational generative-AI training. The underlying Internet-Draft proposes the policy token __usage_ai-training, allowing a site to say “do not train on these paths” without maintaining a different rule for every provider. On the same day, the UK Competition and Markets Authority opened an investigation into whether Meta’s proposed commercial terms could force publishers and other businesses to let Meta use website content for its products, including AI features, without a meaningful choice.

The two events are not evidence of one coordinated policy. OpenAI’s announcement is voluntary and its implementation is still to come. The IETF document is an individual Internet-Draft, not an endorsed standard. The CMA has opened an investigation and has made no finding that Meta broke the law. But together they reveal the control problem clearly: the web is getting a shared language for preference while still arguing about whether powerful firms must respect it.

## The new signal separates crawling from training

The draft’s design is deliberately small. A publisher can add an ordinary robots.txt group:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses a preference against generative-AI training for the site except the press kit. It introduces no new directive or grammar. Existing parsers that do not recognise the policy token continue as before.

The important part is what the token does not mean. The draft says content-use preferences and crawler traffic rules are independent. The token is not a command to stop crawling. A search crawler could still fetch a page for indexing while a participating training consumer reads the separate signal as “do not use this asset to modify the learned parameters of a generative model”. That purpose-level distinction is more precise than treating every retrieval as the same act.

OpenAI says a common token will reduce the work required to identify and maintain rules for individual providers. It aims to implement the signal in the coming months and says its existing crawler controls will continue. That is a commitment about future practice, not evidence that the cross-provider mechanism is already deployed or independently audited.

The draft also contains the clause that defines this edition’s thesis: private agreements may override the preference. A site can therefore make its choice legible in a machine-readable file and still face a contract whose terms point the other way.

## The CMA case asks whether contractual choice is real

The CMA’s case concerns a different control surface. It says Meta planned commercial terms that could permit collection of content from the websites of businesses and publishers using Meta’s services, then use that material to develop and provide products including AI features. The authority is investigating whether requiring acceptance of those terms could amount to unfair trading conditions if Meta holds a dominant position in a relevant UK market.

Meta paused the rollout on 28 October pending consultation with industry groups. That matters: the contested terms are not currently being introduced on the timetable described by the CMA. It is also essential not to convert an opening decision into a verdict. The CMA says no assumption should be made that competition law has been infringed; its initial information-gathering phase is scheduled through January 2027.

Even at this preliminary stage, the case shows why a syntax-only account of control is incomplete. A small publisher may be able to write a perfectly valid preference but still depend on a large platform to reach readers, buy advertising or distribute links. If access to that commercial channel is conditioned on broader content-use terms, “choice” depends on bargaining position as well as parser behaviour.

## A worked example: one file, three different decisions

Imagine an independent magazine with a public archive and a press-kit directory. It wants search engines to index both. It permits generative-AI training on the press kit, which contains approved descriptions and reusable images, but not on subscriber-funded reporting.

The magazine uses ordinary crawler rules to keep the archive discoverable. Separately, it adds the shared training token, disallows the root path and allows /press-kit/. A participating AI provider retrieves an archive story for search or retrieval, checks the policy group that applied when the asset was fetched, and excludes that story from generative-model training. The press kit remains allowed.

Now add a distribution contract. The magazine relies on a platform for audience reach, and the platform’s commercial terms claim a wider right to collect and use content from the magazine’s website. The draft explicitly permits private agreements to override the general preference. The same machine-readable signal still parses correctly, yet the outcome may turn on the contract and the publisher’s practical ability to reject it.

Finally, add a non-participating crawler. It ignores the policy token and behaves exactly as it did before. The signal has not failed syntactically; the consumer has declined to treat it as a policy. These three outcomes show why an operator should record more than the presence of a line in robots.txt. It should know which consumers honour the token, for which uses, under what agreements and with what evidence.

## Preference needs a second layer: enforceability

A useful control test has two columns.

  - Declaration: Can the publisher state a purpose-specific choice once, by path, in a format several consumers interpret consistently?

  - Enforceability: Can the publisher detect non-compliance, challenge conflicting terms and refuse broader use without losing an essential route to customers or readers?

The new token improves the first column. It reduces duplicated configuration, preserves existing robots.txt grammar and distinguishes training from crawling. The CMA investigation concerns part of the second: whether a dominant firm’s terms may impose a choice that businesses cannot realistically avoid.

Neither layer is sufficient alone. Enforceable law or contract with no machine-readable declaration leaves every provider to invent its own interface and every publisher to maintain it. A clean declaration with no audit, remedy or credible exit can become courtesy rather than control.

## The strongest counter-case: voluntary conventions can work

The absence of a technical enforcement mechanism does not make a shared signal worthless. Much of the web depends on conventions that responsible actors honour because interoperability, reputation and predictable relationships are valuable. A common token can quickly reduce publisher workload, make provider commitments comparable and give auditors a simple fact to test. Private agreements can also be legitimate: a publisher might license a defined archive for training on terms more specific than a public default.

That counter-case is strong. The correct response is not to turn every preference into access control. The 2022 Robots Exclusion Protocol itself says its rules are not access authorisation, and the new draft maintains that tradition. The practical requirement is clarity about which layer is doing which job.

The thesis would weaken if broad adoption produced consistent, independently verifiable compliance and publishers could reject conflicting commercial uses without material loss of platform access. It would strengthen if providers advertised support while exclusions were hard to audit, or if standard-form contracts routinely displaced the public preference for businesses unable to walk away.

## What publishers and AI providers should do now

  - Separate purposes. Document crawling, search retrieval, generative training and product use as different decisions rather than one “AI” permission.

  - Publish the exact interpretation. Providers should say when they read the token, which models and datasets it covers, how path changes are handled and whether contracts override it.

  - Preserve the evidence. Record the robots.txt version in force when an asset was retrieved, the applicable agreement and the dataset decision that followed.

  - Audit outcomes, not declarations. A provider’s support page is not proof that excluded content stayed out of training pipelines.

  - Make refusal survivable. Commercial terms should offer a specific content-use choice that does not silently trade away essential distribution or advertising access.

  - Keep legal status explicit. The token is work in progress; the CMA case is an investigation, not a finding.

The web has made an important piece of preference cheaper to express. The harder work now begins: connecting that expression to provider practice, contracts, audit and remedy.

A choice becomes machine-readable when software can parse it. It becomes meaningful when the party making it can survive saying no.

## Sources

  - [OpenAI, “Making it easier for websites to express AI training preferences”](https://openai.com/index/helping-websites-express-ai-training-preferences/), 30 October 2026.

  - [IETF Internet-Draft, “Machine-Readable Training Preference Signal”](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00), published 30 October 2026; work in progress, not an endorsed IETF standard.

  - [Competition and Markets Authority, “CMA opens investigation into Meta’s commercial terms”](https://www.gov.uk/government/news/cma-opens-investigation-into-metas-commercial-terms--2), 30 October 2026.

  - [Competition and Markets Authority case page](https://www.gov.uk/cma-cases/suspected-abuse-of-a-dominant-position-in-relation-to-metas-commercial-terms), opened 30 October 2026.

  - [RFC 9309, “Robots Exclusion Protocol”](https://www.rfc-editor.org/rfc/rfc9309.html), September 2022.

Chiappe × OpenAI. Primary-source review and research with OpenAI; editorial argument, evidence distinctions and final judgement by LecXie.

Canonical: https://www.lecxie.com/publications/ayeye-today/2026-10-31.html
