🔍 Read the full analysis: What The OpenAI, Google, Microsoft And Anthropic Opt-Out Covers on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have each said they will support a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The opt-out is voluntary, will take effect on different schedules, and does not block crawling, remove material from existing models, or cover AI answers grounded in live web content.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The move could simplify how publishers communicate that preference, but it is voluntary, limited to training and scheduled to take effect at different times for each company.
The signal appears in an individual Internet-Draft titled “Machine-Readable Training Preference Signal.” The proposed robots.txt group uses User-Agent: __usage_ai-training followed by Disallow: /. A publisher adding it is expressing a preference against training use; the rule is not a technical barrier that prevents a company from accessing a site.
The companies have described different implementation schedules. Anthropic says it has started implementing support, though it says support will take time to reach its services. OpenAI aims to implement the signal “in the coming months.” Google’s crawler documentation says “early 2027,” while Microsoft says its shared training preference will take effect as it crawls from February 1, 2027.
Microsoft also announced a separate page-level preference for AI Grounding—the use of web content to inform AI answers in Bing, Copilot and Microsoft 365. Microsoft says both new preferences begin February 1, 2027. Its existing NoArchive and NoCache controls will stop counting as training or grounding preferences after March 1, 2027. The companies say existing crawler-specific controls remain applicable during the changeover.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A Simpler Signal, Not a Block
A shared rule could reduce the work publishers face when expressing a training preference to several AI companies. Until now, site operators have had to manage separate crawler instructions, including controls for GPTBot, Google-Extended and ClaudeBot, alongside Microsoft’s page tags. A common signal gives participating companies one instruction to recognize, although publishers may need to keep their existing rules while implementations are rolled out.
The distinction between a preference and enforcement matters. The proposed line does not stop a crawler from fetching a page, and it does not itself establish a payment arrangement or guarantee that a publisher will receive traffic. Microsoft says that neither choice guarantees inclusion, traffic or payment. The signal therefore makes a publisher’s stated preference easier to express, but does not settle broader disputes about licensing, compensation or compliance.
It also separates training from other uses of web content. A site may want to limit model training while permitting search indexing or the use of pages to generate current answers. Microsoft’s separate grounding control reflects that distinction. Publishers will need to decide independently which uses they accept rather than treating one opt-out as a universal restriction on AI systems.
Popular right now
Ninja CREAMi 7-in-1 Ice Cream Maker, NC299AMZ
Dyson Car+Boat Cordless Handheld Vacuum
TCL QM64L 85-Inch QD-Mini LED 4K Smart Fire TV
As an affiliate, we earn on qualifying purchases.
How the Proposal Took Shape
The four companies refer to the same Internet-Draft, authored by Anthropic standards lead Kevin Kelley and published October 30. It is connected to the Internet Engineering Task Force’s AI Preferences working group, known as AIPREF. An Internet-Draft is a proposal for discussion, not an adopted internet standard. The document itself says it has no formal standing in the IETF standards process.
That status is relevant to how much weight to place on the shared approach. At an AIPREF working-group meeting on October 21, one of the group’s chairs said, “We haven’t declared consensus on anything yet.” Company announcements indicate intended support, but do not amount to a final IETF decision or a binding rule for every AI provider.
Google says that if a site uses the shared signal and has no Google-Extended rule, it plans to treat the new signal as if the publisher had disallowed Google-Extended. Google says its support target is 2027. OpenAI describes its effort as voluntary and says its current crawler controls will continue to apply. Anthropic’s crawler help page, as of October 31, still described its existing per-crawler controls.
““has already started implementing support for this draft””
— Anthropic standards lead Kevin Kelley
Adoption and Enforcement Remain Open
It is not yet clear whether Meta will honor the signal. Meta communications chief Andy Stone said the company supports publishers’ ability to state preferences and called a common signal an important step, but did not commit to implementing it. The source material reports no statement from Apple, Amazon, xAI or Perplexity, so their positions should not be assumed.
The proposal is also not an adopted standard, and the announcements do not establish a shared enforcement mechanism. The signal communicates a publisher’s preference; the available information does not show how disputes over noncompliance would be handled. Implementation timing may vary by company and service. Anthropic says rollout across its services will take time, while Google and OpenAI have stated targets rather than a single shared launch date.
The boundaries of the signal are also important: it does not remove content from models already trained, stop crawling, govern a user’s conversations with an AI assistant, or by itself restrict AI answers based on live web content. Those uses require separate controls or policies. Microsoft has announced its own grounding preference, but the shared training line should not be read as covering that use across all providers.
Watch Rollouts and IETF Review
Publishers can add the proposed training-preference group if they want to state that preference, while retaining company-specific crawler rules until each provider’s support is in place. Sites using Microsoft’s NoArchive or NoCache tags should review Microsoft’s announced schedule and plan for the March 1, 2027 change, particularly if they also want to express a separate preference about AI grounding.
The next developments to watch are whether Anthropic extends support across its services, whether OpenAI and Google meet their stated targets, and whether Meta or other providers make explicit commitments. The AIPREF working group’s discussions will also determine whether the proposal advances through the IETF process. Until then, the signal is best understood as a shared, voluntary instruction whose practical effect depends on each company’s implementation.
Key Questions
What does the shared AI training opt-out do?
It lets a website state a preference against having its content used to train generative AI models, using a proposed shared robots.txt rule that participating companies say they will honor.
Does the signal stop AI companies from crawling a website?
No. The signal is a request or preference, not a technical block. It does not prevent crawling, and its stated scope is training rather than every use of website content.
Does it stop AI answers from using a site’s pages?
Not by itself. The shared signal covers training, not AI answers grounded in live web content. Microsoft has announced a separate AI Grounding preference for its services.
When will the four companies support it?
Anthropic says implementation has started; OpenAI aims for the coming months; Google targets early 2027; and Microsoft says its preference takes effect as it crawls from February 1, 2027. The schedules differ.
Is this an official internet standard, and has Meta committed?
No. The signal is in an individual Internet-Draft and has not been adopted as an IETF standard. Meta has voiced support for publishers expressing preferences but has not said it will honor this signal.
Source: ThorstenMeyerAI.com
