By Paul Derrington, Jr.
CEO | Chief Engineer | Research Engineer | Editor, D-CSIL
4 min read
Key Takeaways
OpenAI, Google, Microsoft, and Anthropic announced on October 30 that they will honor one shared robots.txt token, __usage_ai-training, so a website can opt out of AI training across all four companies at once. It is voluntary and still a draft standard, and the rollout is staggered: Anthropic says it has already started implementing support, OpenAI says the coming months, Microsoft says February 1, 2027, and Google says 2027.
What did the four labs agree to?
On October 30, OpenAI, Google, Microsoft, and Anthropic announced they will honor a single shared signal in robots.txt, the file where websites state how their content may be used for generative AI training. Until now, a publisher that wanted out had to write a separate rule for each company's crawlers — GPTBot, Google-Extended, ClaudeBot, and a growing list of others. One token now covers all four.
OpenAI calls it a voluntary effort that “makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once.” The signal comes out of the Internet Engineering Task Force's AI Preferences working group, and the technical write-up is an individual Internet-Draft by Kevin Kelley of Anthropic, dated October 30, 2026. Meta expressed support for the idea but has not committed to implementing it, according to SiliconANGLE.
How does the token work?
A site that wants out adds two lines to its robots.txt file: `User-Agent: __usage_ai-training` followed by `Disallow: /`. That states a preference against generative AI training on the whole site. The draft's own example shows the finer control available: `Disallow: /` for everything except `Allow: /press-kit/`, which opts a single folder back in.
Three details matter. The token must match exactly, apart from letter case. A `User-Agent: *` group does not count as naming the token. And the token is a preference about use, not a crawl rule — the draft states plainly that “this policy token is not a directive for crawling behavior,” so existing crawler rules for GPTBot and the rest still apply as before.
What doesn't it do?
It covers training only — using content to change the learned parameters of a generative model. It does not block search crawling, and it does not keep a page out of AI-written answers that quote it. It does not cover models used only for non-generative tasks, and Microsoft describes its preferences as applying to future model training.
Grounding is a separate question. Microsoft is adding its own page-level AI Grounding preference, with a `DisallowExtended` option that lets content inform Microsoft's own AI experiences — AI Mode, AI Overview, Copilot, and Microsoft 365 — but not the grounding services Microsoft sells to third parties. Google says that if a site uses the shared token to opt out without naming Google-Extended separately, it plans to treat that as if the site had disallowed Google-Extended. Google also says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.
When does each company start honoring it?
Kevin Kelley, the draft's author, wrote on the IETF working group's mailing list that “Anthropic has already started implementing support for this draft,” without giving a date for when it will be live, according to PPC Land. OpenAI says it aims to implement the signal “in the coming months” and will share details when it does. Microsoft starts honoring the new preferences on February 1, 2027, as it crawls.
Google says it aims to support the control “along with other AI providers in 2027.” The draft itself is not a finished standard: it is an individual Internet-Draft with no formal standing in the IETF process, and it expires on May 3, 2027 unless it is revised or adopted.
What do the companies tell site owners?
OpenAI says the new signal will not replace its existing robots.txt crawler controls, which will continue to apply, and Google says company-specific controls such as Google-Extended will continue to function alongside the shared signal.
Microsoft has set dates. Its new preferences replace its NoArchive and NoCache tags, in use since 2023. Microsoft urges sites that use those tags to publish the new preferences before March 1, 2027; after that date, it says, NoArchive and NoCache will no longer be read as a training or grounding preference. For sites using those tags, temporary grounding preferences open in Bing Webmaster Tools on December 15, 2026, and expire September 30, 2027, or sooner when a page is recrawled. As Microsoft puts it, “neither choice guarantees inclusion, traffic, or payment.”