New robots.txt Token Lets Websites Opt Out of AI Training

Website owners will soon be able to refuse the use of their content for AI training with a single robots.txt group. On October 30, 2026, Google, Microsoft, OpenAI, and Anthropic said they will support a shared token, __usage_ai-training, proposed in a draft at the Internet Engineering Task Force (IETF).

One token for many AI providers

Until now, a site that did not want its pages used to train AI models had to write a separate rule for every crawler. OpenAI has GPTBot, Google has Google-Extended, Anthropic has ClaudeBot, and the list keeps growing. Rules like these fall behind whenever a new provider appears or an existing one renames its crawler.

The new proposal replaces that list with one name shared by all providers. The draft calls it a policy token: a name written on the User-Agent line like any crawler name, except that it states the site owner’s position on a specific use of content. The draft’s own example looks like this.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Those three lines say the site owner does not want any of its content used for AI training, except for the /press-kit/ folder. Path matching, precedence, and group structure all follow the robots.txt format that web teams already know, so there is no new syntax to learn.

The mechanism is described in Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic. The draft is dated October 30, 2026, expires on May 3, 2027, and is being discussed on the mailing list of the IETF AI Preferences working group.

Draft status and scope

The document is still an individual submission. The IETF datatracker notes that individual drafts are not endorsed by the IETF and have no formal standing as standards. The text can still change, be replaced, or be withdrawn.

The scope is also narrow. The draft borrows the working group’s vocabulary definition of AI training: using content to modify the learned parameters of a generative AI model. Training models used only for classification, ranking, or scoring is excluded, even if those models could generate text or images.

Several technical details matter for web teams:

• Wildcard groups (User-Agent: *) do not count as a position on AI training. A site with only wildcard rules has expressed no preference.
• The token does not govern crawling. Crawlers still follow the rules written for their own names, such as GPTBot or ClaudeBot. A provider may use the token to decide not to fetch a page, but the draft does not require it.
• The preference that applies to a page is whatever robots.txt said at the moment the page was retrieved.
• Private agreements between a site and an AI provider can override what robots.txt says.

The third point explains a limit highlighted by outlets including The Next Web: changing robots.txt today does not pull back content that was already collected or used to train earlier models.

Timelines at Google, Microsoft, OpenAI, and Anthropic

All four companies announced support on the same day, but their rollout dates differ.

Anthropic. In a message to the IETF mailing list, Kevin Kelley wrote that Anthropic has already started implementing support for the draft and that it will take time to add it across Anthropic’s services. No date has been announced. The support sits alongside Anthropic’s existing crawler controls and does not replace them.

OpenAI. According to PPC Land, OpenAI aims to implement the signal within the coming months and will publish technical details later. OpenAI also said the new signal does not replace its existing robots.txt controls, including GPTBot.

Google. On the Google Search Central blog, John Mueller wrote that Google aims to support the control in 2027, while Google’s crawler documentation says early 2027. If a site uses the shared token to opt out of training and does not write a specific Google-Extended rule, Google will treat it as a Google-Extended disallow. Google also described the proposal as consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act.

Microsoft. On the Bing Blog, Microsoft said it will start honoring AI Training and AI Grounding preferences on February 1, 2027. The NoArchive and NoCache tags it has used as signals will stop counting for training or grounding from March 1, 2027.

Training and grounding are handled separately

Microsoft draws a line between two uses of content. Training means content is used to shape a model. Grounding means a model opens a page to answer a user’s question, for example when Copilot summarizes an article and links to its source.

The robots.txt token covers training only. Grounding preferences on Bing are set per page through a meta tag or HTTP header, with three options: Allow, DisallowExtended, or Disallow. DisallowExtended limits grounding to experiences Microsoft operates itself, so grounding services Microsoft sells to other companies cannot use that page.

On a page, the markup looks like <meta name="Bingbot" content="ai-training:disallow"> for training and ai-grounding:disallow for grounding. An X-Robots-Tag header with the same values also works.

Microsoft is offering a transition period. From December 15, 2026, sites that use NoArchive or NoCache can record a temporary grounding preference in Bing Webmaster Tools. Those temporary preferences expire on September 30, 2027.

Krishna Madhavan, Principal Product Manager at Microsoft Bing, wrote, “Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.” Microsoft said the preferences do not change a page’s eligibility for regular Bing search results. It also cautioned that blocking grounding reduces brand visibility in AI answers, and that neither choice guarantees traffic or payment.

What this means for site owners

For online newsrooms, universities with journal repositories, government agencies that publish public documents, and online stores with thousands of product descriptions, robots.txt now needs to be treated as a policy document. Two decisions sit behind it: whether content may be used to train models, and whether it may appear as a cited source in AI answers.

Steps teams can prepare now:

1. Decide by section of the site. News and opinion pieces may need to be excluded from training, while press pages, organization profiles, or product documentation may benefit from being known to AI models. The token supports per-path rules, so one decision does not have to cover the whole site.
2. Write the group explicitly. Add a User-Agent: __usage_ai-training group with clear Disallow or Allow paths. Existing wildcard rules will not be read as a position on training.
3. Keep the existing rules. Until Google, Microsoft, and OpenAI implement the token, rules for GPTBot, Google-Extended, and ClaudeBot remain the only signals those providers already read.
4. Watch the Google-Extended effect. Opting out through the shared token without a Google-Extended rule will be treated by Google as a Google-Extended disallow. Sites that still want to appear in Gemini app answers need a separate Google-Extended rule.
5. Prepare page-level tags for Bing. Sites that want to control grounding in Copilot and Bing need the meta tag or header in their page templates before March 1, 2027.

For sites built on a content management system, robots.txt changes should be tested on staging first. A single Disallow line in the wrong group can remove an entire site from search results. Teams that run many sites, such as university portals or networks of local government sites, can build these settings into shared templates through web development services instead of configuring each site by hand. For universities and schools, the decision can also become part of the digital repository policy covered in our education technology solutions.

Managing search visibility is not new for online community owners. A similar question came up when public Discord messages began appearing in web search, and server owners had to decide what could be indexed.

Frequently asked questions

Will this token remove my site from Google Search or Bing?

No. The __usage_ai-training token only states a position on AI training. Microsoft said these preferences do not change eligibility for regular Bing search results. Google’s documentation also says Google-Extended does not affect inclusion in Google Search.

Can content already used for training be pulled back?

Not through this mechanism. The IETF draft says the preference that applies is whatever robots.txt said when the page was retrieved. Content collected earlier is not automatically removed from training data or from models that already exist.

When does the token take effect?

It depends on the provider. Anthropic has started implementation without a fixed date, OpenAI is aiming for the coming months, Google for early 2027, and Microsoft from February 1, 2027. Other AI providers have not made commitments.

Closing thoughts

The __usage_ai-training token gives site owners one place to state their position on AI training, even though it is still a draft and rollout will stretch into 2027. The coming months leave enough time to map content, agree on a policy, and test a new robots.txt. If your team wants to review robots.txt, meta tags, and site templates before the Microsoft and Google dates arrive, Merakit can help you put them together.

Sources

• IETF Datatracker: draft-kelley-aipref-training-preference-signal-00, Machine-Readable Training Preference Signal (October 30, 2026)
• IETF ai-control: draft announcement by Kevin Kelley (October 30, 2026)
• Google Search Central Blog: supporting a shared AI training control (October 30, 2026)
• Bing Blog: Your content, your choice: clearer preferences for training and grounding (October 30, 2026)
• OpenAI: Making it easier for websites to express AI training preferences (October 30, 2026)
• PPC Land: Google, Microsoft, OpenAI and Anthropic back one AI training opt-out (October 31, 2026)
• The Next Web: Google, Microsoft, OpenAI and Anthropic back one AI training opt-out (October 31, 2026)