# OpenAI and Google Back One robots.txt Line for Opting Out of AI Training, but It Is a Draft Signal, Not a Crawler Block

Times of AI Desk · 2026-10-31 · Policy

[https://timesof.ai/2026/10/openai-google-microsoft-anthropic-shared-robots-txt-ai-training-opt-out-aipref](https://timesof.ai/2026/10/openai-google-microsoft-anthropic-shared-robots-txt-ai-training-opt-out-aipref)

> OpenAI and Google said on October 30 that they will honor a shared robots.txt policy token, __usage_ai-training, that lets a site state one AI-training preference for every participating provider. The Verge reports that Microsoft and Anthropic have also committed, with Anthropic already supporting it. The token comes from an individual IETF Internet-Draft by Anthropic's Kevin Kelley that has no formal standing yet, and the draft says it is not a directive for crawling. OpenAI says it will implement it in the coming months; Google aims for 2027.

Website owners have had to signal their AI-training preferences to each AI company separately. A single robots.txt line is meant to replace that, but it expresses a preference about training rather than blocking crawlers, and it rests on an Internet-Draft that the IETF has not adopted. **OpenAI** and **Google** said on **October 30** that they will honor a shared robots.txt signal for opting out of generative AI training, and The Verge reported that **Microsoft** and **Anthropic** have committed as well.

## What the signal is

The mechanism is a "policy token" written where a crawler's name would normally go. The Internet-Draft that defines it, published on October 30 by **Kevin Kelley** of Anthropic, names the token `__usage_ai-training`; a group that reads `User-Agent: __usage_ai-training` followed by `Disallow: /` expresses a preference against AI training for the whole site. The draft says the token "is not a directive for crawling behavior," that crawler rules keep working exactly as before, and that a wildcard `User-Agent: *` group must not be read as applying to it. It takes its definition of AI training from the IETF AI Preferences working group's vocabulary draft: using an asset to modify the learned parameters of a generative AI model.

The document is an individual Internet-Draft. Its own header says it "is not endorsed by the IETF and has no formal standing in the IETF standards process," and it expires on May 3, 2027.

## Who has committed, and when

OpenAI said it "will honor a common signal" in robots.txt and aims to implement it "in the coming months," with its existing crawler controls still applying. Google's Search Central blog said that when a site uses the token to opt out and does not name Google-Extended, Google plans to treat it as a Google-Extended disallow, and that it aims to support the control in 2027. Search Engine Roundtable reported that Google's crawler documentation now says support is planned for early 2027.

Microsoft's commitment comes from The Verge, not a Microsoft statement we have read. The Verge reported that Microsoft will offer separate opt-outs for AI training and AI grounding, that sites will still appear in Bing results either way, and that Google and Microsoft plan support next year while Anthropic has already adopted the signal. It also reported that Meta's Andy Stone voiced support without saying whether Meta will honor it.

## The fine print for publishers

Google's post says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. By our reading, a site that opts out through the shared token, without a separate Google-Extended rule, would be treated as having disallowed Google-Extended, which Google ties to that Gemini grounding eligibility; Google has not spelled out that consequence. Google also says the proposal is consistent with the EU's Code of Practice for general-purpose AI model providers, which recognizes robots.txt as a primary way to express machine-readable preferences.

The signal arrives as publishers press for leverage over AI crawlers, including in the fight over [Meta's business terms for crawling publisher sites](https://timesof.ai/2026/10/meta-delays-business-terms-ai-scraping-nov20), and as OpenAI tests other arrangements for site discovery, such as its [pilot with Cloudflare](https://timesof.ai/2026/10/cloudflare-openai-discovery-pilot-early-results).

## Limits

- Microsoft's and Anthropic's commitments are as reported by The Verge; we did not find a Microsoft or Anthropic announcement, and Anthropic's crawler help page does not yet mention the token.
- The Verge describes the companies as restricting crawlers on seeing the token, while the draft itself says the token governs training use, not crawling. We follow the draft.
- The OpenAI and Google announcements we read do not say whether the signal affects content collected before a site adds it.
- Search Engine Roundtable quotes IETF participant Nate Hake saying the work is not finished at the IETF and that significant areas of dispute remain.

## Sources

- [OpenAI: Making it easier for websites to express AI training preferences (October 30, 2026)](https://openai.com/index/helping-websites-express-ai-training-preferences)
- [Google Search Central Blog: Supporting a shared AI training control for websites (October 30, 2026)](https://developers.google.com/search/blog/2026/10/ai-training-control)
- [IETF Datatracker: draft-kelley-aipref-training-preference-signal-00, Machine-Readable Training Preference Signal (October 30, 2026)](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/)
- [The Verge: The biggest names in AI are backing a new way to opt out of scraping (October 30, 2026)](https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out)
- [Search Engine Roundtable: Google Also Adopts New AI Training Control (October 30, 2026)](https://www.seroundtable.com/google-ai-training-control-42430.html)
- [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (April 7, 2026)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)