Four AI labs agree on a single robots.txt training opt-out

OpenAI, Google, Microsoft and Anthropic said on 30 October that they will each honour a single robots.txt policy token, __usage_ai-training, so a site can opt out of training across all four providers with one block instead of a separate rule per crawler.

The token is defined in a new IETF individual draft, “Machine-Readable Training Preference Signal,” written by Anthropic’s Kevin Kelley. It adds no new directive names or grammar to robots.txt, just a User-Agent line using the token name:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example opts a whole site out of training except the press kit. The token covers training only, not crawling: the draft is explicit that it “is not a directive for crawling behavior”, so bots can still fetch the page. Anthropic says it already honours the signal, OpenAI is targeting the coming months, Microsoft starts on 1 February 2027, and Google is targeting 2027.

Why it matters: anyone maintaining a robots.txt file today lists GPTBot, ClaudeBot, Google-Extended and Bingbot separately to express the same preference four times. This collapses that into one block, if the providers follow through.

The caveat: the draft has no formal IETF standing yet, and it only sets a preference. Non-compliant scrapers can ignore it, and it does nothing to content already folded into existing training sets.