OpenAI backs a shared AI-training preference signal, with implementation still ahead

OpenAI has committed to recognizing a shared robots.txt signal for AI-training preferences, giving publishers a potential way to communicate one choice to several participating providers. The October 30 announcement is a deployment plan, not confirmation that the new mechanism is already operating.

The company says implementation is expected in the coming months and that it will publish further details. Existing crawler-specific controls will continue to apply. For site operators, that means the announcement should not be treated as a reason to remove working rules today. Source: OpenAI announcement.

A proposed common vocabulary, not a finished standard

The linked proposal uses a purpose-specific token, __usage_ai-training, inside an ordinary robots.txt User-Agent group. Existing Allow and Disallow directives express preferences for particular paths. A publisher could therefore distinguish areas of a site rather than make only a blanket choice.

The document is an individual Internet-Draft, version 00, published October 30. Its intended standards-track status does not make it an approved IETF standard. It remains work in progress. The proposal also separates preferences about content use from rules about whether a crawler can retrieve a page. A wildcard user-agent group is not a substitute for the explicit policy token, and the absence of that token means no preference is expressed through this mechanism. Private agreements can take precedence. Source: draft specification; document status.

Training and search already have separate controls

OpenAI’s current documentation distinguishes GPTBot, which gathers material that may support training of generative foundation models, from OAI-SearchBot, which supports discovery in ChatGPT search. A site can allow the search crawler while disallowing GPTBot.

ChatGPT-User is different again: it handles visits triggered by user actions rather than automatic crawling, and robots.txt rules may not apply to those requests. These distinctions matter because a training preference is not a universal switch for every form of AI access. Source: OpenAI crawler documentation.

What publishers should watch

The practical value will depend on adoption and implementation details: which providers honor the signal, when their support begins, and how they resolve conflicting instructions. Our assessment is that a shared vocabulary could reduce maintenance, but a declaration should not be mistaken for a technical access barrier or a guarantee about every downstream use.

For now, the concrete next step is to inventory existing training and search rules, preserve intentional differences between them, and watch for implementation notices. This is not a model release, a new search exclusion mechanism, or evidence that previously collected training data will be removed.

Image: original AI-generated conceptual illustration of a shared training-preference signal. Not an OpenAI interface, official diagram or enforcement guarantee.