OpenAI Commits to Honoring Shared robots.txt Signal, Letting Websites Express Training Preferences to Multiple AI Providers Uniformly

OpenAI announced on October 30, 2026, that it will honor the shared signal in robots.txt, helping website operators communicate AI training preferences to participating AI providers. This mechanism originates in the IETF AIPREF working group, with the core being the introduction of the __usage_ai-training policy token.

The Facts

On October 30, 2026, OpenAI formally committed to supporting this shared signal. The signal is defined in an individual Internet-Draft titled Machine-Readable Training Preference Signal, authored by Kevin Kelley of Anthropic, dated October 30, 2026, and valid until May 3, 2027. The draft specifies that websites can express preferences regarding AI training by using User-Agent: __usage_ai-training in robots.txt together with Disallow or Allow directives—for example, prohibiting training on all paths while allowing the /press-kit/ path. Existing robots.txt parsers ignore unrecognized user-agent groups and therefore will not affect existing traffic-management rules.

Google also announced around the same time that it plans to adopt the same proposal, intends to support the __usage_ai-training signal in early 2027, and will update its common crawler page. Google stated that if a website uses the token to opt out of training and does not separately specify Google-Extended, Google will treat Google-Extended as having been disallowed.

How the Mechanism Works

The signal works based on the standard robots.txt structure defined in RFC 9309 and introduces no new directive names or syntax. Website operators need only add a group in robots.txt with __usage_ai-training as the user agent to set training preferences across providers at once. Previously, operators had to maintain separate rules for each AI crawler; now they need only maintain one shared rule. OpenAI explicitly stated that this signal will not replace its existing robots.txt controls, which will continue to be honored. The signal is voluntary, specifically targets foundation model training, and does not cover all possible uses of content.

The draft emphasizes that the wildcard group User-Agent: * must not be interpreted as matching this policy token. If there is no matching group, no preference is expressed.

Industry Impact

For content creators, this mechanism reduces the burden of managing rules across multiple AI providers; a single setup can affect participating providers. However, its practical effect depends on whether other major AI companies adopt it as well; currently, only OpenAI and Google have made commitments.

For AI providers, participation can reduce compliance communication costs while demonstrating respect for creators' control. Non-participants still face the status quo of maintaining rules one by one and may be at a disadvantage in terms of data access convenience.

For developers and website operators, existing robots.txt parsing logic requires no modification, so the implementation barrier is low. However, the signal covers only training uses; other uses, such as search indexing, still require separate controls.

Strategic Assessment

Based on currently public commitments, the most likely next step is for OpenAI to publish specific implementation details in the coming months, and other IETF working group participants may successively state whether they support it. Key things to watch include the actual implementation by Google in 2027 and whether more providers publicly join. This judgment is based on published announcements and is analysis rather than fact.