If you want to control AI training via robots.txt, you will soon be able to set a shared preference for Google, Microsoft and OpenAI. The announced rule simplifies management, but it does not automatically protect your content from every kind of AI use. What matters is whether the providers actually support it and how differently each of them interprets it.
Key takeaways
- Google, Microsoft and OpenAI have announced that they will honour a shared robots.txt rule for AI training preferences.
- As of the reporting date described, none of the three providers supports the shared rule yet.
- Google and Microsoft differ in which uses a shared opt-out will cover in future.
- Companies should document their existing rules and assess changes to AI access and to SEO measurement separately.
How can you control AI training via robots.txt?
The planned control works through an ordinary robots.txt group with the policy token __usage_ai-training and the Allow and Disallow rules. This is meant to let websites set one shared preference, instead of maintaining only separate training rules for each company.
Google, Microsoft and OpenAI announced their support on 30 October. The basis is an Internet-Draft by Kevin Kelley at Anthropic. For practical implementation, though, the announcement alone is not what counts: according to the publicly stated timelines, none of the three providers supports this control as of the reporting date described.
A shared group can only replace several company-specific training rules once the individual providers actually evaluate it. You should therefore not remove existing settings too hastily. First, document which rules your website already contains and which use you want to restrict with them.
The shared syntax also does not mean a uniform effect. Whether a preference covers only training or other uses as well depends on each provider’s own rules.
Why does the same rule work differently at Google and Microsoft?
Google ties the shared preference to Google-Extended, while Microsoft plans to handle training and grounding through separate settings.
Google adopts the preference under one condition
Google plans to treat a shared training opt-out like an opt-out via Google-Extended, provided the website has not defined its own rule for Google-Extended. An existing Google-Extended setting is therefore a key item to check.
According to Google’s crawler documentation, Google-Extended also covers grounding in Gemini Apps. Grounding means using content to support the answers the AI gives. The documented robots.txt controls do not separate Gemini training from grounding in Gemini Apps.
For your decision, this means that you cannot regard the planned shared rule at Google purely as a statement about training. At the same time, check whether the associated treatment of grounding matches the usage policy you want.
Microsoft announces a separate grounding preference
Microsoft intends to use the shared rule for training purposes and to introduce an additional, separate AI grounding preference at page level. Training and grounding are therefore meant to become separately controllable there.
At the same time, Microsoft plans to stop treating NoArchive and NoCache as preferences for training or grounding after 1 March 2027. If you currently use these directives for that purpose, you should plan the switch to the new preferences before that date.
For OpenAI, the summary available to us does not give a comparably detailed breakdown of further usage purposes. The joint announcement therefore does not allow you to conclude that all three providers will be treated identically.
Which decisions should you make before changing anything?
First decide which uses of your content you want to allow, and only then choose the matching technical rule.
Wanting to keep content out of training does not yet answer every question about AI search. The announced differences in grounding show why a blanket instruction such as “exclude AI” remains too vague for technical implementation.
- List the existing robots.txt rules for each individual provider.
- Check whether Google-Extended is already defined explicitly.
- Find out whether NoArchive or NoCache are currently meant to express Microsoft preferences.
- Record separately how you want training and grounding to be treated.
- Before switching, check the published support status of each provider.
If an agency looks after your website, do not simply instruct it to add an entry. Ask for documentation of which existing rules are affected and what effect the change is meant to have according to each provider’s own statements.
AI crawling belongs in your technical SEO review
You can add the review of AI crawlers to your existing technical SEO audits, but you need to look at access and rules provider by provider.
Google’s John Mueller said in conversation with David Iwanow that AI systems build on SEO. The interview from Search Central Deep Dive Europe 2026 was published on 29 October. Mueller expects technical SEO in particular to remain relevant.
He described different purposes of crawling, such as training or fetching current content. For your review, this means that a general statement about whether your website is reachable is not enough. For each crawler, check separately which access is intended and which robots.txt rules apply.
On content formats, too, Mueller advised a solid foundation. Asked about listicles, he described how successful tactics are often imitated badly until algorithms adapt. Short-term format decisions therefore do not replace clear technical control.
Why do you need to check your SEO measurement data at the same time?
The reported changes show that metrics can fluctuate because of data collection problems or new calculation methods, without proving any corresponding change to your website.
Ranking losses can be linked to how results are captured
SE Ranking examined 10,000 Google US desktop keywords. In the analysis published on 3 November, 84 per cent of the conspicuous cases involved unrelated results for a single search term. Of these result blocks, 96.5 per cent started at position 9 or lower.
In addition, data providers report limitations in resolving Google goto links. As a result, a ranking report may show a redirect address instead of the actual target page. If you see striking losses, check the stored URLs and compare impressions and clicks for the same search query in Search Console.
PageSpeed lab values use a changed simulation
PageSpeed Insights now uses Lighthouse 13.6.0. This version changes how page loading is simulated and can therefore produce different scores even though your page has not changed. According to Google’s Barry Pollard, JavaScript-heavy websites may be affected more strongly.
So compare the Lighthouse versions before you present a higher score as an optimisation success. The field data in the report, by contrast, comes from real usage and not from the synthetic Lighthouse test.
Search queries and website visits are different measures
OpenAI puts ChatGPT’s search-like queries at roughly one fifth of Google’s query volume. This refers to questions that send ChatGPT out to the web. The underlying Google figure, the period and the market are not stated in the article.
Referral data, by contrast, measures visits to websites. For your channel planning, you should therefore always state whether a metric describes search activity or traffic actually delivered.
What this means for your business
From our consulting perspective, the announced control particularly affects companies that publish their own specialist content and want to decide deliberately how AI providers may use it. This also applies if a service provider looks after the website. The concrete opportunity lies in simpler management of training preferences in future. A new legal obligation cannot be derived from the announcement.
Over the coming weeks, you should record your existing rules, assign responsibilities and check the providers’ announcements for actual support. If you use NoArchive or NoCache for Microsoft, the switch belongs in your planning before 1 March 2027. Also document whether you want to treat training and grounding differently.
For companies in Germany and the EU, this technical decision does not replace a separate legal assessment. The source does not allow any statement on whether requirements under the GDPR, the BDSG (the German Federal Data Protection Act), the AI Act or NIS2 are met. A business trade-off also remains open that the source does not examine: what benefit or disadvantage would less grounding have for the visibility of your own content?
As a next step, review your robots.txt and any existing Google-Extended, NoArchive and NoCache settings. Put the usage you want and the support status of each provider in writing before you replace any rules.
More on this topic: all articles in the SEO and visibility in search engines and AI search category. We are happy to answer questions about implementation in your company in an initial consultation.
