Google-Extended

It separates two questions that one crawler used to answer together. Google crawls with its usual agents, and reads Google-Extended only in what it calls a control capacity: there is no separate user agent string in the requests^[1]. A group for the token in robots.txt changes what Google may do with content it has fetched, not whether Googlebot fetches it.

Its reach is set out product by product. Google says the token governs whether content may be used to train future generations of the Gemini models that power Gemini Apps and the Vertex AI API for Gemini, and for grounding, which it defines as providing content from the Google Search index to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI^[1]. The same page says it does not affect a site’s inclusion in Google Search and is not used as a ranking signal^[1].

That leaves Google’s AI features in Search outside its reach. AI Overviews and AI Mode are part of Search: Google names the robots.txt rules for Googlebot as the control over how a site is crawled for them, and nosnippet, data-nosnippet, max-snippet or noindex as the ways to limit what they show^[2]. A site that writes Disallow: / under Google-Extended can still be linked in an AI Overview.

Because no request carries the name, server logs never show it. The only record of the choice is the file, read the way RFC 9309 sets out for every token: the group whose user-agent line matches it, without regard to case, and within that group the most specific matching path^[3].

Sites refuse the opt-out tokens of different vendors at much the same rate. In QueryRev’s study, Google-Extended is blocked by name on 411 of the 6,294 sites whose robots.txt could be read, 6.5%, and Applebot-Extended, which the study lists as Apple’s opt-out from model training, on 394, 6.3%^[4].

In the file, host by host. Fetch robots.txt from every host the site serves, find the group that applies to Google-Extended and record the paths it closes; a rule on the main domain says nothing for a shop or a help centre on its own subdomain, for the reasons set out under robots.txt.

Then measure what the rule should leave alone. Search Console reports the clicks and impressions of AI Overviews and AI Mode within the Web search type, with the rest of Search^[2], so a change to Google-Extended should leave those figures where they were. Date the change and read the weeks either side of it.

What it changes on the Gemini side is not reported back to the site. The observable part is the Gemini answers to the prompts you track: whether they still quote or link your pages after the rule, read over several runs.

Sample data: an invented site, to show the method.

A cookware shop publishes 400 recipes and wants them kept out of Gemini training while they stay in Search and in AI Overviews. The first draft of the rule closes /recipes/ to Googlebot, which would take the recipes out of classic results and AI Overviews as well.

The rule that matches the goal names Google-Extended and leaves Googlebot’s group as it was. Search Console clicks to the recipes over the following weeks are the check that nothing else moved.