AI crawler · Google
Google-Extended
Not a crawler: a robots.txt token that controls whether Google may use your pages for Gemini.
- Operator
- Feeds
- Gemini and Vertex AI
- Type
- Control token (Gemini training and grounding)
- robots.txt token
- Google-Extended
What Google-Extended does
Google-Extended is a robots.txt control token with no user-agent string of its own. Google crawls with its usual crawlers; the token governs whether that content may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
Does it follow robots.txt? It only exists in robots.txt. Google documents what a Disallow under this token controls.
How many popular sites block Google-Extended
15% of the Tranco top 500 sites whose robots.txt answered block Google-Extended for the whole site (48 of 319), as of . The number comes from our weekly AI crawler access study, which counts a site as blocking only when its robots.txt disallows the site root for this token.
robots.txt: allow or block Google-Extended
Put the group in the robots.txt at the root of every host it should cover
(example.com and www.example.com each need their own).
Allow
User-agent: Google-Extended Allow: /
Block
User-agent: Google-Extended Disallow: /
Setting rules for several AI crawlers at once? The free AI robots.txt generator writes the groups for all of them.
User-agent: * rules as they are.
What blocking does and does not do
- Does: Per Google, it stops content Google crawls being used to train future Gemini models and for grounding in Gemini Apps and on Vertex AI.
- Does not: Google states it does not affect your inclusion in Google Search and is not a ranking signal.
- Because there is no Google-Extended user agent, a firewall rule cannot block it. Our crawler check reads the robots.txt rule only and makes no server request for it.
- Our crawler check lists Google-Extended under training controls. Google's documentation also covers grounding in Gemini Apps, so read a block here as affecting more than training.
Check whether your site blocks Google-Extended
Enter your page and our free crawler check opens with it filled in. It reads your robots.txt for Google-Extended and the other nine AI crawler tokens. No signup.
Frequently asked questions
How do I block Google-Extended in robots.txt?
Add a group that names it: "User-agent: Google-Extended" on one line and "Disallow: /" on the next, in the robots.txt at the root of each host you want covered.
Does blocking Google-Extended remove my site from AI answers?
Per Google, it stops content Google crawls being used to train future Gemini models and for grounding in Gemini Apps and on Vertex AI. Google states it does not affect your inclusion in Google Search and is not a ranking signal.
Is Google-Extended a live crawler or a training crawler?
Neither: it is a robots.txt control token, not a crawler. Google documents that it covers Gemini training and grounding in Gemini Apps and on Vertex AI, and that it does not affect Google Search.
How can I check whether my site blocks Google-Extended?
Run the free SEO Sets AI crawler check. It reads your robots.txt rules for this and nine other AI crawler tokens, and for real user agents it also requests your page with that name to see whether your server or CDN refuses it.
Official documentation
Google: Google's common crawlers. Facts on this page were read from it on November 17, 2026. Vendors change their crawlers; the vendor's page wins if it disagrees with ours.
Related
- AI crawler checkFree, no signup: is any AI crawler blocked on your site?
- AI robots.txt generatorFree: allow or block each AI crawler and copy the rules.
- AI crawler directoryEvery AI crawler our check reads, with robots.txt rules.
- Claim your business in ChatGPTA first-hand, step-by-step guide.
- Compare SEO SetsDated, sourced comparisons with Otterly and Peec.
See what to fix on your own site
The free Signal plan connects your Google Search Console and shows your top 3 pages with the most search upside, with no time limit and no credit card. AI citation tracking starts on Surge at $49 a month.