Google AI Overviews Crawlers Explained: User Agents, robots.txt and Access Control
Three robots.txt tokens matter for Google AI Overviews, and only one of them gates it: Googlebot (the Search crawler whose index AI Overviews retrieves from), Google-Extended (a training/grounding control token for Gemini that does not affect Search), and GoogleOther (a general-purpose crawler for non-Search uses). Most confusion around "blocking AI" on Google stems from conflating these three — this page separates them precisely.
The user agent reference table
| Token / UA | What it is | Feeds AI Overviews? | Blocking it means |
|---|---|---|---|
Googlebot (desktop + smartphone) | Primary Search crawler | Yes — sole retrieval source | Out of Search AND AI Overviews entirely |
Google-Extended | robots.txt control token, not a distinct fetcher | No | Gemini apps/training lose your content; Search and AIO unaffected |
GoogleOther (+ variants) | General crawler for non-Search internal uses, split out in 2023 | No | Non-Search Google products lose crawl access |
Googlebot-Image / Googlebot-Video | Media crawlers | Media in overviews only | Images/video drop from Search features |
Google-InspectionTool | URL Inspection / Rich Results Test fetcher | Indirectly (diagnostics) | Your own Search Console tests break |
Full UA strings include version tokens, e.g. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html); match on the product token, not the full string, in robots.txt and WAF rules.
Why is there no "AI Overviews bot"?
Architecture. AI Overviews, launched May 2024 and expanded to 100+ countries by October 2024, is a Search feature: a Gemini model composes answers from passages already sitting in the Search index. Google deliberately did not create a separate opt-out crawler for it — the controls it points publishers to are the snippet directives (nosnippet, max-snippet, data-nosnippet), which govern how much of an indexed page generative features may quote. This coupling is strategic: opting out of AI Overviews means giving up snippet real estate in classic results too.
Safe robots.txt configurations by intent
Full AI visibility (recommended for most brands):
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /
Search visibility, but no Gemini training/grounding:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /
This second config keeps AI Overviews citations intact — a fact worth internalizing, because it is the single most misunderstood point in Google crawler control. To exclude specific content from AI Overviews without leaving Search, use data-nosnippet around the sensitive HTML sections instead of robots.txt.
Verification: never trust the user-agent header
Scrapers impersonate Googlebot constantly. Genuine verification is two-step: reverse-DNS the IP (must resolve to *.googlebot.com or *.google.com), then forward-resolve that hostname and confirm it returns the original IP. At scale, use Google's published crawler IP ranges (JSON files under developers.google.com/search) in your WAF instead of per-request DNS. This matters for AI visibility because overzealous bot protection is a common silent killer: Cloudflare and similar WAF presets can challenge or rate-limit legitimate Googlebot fetches, degrading crawl coverage — and with it, passage availability for overviews. Audit WAF logs for 403/429 responses to verified Google IPs quarterly.
How access failures show up downstream
A robots block is loud (Search Console flags it); a WAF challenge or geo-fence is quiet — pages drift out of the index or index with stale content, and your AI Overview citations decay without an obvious cause. The diagnostic chain is: verified-crawler access logs → URL Inspection index status → live query sampling. For that last step, Menra's visibility monitoring tracks whether your pages actually appear as overview citations over time, which is the only signal that confirms the whole chain end to end. For definitions of crawler and directive terms used here, the GEO glossary covers the full vocabulary, and the optimization guide covers what to do once access is proven clean.
Frequently asked questions
- Which crawler do I need to allow for AI Overviews?
- Googlebot — the same crawler that powers ordinary Search. AI Overviews retrieves from the standard Search index, so Googlebot access plus snippet eligibility is the entire access requirement. There is no dedicated AI Overviews user agent to allow or block.
- What does Google-Extended actually control?
- Google-Extended is a robots.txt token (introduced September 2023) that controls whether your content is used for Gemini model training and for grounding in some Gemini surfaces. It is not a separate crawler with its own fetches, and disallowing it has no effect on Search rankings or AI Overviews inclusion.
- How do I verify a request claiming to be Googlebot is genuine?
- Reverse-DNS the requesting IP: genuine Googlebot IPs resolve to googlebot.com or google.com hostnames that forward-resolve back to the same IP. Google also publishes JSON files of official crawler IP ranges. User-agent strings alone are trivially spoofed and should never gate anything.
Keep exploring
See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra