Gemini Crawlers Explained: User Agents, robots.txt and Access Control

Gemini has no crawler of its own. It grounds answers in Google's Search index, so the "Gemini crawlers" question resolves into three Google mechanisms: Googlebot, which builds the index Gemini retrieves from; Google-Extended, a robots.txt-only token that gates Gemini's use of that crawled content; and GoogleOther, a general-purpose fetcher Google uses for research and product development outside Search. Understanding which is a bot and which is a switch prevents the two most common misconfigurations in AI-era robots.txt files.

The user agents and tokens, itemized

The load-bearing row is Google-Extended, introduced in September 2023. Because it piggybacks on Googlebot's crawl, you cannot rate-limit it, cannot see it, and cannot firewall it — robots.txt is the only interface.

Safe robots.txt configurations

Full Gemini visibility (the default posture for most brands):

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

Search visibility without Gemini participation (a licensing stance some publishers take):

User-agent: Google-Extended
Disallow: /

Note the absence of an explicit Googlebot block in the second config — Search rankings survive untouched. Path-scoped rules also work: some sites disallow Google-Extended only on premium content directories while leaving marketing pages groundable. Audit annually; blanket AI-bot blocklists copied from 2023-era templates frequently contain a Google-Extended disallow the site owner no longer intends.

Verifying genuine Google crawlers

Fake Googlebots are endemic — scrapers spoof the UA string to slip past WAF rules. Verification is a two-step DNS dance: reverse-resolve the requesting IP (a real crawler resolves to crawl-***.googlebot.com or google.com), then forward-resolve that hostname and confirm it returns the original IP. For infrastructure-level enforcement, Google publishes machine-readable JSON files of official crawler IP ranges at developers.google.com, refreshed regularly; load them into your WAF allowlist instead of trusting user-agent headers. Never hard-block on UA mismatch alone, and never challenge verified Google IPs with CAPTCHAs — an unrenderable challenge page is what gets indexed.

WAF and CDN pitfalls specific to Gemini visibility

Because Gemini inherits the index, anything that degrades Googlebot's crawl degrades Gemini: aggressive bot-score thresholds that intermittently 403 real crawlers, geo-blocks covering Google's US crawl ranges, and JS-challenge interstitials served to non-browser clients. Symptoms surface as Search Console crawl anomalies before they surface in Gemini answers, so wire crawl-error alerting into whatever dashboard you already watch. The upstream dependency cuts both ways — fixing crawl health pays off in classic rankings and generative visibility simultaneously, and the glossary maps the rest of the crawler vocabulary across engines if you are hardening robots.txt for all of them at once.