What Is robots.txt? The Robots Exclusion Protocol in the AI Era
robots.txt is a plain-text file at a site's root that implements the Robots Exclusion Protocol — the convention by which site owners tell crawlers which paths they may or may not fetch. Created by Martijn Koster in 1994 and formally standardized as RFC 9309 in September 2022, it has become the primary policy lever for controlling AI crawlers.
What changed in the AI era?
Between 2023 and 2024, every major AI lab introduced its own robots.txt token: OpenAI's GPTBot (August 2023), Google-Extended (September 2023, controlling Gemini training without touching Googlebot), Anthropic's ClaudeBot, Apple's Applebot-Extended, and Meta-ExternalAgent. The file that once managed one search crawler now governs a dozen agents with different purposes — training, search indexing, and live answer retrieval — each deserving a separate decision.
What does a recommended AI policy matrix look like?
The key distinction is purpose. Blocking a retrieval agent costs you answer-engine citations; blocking a training crawler only affects future model knowledge.
| Agent | Purpose | Typical recommendation |
|---|---|---|
| OAI-SearchBot, PerplexityBot, Claude-SearchBot | Answer-engine indexing | Allow |
| ChatGPT-User, Claude-User, MistralAI-User | Live user-initiated fetches | Allow |
| GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent | Model training | Business decision |
| CCBot, Bytespider | Bulk scraping / opaque training | Often disallow |
For most brands pursuing GEO, the answer-path agents should be explicitly allowed, because AI visibility is impossible without them.
How does the syntax work?
The protocol defines User-agent, Disallow, and Allow records, plus the widely supported Sitemap directive. Rules are grouped per agent, longest-path match wins, and an empty Disallow: permits everything. Crawl-delay is non-standard and ignored by Google. Full syntax lives in Google's robots.txt documentation.
What are the common failure modes?
Three recur constantly: blanket Disallow: / rules left over from staging, CDN bot-fight settings that override a permissive robots.txt at the network layer, and blocking a token that never appears in logs anyway (Google-Extended is a robots token only — there is no Google-Extended user agent to see). Audit the file quarterly against your server logs, because your effective crawl policy is what your infrastructure enforces, not what the file declares.
Frequently asked questions
- Is robots.txt legally enforceable?
- No. It is a voluntary convention, standardized as RFC 9309 in September 2022. Reputable crawlers from OpenAI, Google, Anthropic, and Microsoft honor it, but it is not access control — bad actors ignore it. Enforcement requires WAF or bot-management rules.
- Does blocking GPTBot remove my site from ChatGPT?
- Blocking GPTBot opts you out of OpenAI's training crawls, but ChatGPT search visibility depends on OAI-SearchBot and ChatGPT-User. You can block training while staying citable in ChatGPT answers by allowing those two agents.
- Where does robots.txt live?
- Always at the domain root: https://example.com/robots.txt. Crawlers do not look for it anywhere else, and each subdomain needs its own file.
Keep exploring
See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra