Which AI Crawlers Should I Allow in robots.txt?
Allow the search-retrieval and user-triggered bots, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, plus Googlebot and Bingbot, since they generate citations and referrals. Training bots (GPTBot, ClaudeBot, Google-Extended) are optional but recommended for brands; the common blocks are CCBot and Bytespider.
Per-bot reasoning
| User agent | Operator | Job | Recommendation |
|---|---|---|---|
| Googlebot | Search + AI Overviews | Allow, blocking kills Google entirely | |
| Bingbot | Microsoft | Bing index, feeds Copilot and ChatGPT search | Allow |
| OAI-SearchBot | OpenAI | ChatGPT search index | Allow, required for ChatGPT citations |
| ChatGPT-User | OpenAI | Live fetch when a user's prompt needs your page | Allow, direct answer presence |
| GPTBot | OpenAI | Model training | Allow for brands; block if licensing content |
| PerplexityBot / Perplexity-User | Perplexity | Answer index + live fetches | Allow |
| ClaudeBot | Anthropic | Model training | Allow for brands |
| Claude-SearchBot / Claude-User | Anthropic | Claude web search | Allow |
| Google-Extended | Gemini training control | Allow; note it does not affect AI Overviews | |
| CCBot | Common Crawl | Open corpus used by many LLMs | Your call, widest reach, zero control |
| Bytespider | ByteDance | Training | Block, repeatedly reported ignoring robots.txt |
| Meta-ExternalAgent | Meta | Training + indexing | Allow if you want Meta AI presence |
Two details trip people up. First, Google-Extended is a control token, not a crawler, regular Googlebot does the fetching, and AI Overviews eligibility follows normal Google indexing, not Google-Extended. Second, ChatGPT's live search also leans on Bing's index, so Bing Webmaster Tools hygiene quietly matters for ChatGPT visibility.
Copy-paste configuration
A brand-friendly policy that keeps every answer engine open while trimming uncontrolled corpora:
# AI search and user-triggered fetchers, keep open
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
# Training crawlers, open by default for visibility
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
# Uncontrolled or non-compliant
User-agent: CCBot
User-agent: Bytespider
Disallow: /
Remember that robots.txt is voluntary under the Robots Exclusion Protocol (standardized as RFC 9309 in 2022), compliant operators honor it, and non-compliant ones need CDN or WAF enforcement.
Keep it under review
New bots appear every few months, Apple's Applebot-Extended and Amazon's Amazonbot both arrived after the first GPTBot wave in August 2023. Re-audit your robots.txt quarterly against your server logs, and confirm your CDN is not silently overriding the policy you wrote; see the GEO optimization guide for the full crawl-access checklist.
Frequently asked questions
- Which AI bots are essential for visibility?
- OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, Claude-SearchBot and Claude-User for Claude, plus regular Googlebot and Bingbot, which feed AI Overviews and Copilot respectively. Blocking any of these removes you from that engine's answers.
- Is allowing everything a reasonable default?
- For most commercial sites, yes. The main candidates for blocking are CCBot, which feeds unaudited third-party training corpora, and Bytespider, which has documented robots.txt compliance problems. Everything else trades content access for potential citations.
Keep exploring
See how AI engines talk about your brand, track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra