Which AI Crawlers Should I Allow in robots.txt?
Allow the search-retrieval and user-triggered bots — OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User — plus Googlebot and Bingbot, since they generate citations and referrals. Training bots (GPTBot, ClaudeBot, Google-Extended) are optional but recommended for brands; the common blocks are CCBot and Bytespider.
Per-bot reasoning
| User agent | Operator | Job | Recommendation |
|---|---|---|---|
| Googlebot | Search + AI Overviews | Allow — blocking kills Google entirely | |
| Bingbot | Microsoft | Bing index, feeds Copilot and ChatGPT search | Allow |
| OAI-SearchBot | OpenAI | ChatGPT search index | Allow — required for ChatGPT citations |
| ChatGPT-User | OpenAI | Live fetch when a user's prompt needs your page | Allow — direct answer presence |
| GPTBot | OpenAI | Model training | Allow for brands; block if licensing content |
| PerplexityBot / Perplexity-User | Perplexity | Answer index + live fetches | Allow |
| ClaudeBot | Anthropic | Model training | Allow for brands |
| Claude-SearchBot / Claude-User | Anthropic | Claude web search | Allow |
| Google-Extended | Gemini training control | Allow; note it does not affect AI Overviews | |
| CCBot | Common Crawl | Open corpus used by many LLMs | Your call — widest reach, zero control |
| Bytespider | ByteDance | Training | Block — repeatedly reported ignoring robots.txt |
| Meta-ExternalAgent | Meta | Training + indexing | Allow if you want Meta AI presence |
Two details trip people up. First, Google-Extended is a control token, not a crawler — regular Googlebot does the fetching, and AI Overviews eligibility follows normal Google indexing, not Google-Extended. Second, ChatGPT's live search also leans on Bing's index, so Bing Webmaster Tools hygiene quietly matters for ChatGPT visibility.
Copy-paste configuration
A brand-friendly policy that keeps every answer engine open while trimming uncontrolled corpora:
# AI search and user-triggered fetchers — keep open
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
# Training crawlers — open by default for visibility
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
# Uncontrolled or non-compliant
User-agent: CCBot
User-agent: Bytespider
Disallow: /
Remember that robots.txt is voluntary under the Robots Exclusion Protocol (standardized as RFC 9309 in 2022) — compliant operators honor it, and non-compliant ones need CDN or WAF enforcement.
Keep it under review
New bots appear every few months — Apple's Applebot-Extended and Amazon's Amazonbot both arrived after the first GPTBot wave in August 2023. Re-audit your robots.txt quarterly against your server logs, and confirm your CDN is not silently overriding the policy you wrote; see the GEO optimization guide for the full crawl-access checklist.
Frequently asked questions
- Which AI bots are essential for visibility?
- OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, Claude-SearchBot and Claude-User for Claude, plus regular Googlebot and Bingbot, which feed AI Overviews and Copilot respectively. Blocking any of these removes you from that engine's answers.
- Is allowing everything a reasonable default?
- For most commercial sites, yes. The main candidates for blocking are CCBot, which feeds unaudited third-party training corpora, and Bytespider, which has documented robots.txt compliance problems. Everything else trades content access for potential citations.
Keep exploring
See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra