Should I Block AI Crawlers or Let Them In?
Most businesses should let AI crawlers in, because AI answers are becoming a discovery channel and blocked sites cannot be cited. The real decision is more granular: allow retrieval and user-triggered bots almost always, and decide on training bots based on whether your content is a licensable asset or a marketing asset.
The question behind the question
Blocking is not one switch. OpenAI alone runs three bots — GPTBot (training), OAI-SearchBot (search index), ChatGPT-User (live fetches for a user's prompt) — and Anthropic mirrors that split. Blanket-blocking "AI" usually means accidentally cutting the retrieval bots that produce citations and referral clicks, to stop the training bots that were the actual concern.
Decision matrix by business model
| Your situation | Training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) | Retrieval bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) | User-triggered (ChatGPT-User, Perplexity-User) |
|---|---|---|---|
| B2B SaaS, services, brands | Allow | Allow | Allow |
| Ecommerce | Allow | Allow | Allow |
| Ad-funded publisher | Consider blocking | Case-by-case | Allow |
| Licensable content (data, stock, research) | Block, negotiate | Case-by-case | Allow |
| Paywalled journalism | Block, negotiate | Block unless licensed | Case-by-case |
The logic: if AI mentions drive customers to you, crawl access is free distribution. If AI answers substitute for your monetized page view, access is uncompensated extraction — and blocking creates negotiating leverage, which is why News Corp and Axel Springer signed licensing deals with OpenAI rather than staying blocked forever.
What the market actually does
Blocking is common but far from universal. Originality.ai's ongoing tracking found roughly a third of the top 1,000 websites had blocked GPTBot within about a year of its August 2023 launch — dominated by news publishers. Meanwhile ecommerce and SaaS sites overwhelmingly allow AI crawlers, because being recommended when a buyer asks ChatGPT "best tool for X" is the whole prize of GEO.
Recommendation
Default policy for a commercial brand: allow everything except CCBot and Bytespider if you want a conservative posture (Common Crawl feeds many third-party models you cannot audit; Bytespider has documented compliance problems), and revisit quarterly. Whatever you choose, make it deliberate — audit your robots.txt and CDN bot settings today, because many sites are blocking crawlers unintentionally through default WAF rules and losing citations without knowing why.
Frequently asked questions
- Does blocking GPTBot remove my site from ChatGPT?
- It removes you from future OpenAI training data, but ChatGPT search retrieval is governed by OAI-SearchBot. Block GPTBot alone and you can still be cited in ChatGPT search answers; block OAI-SearchBot and you disappear from them.
- Can I block AI training but keep AI search visibility?
- Yes. Disallow the training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) while allowing the search and user-triggered bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot). The two systems use separate user agents at OpenAI and Anthropic.
- Who should block AI crawlers entirely?
- Publishers whose entire business is monetizing page views on commodity content, and sites whose content is their licensable asset — stock media, proprietary data, paywalled research. For most brands, full blocking trades away discovery for no compensating revenue.
Keep exploring
See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra