Claude Crawlers Explained: User Agents, robots.txt and Access Control

Anthropic operates three user agents with cleanly separated jobs: ClaudeBot gathers training data, Claude-SearchBot crawls for the search index behind Claude's web search (launched March 2025), and Claude-User fetches pages live when a user's request requires a specific URL. Because the functions are split by UA, you can express nuanced policies — for example, opting out of training while remaining fully citable — with three robots.txt blocks.

What does each Anthropic user agent do?

The separation is the point. During 2023-2024, ClaudeBot appeared on blocklists across the web as publishers opted out of AI training — an understandable stance that became a problem when Claude gained web search in March 2025, because sites with blanket "Claude*" wildcard rules or copied blocklists had unknowingly pre-blocked the retrieval agents too. The training-era robots.txt audit is the first task on any Claude visibility checklist.

What do the standard robots.txt configurations look like?

Fully open:

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Searchable but not trainable — the config many publishers actually want:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Path-level rules compose normally: a SaaS company might disallow /app/ and /customers/private/ for all three agents while leaving marketing and docs open. Anthropic documents its crawlers and states they respect robots.txt; treat that as reliable convention, and use WAF enforcement where you need a hard guarantee.

How do you verify traffic actually comes from Anthropic?

User-agent strings cost nothing to fake, and scrapers routinely impersonate AI crawlers to borrow their welcome. Verification is IP-based: check requesting IPs against Anthropic's published crawler information rather than trusting the UA header, and treat UA-claiming traffic from unrelated cloud IPs as ordinary bot traffic subject to your normal rules. In server logs, genuine Anthropic crawl patterns are also behaviorally distinct — Claude-SearchBot crawls broadly on a schedule, while Claude-User arrives as single-page fetches correlated with nothing.

Where do CDNs and WAFs silently break Claude access?

Three places. Bot-management products score unfamiliar crawlers as threats and serve them JavaScript challenges — which an HTML-only fetcher cannot solve, converting "challenge" into "block." One-click AI-bot toggles (Cloudflare shipped its block-AI-bots switch in July 2024) block training and retrieval agents together, re-creating the wildcard mistake at the CDN layer. And rate-limit rules tuned for human browsing throttle legitimate crawl bursts into 429s. The fix pattern is the same for all three: explicit allow rules for verified Claude-SearchBot and Claude-User, exempted from challenges, with the decision documented so the next infrastructure change doesn't silently revert it.

How does this fit the larger visibility picture?

Crawler access is binary plumbing: it can only make citations possible, never earn them. Once the three agents fetch clean 200s and your pages sit in the upstream index, the work shifts to content extractability and corroboration — the GEO fundamentals layer. Confirm the plumbing holds over time by watching whether Claude actually cites you: citation tracking across a prompt set will surface an access regression within days, usually before anyone thinks to re-read robots.txt.