Should publishers block ClaudeBot but allow Claude-SearchBot?

That is the standard split. ClaudeBot gathers training data, while Claude-SearchBot feeds the search index that produces cited, linked answers. Blocking all Anthropic crawlers removes you from Claude's answers entirely; blocking only ClaudeBot preserves citations while opting out of training.

Can Claude read paywalled articles?

No. Claude-User fetches pages as an ordinary client and respects robots.txt, so hard paywalls return the same truncated page any logged-out visitor sees. Publishers using metered or lead-in paywalls give Claude the visible portion — often enough for a citation but not a full summary.

Does Claude send referral traffic to publishers?

Yes — cited sources appear as links users can click through. Volume is smaller than Google search referrals, but it is measurable: filter your analytics for claude.ai referrers and traffic from Claude-User fetches to size it for your property.

News and Publisher Visibility in Claude

Last updated: July 5, 2026 · Menra Team

Claude surfaces news through its web search tool, citing and linking the articles it draws on — which makes the publisher question less "should we be in Claude" and more "on what terms." Anthropic operates three crawlers with distinct purposes, and the robots.txt decisions you make for each determine whether your journalism appears as a cited, linked source or disappears from Claude's answers entirely.

The three-crawler decision every publisher faces

Crawler	Purpose	If you block it
`ClaudeBot`	Training data collection	No effect on citations; content excluded from future model training
`Claude-SearchBot`	Builds the search index behind Claude's web answers	Your articles stop being retrievable — no citations, no referrals
`Claude-User`	Live fetch when a user's prompt triggers it	Claude cannot read your page on demand, even when a user pastes your URL

The pragmatic default for most newsrooms: allow Claude-SearchBot and Claude-User to stay citable, and decide on ClaudeBot according to your licensing posture. All three respect robots.txt, so the control is real and granular.

How does Claude handle freshness and breaking news?

Claude's web search, launched in March 2025, retrieves against a Brave Search-backed index, so your visibility for breaking stories depends on how quickly that index picks up new URLs. Publishers should verify their coverage appears in Brave Search and submit through Brave's webmaster tooling if it does not. For developing stories, Claude tends to favor sources that timestamp clearly and update visibly — an article with an explicit "Updated July 5, 2026" line and a dateline gives the model the recency evidence it needs to prefer your version over a wire rewrite.

Attribution behaves conservatively: Claude typically names the outlet and links the article rather than reproducing long verbatim passages. That makes the citation itself the asset, and headline-plus-lede quality the lever — a lede that states who, what, when, and where in 40-80 self-contained words is exactly the passage Claude extracts.

Paywalls: what Claude actually sees

Claude-User fetches your page like a logged-out reader with no JavaScript execution. Hard paywalls serve it nothing beyond the teaser; client-side paywalls that ship the full text in HTML and hide it with JS effectively give Claude the entire article. Audit which of these your CMS does — many publishers discover their "paywalled" content is fully machine-readable. A deliberate middle path: serve a substantive standfirst and key facts in the free portion so Claude can cite and link you, while the analysis stays behind the meter.

Schema and structure for news

Implement NewsArticle JSON-LD with headline, datePublished, dateModified, author (linked to a Person entity with a bio page), and publisher. Claude reads raw HTML, so server-rendered schema gives it unambiguous bylines and timestamps — the inputs behind both freshness ranking and correct attribution. E-E-A-T scaffolding matters here more than in most verticals: named authors with credentials and an about page distinguish original reporting from aggregation, and answer engines demote anonymous content in news-adjacent spaces.

Measuring what Claude does with your journalism

Track three numbers monthly. Citation share: for the story categories you own, how often does Claude cite you versus rival outlets — citation tracking can run this prompt sampling continuously. Referral traffic: claude.ai referrers in analytics, trended. Crawler activity: Claude-SearchBot and Claude-User hits in server logs, which reveal what Claude's users are actually asking about your coverage. Together these turn the AI-and-publishers debate into an empirical dashboard for your own property — and give you the data to renegotiate your robots.txt posture if the citation-to-crawl ratio stops making sense.

Frequently asked questions

Should publishers block ClaudeBot but allow Claude-SearchBot?: That is the standard split. ClaudeBot gathers training data, while Claude-SearchBot feeds the search index that produces cited, linked answers. Blocking all Anthropic crawlers removes you from Claude's answers entirely; blocking only ClaudeBot preserves citations while opting out of training.
Can Claude read paywalled articles?: No. Claude-User fetches pages as an ordinary client and respects robots.txt, so hard paywalls return the same truncated page any logged-out visitor sees. Publishers using metered or lead-in paywalls give Claude the visible portion — often enough for a citation but not a full summary.
Does Claude send referral traffic to publishers?: Yes — cited sources appear as links users can click through. Volume is smaller than Google search referrals, but it is measurable: filter your analytics for claude.ai referrers and traffic from Claude-User fetches to size it for your property.

Keep exploring

See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra