Claude Technical SEO Requirements: Crawling, Rendering and Indexing

Claude's retrieval stack has four technical dependencies: indexation in the upstream search layer (Brave Search has been reported as the backbone since Claude's web search launched in March 2025), fetch access for Anthropic's user agents (Claude-SearchBot for search, Claude-User for user-triggered fetches, ClaudeBot for training), server-rendered HTML because no JavaScript is executed, and clean canonical and sitemap hygiene so the right URLs get indexed. Fail any of these and content quality never gets evaluated.

What is the pass/fail requirements table?

Why is Brave Search the overlooked dependency?

Teams optimize for Google reflexively, but Claude's web search was reported at launch (March 2025) to be built on Brave Search — a fully independent index, not a Bing or Google reseller. Brave crawls the web itself, which means your Google rankings transfer nothing automatically. Verify your important pages actually appear in Brave, submit and monitor through Brave's Webmaster Tools, and remember that Brave's independent crawler has its own coverage gaps on newer or low-authority sites. If your domain is thin in Brave, that single upstream fact caps your Claude visibility regardless of on-page quality.

What does the no-JavaScript rule break?

Everything a modern SPA does after first paint. Client-rendered product descriptions, tabbed content loaded on click, JS-injected JSON-LD, cookie-wall interstitials that gate the HTML — all of it reads as empty or garbage to a fetcher that takes the server response and stops. The fix hierarchy: full server-side rendering or static generation for content pages first; prerendering for crawler UAs as a workaround; at minimum, ensure the answer-bearing text and schema markup are in the initial HTML payload even if enhancements hydrate later. The curl test is the honest arbiter — if the fact isn't in the response body, Claude has never read it.

How should robots.txt distinguish the three Anthropic agents?

They serve different functions and deserve separate decisions. ClaudeBot gathers training data; blocking it expresses a training-consent choice. Claude-SearchBot builds search retrieval; blocking it removes you from Claude's searched answers. Claude-User fetches when a user directly asks about your URL; blocking it breaks even explicit user requests. The costly mistake, common since the 2023-2024 crawler backlash, is a copied blocklist that disallows all three when the site only meant to opt out of training. Audit for wildcards and stale rules; the crawler-level details and safe configs are itemized in our AEO checklist.

How do you confirm the whole chain works?

End-to-end verification beats component checks: ask Claude questions your pages should win, with search triggered, and see whether your domain appears in citations. Do it across a structured prompt set and over time — one-off spot checks miss regressions from WAF changes, replatforms, or robots.txt edits. Continuous visibility monitoring catches the "we redesigned the site and vanished from Claude" failure while it's days old instead of quarters.