What Is Pay-Per-Crawl?

Pay-per-crawl is an emerging compensation model in which AI companies pay websites for the right to crawl their content, replacing the all-or-nothing choice between free access and blocking. The flagship implementation is Cloudflare's Pay Per Crawl marketplace, announced July 1, 2025, alongside a policy shift that made Cloudflare block AI crawlers by default for newly onboarded domains.

Why did pay-per-crawl emerge?

The classic web bargain — crawl my pages, send me traffic — broke under answer engines. Crawl-to-referral ratios published on Cloudflare Radar showed AI companies crawling hundreds to tens of thousands of pages per visit referred, versus roughly 14:1 for Google. Publishers had only blunt instruments: robots.txt (voluntary), lawsuits (slow), or bilateral licensing deals (available only to the largest media brands — OpenAI's deals with Axel Springer, the FT, and News Corp set the pattern from 2023 onward). Pay-per-crawl generalizes licensing into infrastructure any site can use.

How does it work?

Cloudflare's design is elegantly retro — it activates HTTP 402 Payment Required, a status code reserved since the 1990s but never standardized in practice:

1. Publisher sets a price per request (or allows/blocks specific crawlers for free)
2. An AI crawler requesting a page receives 402 Payment Required with pricing signals
3. Crawlers enrolled in the marketplace retry with payment intent headers
4. Content is served; Cloudflare aggregates charges and pays the publisher

Related mechanisms are converging on the same layer: startups like TollBit broker per-crawl licensing, and the proposed Content-Signal and IETF AI-preferences work aim to standardize how sites express training versus search permissions machine-readably.

What should publishers and brands consider?

The calculus splits by business model. Publishers monetizing content itself can price training crawlers high and answer crawlers low — search-facing bots like OAI-SearchBot can still return citation traffic. Brands monetizing what content sells usually want maximum AI ingestion: for them, charging crawlers is friction against their own AI visibility. Either way, the decision should be explicit and monitored, not inherited from a CDN default — as of mid-2025, that default may already be "block."

Example

A B2B software company discovers its docs subdomain returns 402s to GPTBot because the CDN's new default kicked in during a migration. Three weeks later, ChatGPT answers about its API cite a competitor's tutorials instead. The fix is a one-line crawler policy change — and a lesson in owning the access decision deliberately.