What Is Pay-Per-Crawl?

Pay-per-crawl is an emerging compensation model in which AI companies pay websites for the right to crawl their content, replacing the all-or-nothing choice between free access and blocking. The flagship implementation is Cloudflare's Pay Per Crawl marketplace, announced July 1, 2025, alongside a policy shift that made Cloudflare block AI crawlers by default for newly onboarded domains.

Why did pay-per-crawl emerge?

The classic web bargain, crawl my pages, send me traffic, broke under answer engines. Crawl-to-referral ratios published on Cloudflare Radar showed AI companies crawling hundreds to tens of thousands of pages per visit referred, versus roughly 14:1 for Google. Publishers had only blunt instruments: robots.txt (voluntary), lawsuits (slow), or bilateral licensing deals (available only to the largest media brands, OpenAI's deals with Axel Springer, the FT, and News Corp set the pattern from 2023 onward). Pay-per-crawl generalizes licensing into infrastructure any site can use.

How does it work?

Cloudflare's design is elegantly retro, it activates HTTP 402 Payment Required, a status code reserved since the 1990s but never standardized in practice:

1. Publisher sets a price per request (or allows/blocks specific crawlers for free)
2. An AI crawler requesting a page receives 402 Payment Required with pricing signals
3. Crawlers enrolled in the marketplace retry with payment intent headers
4. Content is served; Cloudflare aggregates charges and pays the publisher

Related mechanisms are converging on the same layer: startups like TollBit broker per-crawl licensing, and the proposed Content-Signal and IETF AI-preferences work aim to standardize how sites express training versus search permissions machine-readably.

What should publishers and brands consider?

The calculus splits by business model. Publishers monetizing content itself can price training crawlers high and answer crawlers low, search-facing bots like OAI-SearchBot can still return citation traffic. Brands monetizing what content sells usually want maximum AI ingestion: for them, charging crawlers is friction against their own AI visibility. Either way, the decision should be explicit and monitored, not inherited from a CDN default, as of mid-2025, that default may already be "block."

Example

A B2B software company discovers its docs subdomain returns 402s to GPTBot because the CDN's new default kicked in during a migration. Three weeks later, ChatGPT answers about its API cite a competitor's tutorials instead. The fix is a one-line crawler policy change, and a lesson in owning the access decision deliberately.