How Do AI Engines Handle Paywalled Content?
Paywalled content and AI describes the collision between subscription publishing and answer engines: crawlers can only learn from what they can fetch, so a paywall is simultaneously a revenue wall and a visibility wall. Publishers manage the trade-off through structured paywall markup, crawler blocking, and — increasingly — direct licensing deals.
What actually happens at the crawl layer
An AI crawler hitting a hard paywall retrieves only the teaser HTML, so the engine can neither train on nor cite the substance. Three publisher postures have emerged. Some block entirely — The New York Times blocked GPTBot in August 2023 and sued OpenAI and Microsoft that December. Some allow crawling of free tiers and metered previews. And some license: content flows to the AI company through negotiated feeds, bypassing the public paywall altogether.
The structured markup option
Google's flexible sampling framework (which replaced "first click free" in 2017) established the pattern engines expect: serve full content to the crawler, declare the paywall honestly in markup.
isAccessibleForFree: falseon the Article JSON-LD signals paid content.- A
hasPartblock referencing the paywalled section's CSS selector tells the engine which portion is gated. - Without this markup, showing crawlers different content than users risks being classified as cloaking.
This markup governs classic indexing; AI answer engines inherit the crawl access it enables, which is why the licensing question has become the sharper lever.
Why this matters for citation competition
Every paywalled analysis that engines cannot read is a citation opportunity transferred to whoever publishes the nearest free equivalent. Wire services, government sources, and open-access competitors absorb the AI visibility that gated originals forfeit. Publishers weigh subscription revenue against becoming invisible in interfaces where a growing share of research happens — the same calculus driving deals like News Corp's reported five-year OpenAI agreement in May 2024.
Example
A research firm keeps its full reports gated but publishes a free statistics digest with sourced headline numbers. AI engines cite the digest, the citations carry the firm's name into thousands of answers, and the paywall stays intact for the deep material — a deliberate free-layer strategy many subscription publishers now copy.
Related terms
See AI content licensing deals, GPTBot, robots.txt, and pay-per-crawl.
Frequently asked questions
- Can ChatGPT read content behind a paywall?
- Not through the paywall itself. Engines only see what their crawlers can fetch — a hard paywall exposes nothing, a metered or lead-in paywall exposes the free portion, and licensed content arrives via direct data feeds regardless of the public paywall.
- What is isAccessibleForFree?
- A schema.org property (true/false) declared on CreativeWork types that tells crawlers whether content requires payment. Combined with hasPart markup flagging the paywalled section's CSS class, it lets publishers show Google full content for indexing without being treated as cloaking.
Keep exploring
See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra