How Often Do AI Engines Recrawl Pages?

There is no fixed recrawl schedule. Log studies show training crawlers like GPTBot and ClaudeBot revisiting high-authority, frequently updated pages daily, while static deep pages can wait weeks or months. Retrieval fetchers such as OAI-SearchBot, ChatGPT-User, and Perplexity-User work differently: they fetch on demand when a prompt triggers a search, so they follow user demand, not a calendar.

What server logs actually show

Vercel's December 2024 analysis of traffic across its network counted roughly 569 million GPTBot requests and 370 million ClaudeBot requests in a single month — volumes approaching a meaningful fraction of Googlebot's. That scale means popular pages on established domains get revisited often, but crawl budget concentrates heavily: homepages, hub pages, and recently changed URLs get the most repeat visits, while orphaned pages deep in the architecture may never see a second fetch.

The same study confirmed AI crawlers do not execute JavaScript, so a client-rendered page that changed its content but not its server-delivered HTML looks unchanged to every AI bot regardless of how often it is crawled.

Crawlers vs. fetchers: two different clocks

This distinction matters for expectations. A pricing change can appear in ChatGPT search answers within hours if OAI-SearchBot or a live fetch picks it up, while the parametric memory of the underlying model only updates at the next training cutoff.

How to raise your recrawl priority

Four levers move actual revisit frequency. First, keep an accurate XML sitemap with honest lastmod values — inflating dates trains crawlers to ignore you. Second, push changes through IndexNow, which updates Bing's index instantly and flows into Copilot and ChatGPT search. Third, strengthen internal links to updated pages; crawlers rediscover URLs through link paths, and a page one click from the homepage gets refetched far sooner than one buried five levels deep. Fourth, return correct Last-Modified and ETag headers so conditional requests stay cheap and crawlers can afford more of them.

Then verify it worked: grep your access logs for the bot user agents before and after a change. Recrawl behavior is one of the few GEO variables you can measure directly from your own infrastructure, without any third-party tool.