ChatGPT Crawlers Explained: User Agents, robots.txt and Access Control

OpenAI operates three crawlers, and confusing them is the most common access-control mistake on the web today. GPTBot gathers model training data, OAI-SearchBot builds the index behind ChatGPT Search, and ChatGPT-User fetches pages live when a user's prompt triggers browsing. Only the last two affect whether ChatGPT can cite you — so a robots.txt written in 2023 to block "AI scraping" often blocks the citations a marketing team now wants.

What does each OpenAI user agent do?

Full UA strings include version and contact info, for example Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot. Match on the token, not the whole string — OpenAI revises the surrounding boilerplate.

What does a safe robots.txt look like?

The recommended pattern for brands that want citations but not training use:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

If you are comfortable with training use — and there is an argument for it, since parametric brand knowledge shapes answers even without browsing — allow all three. What you should never do is User-agent: * disallow rules intended for one bot: robots.txt matching is per-agent, and the most specific block wins, so sloppy wildcards produce silent overblocking. Keep sensitive paths (/cart/, /account/, internal search) disallowed for all agents; crawlers gain nothing from them and they waste fetch budget.

How do you verify real OpenAI traffic?

The user-agent header is trivially spoofable, and scrapers impersonate OpenAI bots constantly. OpenAI publishes machine-readable IP range lists for each agent, linked from its bots documentation. Verification is a two-step check: UA token matches, and source IP falls inside the published ranges. Requests that pass step one but fail step two are impostors — block them freely without touching your real ChatGPT visibility.

Build this into log analysis rather than doing it ad hoc. A monthly query counting verified hits per agent tells you three useful things: whether OAI-SearchBot is crawling at all, which pages ChatGPT-User fetches most (a direct proxy for which of your pages appear in answers), and whether impostor traffic is inflating your bot numbers.

Which WAF and CDN settings silently block ChatGPT?

Robots.txt is only the first gate. The failures that actually take sites out of ChatGPT usually happen at the edge:

1. Cloudflare AI-bot blocking. Cloudflare introduced one-click and later default-on AI crawler blocking in 2024. Check Security → Bots and explicitly allow the two retrieval agents.
2. Bot-score challenges. JavaScript challenges and CAPTCHAs return challenge pages to crawlers that cannot solve them. OpenAI's fetchers do not complete JS challenges; the crawl sees a 403 or an interstitial instead of your content.
3. Rate limiting. Aggressive per-IP limits can throttle OAI-SearchBot mid-crawl. Whitelist the published ranges from rate rules.
4. Geo-blocking. Crawler egress IPs sit in US cloud regions; region blocks aimed at abuse can catch them.

After any change, test from the outside: curl -A "OAI-SearchBot" https://yoursite.com/key-page and confirm a 200 with full HTML body.

How do you know the access work paid off?

Access is a means; citations are the end. Once logs show verified OAI-SearchBot crawls and ChatGPT-User fetches, the remaining question is whether those fetches convert into source-chip citations for prompts that matter. Running a fixed prompt set through Menra's citation tracking closes that loop — if crawl volume is healthy but citations stay flat, your problem has moved from access to content and ranking, which is exactly where you want it. Definitions for every crawler and directive mentioned here live in the GEO glossary.