What Is Bot Verification? Confirming Crawler Identity

Bot verification is the process of confirming that a crawler claiming a given identity — Googlebot, GPTBot, ClaudeBot — actually belongs to that operator, using network-level evidence rather than the trivially forgeable user-agent header. It protects in both directions: it stops scrapers that impersonate trusted bots to slip past defenses, and it prevents you from making policy decisions based on log data polluted by fakes.

Why is the user-agent string not enough?

Any HTTP client can send any user-agent. Spoofing well-known crawlers is a standard scraper tactic because many sites whitelist "Googlebot" by string match alone. Studies of server logs regularly find a meaningful share of self-declared search-bot traffic failing verification. If you are counting AI-crawler visits as a demand signal — or granting them WAF exceptions — unverified strings will mislead you.

How do the two verification methods work?

• Reverse-then-forward DNS. Resolve the requester's IP to a hostname, check the domain (Googlebot resolves under googlebot.com or google.com; Bingbot under search.msn.com), then resolve that hostname forward and require it to match the original IP. Google documents this in its crawler verification guide.
• Published IP ranges. Most AI operators skip DNS and publish machine-readable IP lists instead: OpenAI publishes JSON files of ranges for GPTBot, OAI-SearchBot, and ChatGPT-User; Google publishes googlebot.json; Perplexity and Anthropic publish their ranges as well. Verification is a set-membership check you can automate on log ingest.

A third mechanism is emerging: cryptographic request signing. Cloudflare proposed Web Bot Auth in 2025, using HTTP Message Signatures so an agent proves identity per request — a response to the explosion of AI agents that makes IP lists hard to maintain.

What should you do with verification in practice?

Two habits pay off. First, verify before you trust: build your WAF allowlist on verified ranges, not UA strings, so spoofers cannot ride GPTBot's reputation into your origin. Second, verify before you measure: when analyzing AI crawler activity in logs, filter to verified requests, otherwise scraper noise inflates your sense of machine attention. Most CDNs expose a "verified bot" flag that does this work for you — use it as a filter dimension rather than reimplementing DNS checks yourself.