What Is a User Agent? Reading AI Crawler Strings in Your Logs
A user agent (UA) is the string an HTTP client sends in the User-Agent request header to identify itself — browser, version, device, or bot name. Defined as part of HTTP semantics (RFC 9110), it is the primary key for recognizing which AI systems are fetching your content, since every legitimate crawler embeds a stable, documented token in its string.
How do you read a UA string in practice?
Crawler strings follow a loose convention: a Mozilla compatibility prefix, then the bot token and version, often with a URL explaining the bot. For example, OpenAI's search crawler sends a string containing OAI-SearchBot/1.0; +https://openai.com/searchbot. In log analysis you match on the token substring — GPTBot, ClaudeBot, PerplexityBot — rather than the full string, which operators revise without notice.
Which AI-related user agents matter?
| Token | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training crawl |
| OAI-SearchBot | OpenAI | ChatGPT search indexing |
| ChatGPT-User | OpenAI | Live fetch during a user chat |
| ClaudeBot | Anthropic | Training crawl |
| Claude-SearchBot / Claude-User | Anthropic | Search indexing / live fetch |
| PerplexityBot / Perplexity-User | Perplexity | Index crawl / live fetch |
| Meta-ExternalAgent | Meta | AI training |
| MistralAI-User | Mistral | Le Chat live fetch |
| Amazonbot | Amazon | Alexa and AI crawl |
| Applebot-Extended | Apple | Robots token for AI training opt-out |
| Bytespider | ByteDance | Bulk crawl (often ignores robots.txt) |
| CCBot | Common Crawl | Open dataset used in LLM training |
Note the pattern: each major lab splits identity by purpose — training, indexing, and user-initiated retrieval — so you can set policy per function rather than per company.
Why is the training/retrieval split the key insight?
A ChatGPT-User hit means a human is asking about something your page answers right now; a GPTBot hit means your content may inform future model weights. Counting them together destroys the signal. Segment your log dashboards by purpose class, and track the retrieval-class agents as a leading indicator of AI answer demand — rising Perplexity-User fetches to a page reliably precede rising Perplexity citations of it. OpenAI documents its three bots and their published IP ranges at platform.openai.com/docs/bots, and other labs maintain equivalent pages worth bookmarking for the inevitable string changes.
Frequently asked questions
- Why does Google-Extended never appear in my logs?
- Because it is not a crawler. Google-Extended is a robots.txt token that tells Google whether Googlebot-fetched content may be used for Gemini training and grounding. The fetching is done by Googlebot itself, so there is no separate user-agent string to observe.
- Are user-agent strings reliable?
- Only as a first filter. Any client can claim any string, so meaningful analysis pairs the UA with verification against published IP ranges or reverse DNS. Treat unverified bot strings in logs as claims, not facts.
Keep exploring
See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra