What Is Log File Analysis? Ground Truth for AI Crawler Attention

Log file analysis is the practice of mining raw server or CDN access logs to see exactly which clients requested which URLs, when, and with what result. Every request leaves a line — timestamp, IP, URL, status code, user agent — and because AI crawlers never execute the JavaScript that powers conventional analytics, logs are the only ground truth for how much machine attention your site actually receives.

What questions should logs answer for GEO?

Four analyses cover most of the value. Which AI agents visit, and how often — segment hits by user-agent token (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) after verifying against published IP ranges so spoofers do not inflate counts. What they fetch — the distribution of bot hits across your URL space reveals whether crawlers reach deep programmatic pages or bounce off the homepage. What they receive — a burst of 403s or 429s for one agent is a bot-management block you did not know about; 200s with tiny byte counts suggest an empty JS shell. And the crawl-to-referral ratio — comparing crawler fetch volume against the human referral traffic engines send back quantifies whether the exchange is worth it, a metric Cloudflare popularized in 2025 when it published ratios showing some AI platforms crawling thousands of pages per referred visitor.

How does the workflow look in practice?

Get the logs first: origin server access logs, or better, CDN logs (Cloudflare Logpush, Fastly streaming) since edge-blocked requests never reach origin. Parse and filter to known bot tokens, verify IPs, then trend weekly. Purpose-built options include Screaming Frog's Log File Analyser for audits and BigQuery for large exports. The pattern to watch is change: an AI crawler that abruptly stops visiting usually signals a new block or an expired allowlist rule, weeks before anyone notices citations dropping.

Why call it "ground truth"?

Every other GEO measurement is downstream inference — rankings sampled, answers probed, citations counted. Logs are direct observation: PerplexityBot either fetched /pricing on Tuesday or it did not. When citation tracking shows an engine going quiet on your brand, logs are where you distinguish "stopped crawling us" from "crawling but not citing," two problems with completely different fixes. Google's own crawler documentation lists the strings to match for its fleet; each AI lab publishes equivalents.