Perplexity Crawlers Explained: User Agents, robots.txt and Access Control

Perplexity operates two distinct user agents: PerplexityBot, which builds the search index that grounds answers, and Perplexity-User, which fetches a page live when a specific user request requires it. Allowing both in robots.txt is the baseline requirement for appearing in Perplexity's numbered citations; blocking either one cuts off a different part of the citation pipeline.

What does each Perplexity user agent do?

The split matters for access control. PerplexityBot determines whether your pages can be retrieved for arbitrary queries. Perplexity-User determines whether a page renders correctly when a user directly asks about it. Perplexity states in its crawler documentation that user-triggered agents may not honor robots.txt because the fetch happens on behalf of a human, not an automated crawl.

How should robots.txt be configured for Perplexity?

The permissive configuration most sites want looks like this:

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

If you need to keep private sections out of AI answers, disallow specific paths rather than the whole agent. A publisher blocking /subscriber/ while allowing the rest keeps free content citable. Remember that robots.txt controls crawling, not training — Perplexity is a retrieval engine first, and its index crawl is what produces citations.

How do you verify genuine Perplexity traffic?

User-agent strings are spoofed constantly, so verification means checking source IPs. Perplexity publishes the IP ranges for its crawlers in its developer documentation, and serious infrastructure teams should validate against that list before whitelisting. This became a live issue in June 2024, when reporting from Wired and Forbes accused undisclosed Perplexity fetches of bypassing robots.txt — an episode that pushed Perplexity to document its agents and IPs more explicitly and pushed CDNs to act.

The CDN layer is where most accidental blocking happens. Cloudflare shipped a one-click "block AI bots" toggle in July 2024, and many sites enabled it without realizing it removes them from answer engines they actually want traffic from. Audit your WAF rules, bot-management scores, and challenge pages: a JavaScript challenge served to PerplexityBot is functionally a block, because the crawler will not execute it.

What does a safe WAF configuration look like?

Treat verified Perplexity crawlers as good bots with three rules. First, allowlist by IP range plus UA match, not UA alone. Second, exempt allowlisted crawlers from JavaScript challenges and CAPTCHA — they need raw HTML with a 200 status. Third, monitor crawl volume in your logs; PerplexityBot's request rate is modest for most sites, and a sudden spike from unverified IPs claiming its UA is scraper traffic you can safely block.

Since Perplexity favors recently updated pages, fast re-crawling is an asset. Keep an accurate XML sitemap with real lastmod values so PerplexityBot spends its crawl budget on pages that changed. Sites that gate content behind logins or render body copy client-side in JavaScript serve the crawler an empty shell — the same failure mode as blocking, with none of the intent.

How do you confirm the configuration works?

Grep your server logs for both UA strings and confirm 200 responses on your key pages. Then run live prompts in Perplexity that your content should win and check whether your domain appears among the numbered sources. Systematic verification at scale — tracking which pages get cited, for which prompts, over time — is what citation tracking tooling like Menra automates, turning a one-time robots.txt fix into a monitored acquisition channel. For strategic context on why crawler access is step one of a larger program, see the GEO fundamentals guide.