Does Perplexity Respect robots.txt?

Perplexity states that its indexing crawler, PerplexityBot, respects robots.txt directives, while its user-triggered agent, Perplexity-User, fetches pages on demand and is not treated as a crawler that robots.txt governs. In 2024, security researchers at Cloudflare and journalists at Wired documented fetches that appeared to bypass disallow rules, prompting Perplexity to publish clearer crawler documentation. Behavior is contested, so verify against your own logs.

What did the crawling controversy involve?

In June 2024, Wired and developer Robb Knight reported that Perplexity retrieved and summarized content from pages that blocked its declared user agent, sometimes using a generic browser string from an undeclared IP. In August 2024, Cloudflare published network evidence of stealth crawling and added Perplexity to its managed bot-blocking rules. Perplexity responded that Perplexity-User acts on a specific human request — closer to a person pasting a URL than to bulk crawling — and argued robots.txt was designed for automated indexing, not on-demand retrieval.

What is the current stated behavior?

Perplexity's published policy separates two agents with different rules:

This split is the crux: even a correct Disallow: / for PerplexityBot will not stop Perplexity-User from fetching a page when a user names it. To restrict that path, you must block at the network layer by user agent and verified IP, not by robots.txt alone.

How do I verify what Perplexity actually does?

Do not trust the directive — trust your logs. Grep your access logs for both strings:

• grep -iE "perplexitybot|perplexity-user" access.log to isolate declared hits.
• Reverse-DNS each source IP and confirm it resolves to Perplexity's published ranges, then forward-confirm — the same verification you would run for GPTBot or Googlebot.
• Compare fetch timestamps against pages you disallowed. A hit on a blocked path from a declared agent is a policy violation worth documenting.

Because agents and IP ranges shift, treat verification as ongoing rather than one-time. If your goal is visibility, monitor which of your URLs Perplexity actually retrieves and cites — citation tracking surfaces the deep URLs Perplexity pulls, which is a more useful signal than whether a single crawl obeyed a rule. If your goal is exclusion, layered enforcement plus periodic log audits is the only reliable approach.