PerplexityBot

It is the door to one engine’s answers. Perplexity documents PerplexityBot as designed to surface and link websites in its search results, recommends allowing it in robots.txt and permitting requests from its published IP ranges for a site to appear there, and says it is not used to crawl content for AI foundation models^[1]. A rule against it is a choice about Perplexity’s results, not about model training.

A second agent handles questions in the moment. Perplexity-User may visit a page when a user asks Perplexity a question, to help give an accurate answer and link the page in the response, and because a user asked for the fetch it generally ignores robots.txt rules^[1]. OpenAI draws the same line between OAI-SearchBot and ChatGPT-User^[3]. The group for PerplexityBot decides whether a page is in the index Perplexity’s answers draw on; it does not stop every visit from Perplexity.

Firewalls decide as often as the file. Perplexity’s documentation gives rules for web application firewalls such as Cloudflare’s and AWS’s that allow its agents by user agent and source address together, and asks that the address lists be refreshed from its JSON endpoints, because they change^[1]. A crawler that robots.txt allows and the edge refuses is blocked all the same.

The name follows the rules of the protocol. A crawler finds its group by matching its product token against the user-agent lines without regard to case, and with no group of its own it obeys the catch-all group^[2]. Perplexity says its systems may take up to 24 hours to reflect a change^[1].

Most sites that write AI rules choose between crawlers rather than close the door on all of them. In QueryRev’s study, PerplexityBot is blocked by name on 323 of the 6,294 sites whose robots.txt could be read, 5.1%, while only 3.2% of the 1,339 sites whose robots.txt names an AI crawler block every one of the crawlers covered^[4].

In the server logs, by user agent and address. Search for PerplexityBot and Perplexity-User, keep the requests whose source address is in the published list for that agent, and read the status each one got. A 200 means the page was served; a 403 or a challenge page means something in front of the site refused it.

In the file, host by host: which group applies to the token, and whether the pages you want in Perplexity’s answers are allowed. Then in the answers: ask your tracked prompts in Perplexity and record which of your pages it cites. A page it may crawl and never cites is a content question; a page it cannot crawl is an access one.

Compare the two agents. PerplexityBot fetches on Perplexity’s schedule and Perplexity-User when a reader asks, so many user fetches of a page that the crawler never reaches can point to a block on the crawler while readers still ask about the page.

Sample data: an invented site, to show the method.

A software company finds that Perplexity cites a review site’s summary of its pricing, not its own pricing page. Its robots.txt allows PerplexityBot. A week of logs shows 212 PerplexityBot requests, every one answered 403 by a firewall rule that challenges unverified bots, and 64 Perplexity-User requests served normally.

The firewall rule is changed to allow PerplexityBot from Perplexity’s published addresses only. Two weeks later the pricing page has been crawled and served, and the tracked prompts are read again to see whether it is cited.