AI Crawler

The three jobs carry different stakes, and the vendors document them as separate agents. OpenAI runs GPTBot for content that may be used in training, OAI-SearchBot to surface sites in ChatGPT’s search features and ChatGPT-User for pages a person asks about, and states that each robots.txt setting is independent of the others^[1]. Perplexity draws the same line between PerplexityBot, which surfaces and links sites in its search results, and Perplexity-User, which visits a page to help answer a question^[2]. Refusing a training crawler asks that content stay out of future training; refusing a search crawler keeps the site out of that engine’s answers.

Fetchers that act for a user sit partly outside robots.txt. OpenAI says that because ChatGPT-User acts on a user’s request, robots.txt rules may not apply, and Perplexity says Perplexity-User generally ignores them^[1][2]. A site that wants to refuse those requests has to do it at the server or the firewall, and gives up the reader who asked about the page.

Some tokens are not crawlers at all. Google-Extended has no user agent string of its own: Google crawls with its usual agents and reads the token only as a control over whether content may be used to train Gemini models and to ground answers in Gemini Apps, and it has no effect on inclusion or ranking in Google Search^[3]. Google’s AI features in Search, AI Overviews and AI Mode, follow the robots.txt rules for Googlebot^[4].

Sites treat the crawlers very differently. In QueryRev’s study of AI crawler access, the three most often blocked by name are Bytespider at 9.5% of the sites read, CCBot at 9.4% and GPTBot at 9.2%^[5]. The most blocked of them is the one whose purpose the study found no documentation for: a rule is written against a name, so it pays to know what the name does.

In two places: the rules and the requests. For the rules, read robots.txt on every host the site serves and work out, for each product token, which group applies and which paths it closes. For the requests, read the server logs by user agent: which crawler fetched which URLs, how often and with what status. A crawler that robots.txt allows and a firewall refuses with 403 is blocked all the same, which is why Google’s checklist for its AI features asks that crawling be allowed in robots.txt and by any CDN or hosting infrastructure^[4].

Verify before you count. A user agent string is easy to copy, so OpenAI and Perplexity publish the IP ranges their agents use, and Perplexity recommends matching both the user agent and the address^[1][2]. A request that calls itself GPTBot from an address outside OpenAI’s list is not GPTBot.

Read the rule that applies, not only the one that names the crawler. A crawler with no group of its own follows the catch-all group, so in QueryRev’s study the share of sites that keep GPTBot from the whole site rises from 9.2% by a group naming it to 11.6% by whichever group applies^[5].

Sample data: an invented site, to show the method.

A furniture retailer reads one week of server logs for the AI agents it cares about. GPTBot, ChatGPT-User and PerplexityBot fetched pages and got 200. OAI-SearchBot got 403 on every request: a bot rule at the CDN, added during a scraping incident, that nobody connected with search.

A fifth agent called itself GPTBot but came from an address outside OpenAI’s published ranges, so it is set apart as unverified. One fix follows: allow OAI-SearchBot at the CDN, then read the logs again a week later.