GPTBot, OAI-SearchBot and robots.txt: allow or block AI crawlers

OpenAI documents its crawlers and user agents on one page^[1]. Three of them, GPTBot, OAI-SearchBot and ChatGPT-User, decide how a site works with ChatGPT search, model training and requests people make in ChatGPT. Each setting is independent of the others: a site can allow OAI-SearchBot to appear in search results while it disallows GPTBot to say its pages should not be used for training^[1]. A fourth agent, OAI-AdsBot, only visits pages submitted as ads on ChatGPT.

OpenAI adds that, for search results, a change to robots.txt can take about 24 hours to reach its systems, and it publishes the IP addresses each agent uses, so a request that claims one of these names can be checked^[1].

The consequence is the one most people search for: blocking GPTBot does not take a site out of ChatGPT search. Only a rule for OAI-SearchBot does that. And a rule for GPTBot alone does not stop a page being fetched when a person asks ChatGPT to read it.

Other vendors split the same jobs in their own ways. Some run a crawler of their own for training, and some for search, such as PerplexityBot. Others offer a product token, such as Google-Extended: a name you write in robots.txt that no crawler sends, which says how the pages their ordinary crawler fetches may be used.

This guide covers these crawlers and tokens. Others exist, and vendors add and rename them; each vendor’s own page is the reference for its current names.

robots.txt is a plain text file at the root of each host, standardised as RFC 9309^[8]. It holds groups. Each group starts with one or more User-agent lines and lists Allow and Disallow rules for paths. Five rules of reading cause most surprises:

• A crawler obeys the group that names it and ignores the rest. Only a crawler with no group of its own falls back to the group for User-agent: *^[8].
• So a group for GPTBot that says only Disallow: /private/ lets GPTBot fetch everything else, even paths your User-agent: * group closes. Repeat in every named group the rules that should hold for every crawler.
• Within a group, the most specific rule wins: the one with the longest matching path. When an Allow and a Disallow rule match equally, the Allow rule should be used^[8].
• The file covers one host and one protocol: shop.example and www.shop.example each need their own. Google reads at most 500 KiB of it and ignores the rest^[9].
• robots.txt is a request, not a lock. The standard says its rules are not a form of access authorization^[8]. A crawler that ignores them, or a fetch made on a person’s behalf, is stopped only by the server: a login, or a firewall rule.

Rules for a product token, such as Google-Extended or Applebot-Extended, decide how the pages the vendor’s ordinary crawler fetches may be used. They do not stop that crawler, and they do not take a site out of the vendor’s search^[3][4].

Google’s AI features in Search, such as AI Overviews, draw on the pages Google crawls for Search, so Google-Extended does not control them. Their controls are the ones of Search itself: a page must be indexed and allowed a snippet to be shown, and nosnippet, data-nosnippet, max-snippet and noindex limit what appears^[10].

QueryRev Research reads the robots.txt of the Tranco top 10,000 domains every month for its State of AI crawler access study. In the October 2026 edition, 6,294 sites gave an answer that could be read, and 14.2% of them shut at least one AI crawler out of the whole site by name^[11].

The study measures two crawlers this guide does not cover: Bytespider, ByteDance’s crawler, whose purpose is not documented and which the study counts with the training crawlers, and Amazonbot, which it counts with AI search. The table shows all eleven, as the study classes them.

The pattern is clear: sites refuse training more often than search. Across those eleven, the crawlers that collect pages for training are blocked by name on 7.9% of the sites on average, those that index pages for AI search on 4.8%^[11]. Far fewer sites shut out OAI-SearchBot, at 2.7%, than GPTBot, at 9.2%^[11]. And only 0.7% shut out every crawler the study covers^[11].

Three questions settle most robots.txt files. Answer them vendor by vendor, because each vendor splits its crawlers differently.

1. 1Do you want your pages found and cited in AI search answers? Then allow the search crawlers, OAI-SearchBot and PerplexityBot: OpenAI and Perplexity both recommend it for a site that wants to appear in their search results^[1][2]. A search crawler that may not fetch a page cannot read what it says, and a page it has not read cannot be quoted as an AI citation.
2. 2Do you accept that your pages may be used to train models? If not, disallow the training crawlers and tokens: GPTBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. By their vendors’ own documentation, these rules do not take your pages out of OpenAI’s, Google’s or Apple’s search^[1][3][4].
3. 3Do you need to stop fetches a person asks for? robots.txt is the wrong tool for that: OpenAI, Perplexity and Meta all say their user-started agents may not follow it^[1][2][6]. If a section must stay private, put it behind a login, not behind a Disallow line.

Every block has a cost to weigh: a crawler you block cannot read the pages you later want it to read, such as a new pricing page, until the rule changes.

Whatever you choose, write the reason down beside the rule. A robots.txt outlives the person who wrote it, and a rule nobody can explain is the hardest to change safely.

Each file below is complete for the crawlers it names. Keep your own rules for User-agent: * as they are, and repeat in every named group any path that should stay closed to every crawler, such as an account area.

# No named groups: every crawler, AI crawlers included,# follows the rules of User-agent: *User-agent: *Disallow: /account/ Sitemap: https://www.shop.example/sitemap.xml

# GPTBot reads only its own group: it is closed out.# OAI-SearchBot has no group, so it follows the rules of *.User-agent: GPTBotDisallow: / User-agent: *Disallow: /account/ Sitemap: https://www.shop.example/sitemap.xml

# Search crawlers: the same paths as everyone else.User-agent: OAI-SearchBotUser-agent: PerplexityBotDisallow: /account/ # Training crawlers and tokens: closed out.User-agent: GPTBotUser-agent: Google-ExtendedUser-agent: Applebot-ExtendedUser-agent: CCBotUser-agent: Meta-ExternalAgentDisallow: / User-agent: *Disallow: /account/ Sitemap: https://www.shop.example/sitemap.xml

A group may start with several User-agent lines, as the last file does: they share its rules^[8]. Search crawlers you do not name follow the group for User-agent: *, so the last file keeps them where they are.

1. 1Open /robots.txt on every host you serve, with and without www. It should answer 200 with plain text.
2. 2Read each named group as its crawler would: only its own group counts. Make sure the paths you closed for User-agent: * are closed there too.
3. 3Search your server logs for each user agent and confirm what it fetched after the change. OpenAI, Perplexity, Common Crawl and Apple publish the addresses or host names their crawlers use, so a real request can be told from one that only borrows the name^[1][2][5][4].
4. 4Check your CDN and firewall. A bot rule there can refuse a crawler that robots.txt allows; Google asks that crawling be allowed by any CDN or hosting infrastructure as well, and Perplexity explains how to let its crawlers through a firewall^[10][2].
5. 5Give the change a day: OpenAI and Perplexity both say an update can take up to about 24 hours to reach their systems^[1][2].
6. 6Then watch the result where it shows: which AI answers cite your pages, prompt by prompt, over the weeks that follow.

If you also publish an llms.txt, the crawlers you allow must be able to fetch it too. What the file is, and whether your site needs one, is the subject of its own guide.