robots.txt

The file is the first thing a well-behaved crawler reads, and its grammar is stricter than it looks. A crawler finds the group whose user-agent line matches its product token, without regard to case, and obeys that group alone; only when no group names it does it fall back to the group for *^[1]. Within the group the longest matching path wins, and an allow beats an equal disallow^[1]. A site that adds a short group for one AI crawler has, often without meaning to, released that crawler from every rule of the catch-all group.

It controls crawling, not indexing. Google says the file is used mainly to avoid overloading a site and is not a mechanism for keeping a page out of Google: a blocked URL can still be indexed from links that point to it, and the way to keep a page out is noindex or a password^[2]. The standard adds that listing a path in the file makes it public, so the file is no place to hide anything^[1].

How the server answers matters as much as what the file says. Under RFC 9309 a 4xx answer lets the crawler access everything, while a server or network error means it must assume complete disallow^[1]. Google treats every 4xx except 429 as no file, stops crawling for the first 12 hours of 5xx errors, then works from its last good copy for up to 30 days, and in normal times caches the file for up to 24 hours^[3]. A file that times out can halt the crawl of a healthy site; one that answers 404 opens it.

Rules apply only to the host, protocol and port that serve them, so a file on example.com says nothing about shop.example.com^[3].

Across the web the file is used bluntly as often as finely. In QueryRev’s study, 6,294 of the 9,973 domains gave a robots.txt answer that could be read, and 190 of those, 3.0%, close the whole site in the catch-all group, the rules every crawler without a group of its own follows^[4]. Newer lines appear beside the rules: 164 of the 5,637 files read, 2.9%, carry a Content-Signal line stating preferences for search, AI input and AI training^[4].

Fetch the file as a crawler does, from each host and protocol the site serves, and record the status code, the redirect chain and the time it took. Then read it per product token: which group applies, and whether a given URL is allowed. Google reads at most 500 KiB of the file^[3], the least that RFC 9309 lets a crawler parse^[1].

Test the URLs that matter instead of reading the rules by eye: the pages you want crawled, the pages you want left alone, and the paths that sit between two rules. Server logs then show whether each crawler behaved as the file asks.

Watch the file as you would any release. A deploy that ships a staging file with Disallow: / closes the whole site, and it shows first as a fall in crawl requests in the logs, before rankings move.

Sample data: an invented site, to show the method.

A shop’s robots.txt keeps every crawler out of /checkout/ and /account/ in the catch-all group. Last month someone added a group for GPTBot with a single rule, to keep it out of /blog/drafts/. GPTBot now obeys that group alone, so /checkout/ and /account/ are open to it.

The fix is to repeat the shared rules inside the GPTBot group. Testing the three paths against both groups, before and after, shows the difference in a minute.