User Agent

HTTP uses the term broadly: any client program that starts a request, from a web browser to a spider, a command-line tool or a household appliance^[1]. The User-Agent header carries one or more product identifiers, each a name with an optional version, listed in decreasing order of significance^[1]. OpenAI’s training crawler, for example, sends a string that ends compatible; GPTBot/1.4 and the address of its documentation^[3].

Robots.txt matches a shorter name. RFC 9309 calls it the product token: letters, underscores and hyphens only, and it should be a substring of the User-Agent header, so a crawler that sends Mozilla/5.0 (compatible; ExampleBot/0.1) obeys the group User-agent: ExampleBot^[2]. The match ignores case, groups that name the same token are combined, and a crawler with no group of its own obeys the * group^[2].

A token is not always a crawler. Google-Extended is read in robots.txt but never sent: Google crawls with its usual agents and uses the token only as a control^[4]. The reverse also exists. Agents that fetch on a user’s request may not follow robots.txt at all, as OpenAI says of ChatGPT-User^[3] and Google says of its user-triggered fetchers^[5].

The header proves nothing on its own. HTTP asks implementations not to use other products’ tokens to claim compatibility, and notes that one user agent can masquerade as another^[1]. Google’s answer is verification: a reverse DNS lookup on the requesting address, or a match against the IP ranges it publishes for each kind of crawler and fetcher^[5].

Every agent a file does not name inherits its catch-all rules. In QueryRev’s study, the catch-all group closes part of the site on 1.1% of the sites whose robots.txt could be read^[6], rules that bind every user agent without a group of its own.

Read the logs by user agent, then verify. Group the requests by product token and, for each crawler that matters, check the source address against the vendor’s published list or a reverse DNS lookup. Requests that copy a known name from other addresses are counted apart.

Test the rules token by token. For each user agent you care about, find the group of robots.txt that applies to it and test the URLs that matter against that group instead of reading the file by eye; robots.txt sets out how groups and errors are read.

Watch what the server sends back to each agent. Firewalls, CDNs and bot settings decide by user agent too, so a crawler that the file allows can still get a 403 or a challenge page. The status codes by agent show it.

Sample data: an invented site, to show the method.

A shop reads one day of logs and finds 3,400 requests whose user agent contains Googlebot. A reverse DNS lookup and Google’s published ranges confirm 2,950 of them. The other 450 come from a hosting provider’s addresses and fetch the shop’s internal search pages at a steady rate.

The 450 are a scraper using Googlebot’s name. The shop blocks them by address at the firewall, not by user agent, because a rule against the name would refuse the real crawler too.