Skip to content
Join the waitlist

Research · October 2026 edition

State of AI Crawler Access What the top 9,973 sites tell AI crawlers, and whether their llms.txt holds up.

14.2% of the sites we could read shut at least one AI crawler out by name in robots.txt. 14.1% of the sites we asked publish an llms.txt.

Get the dataHow we measured
Measured
15 October 2026
Panel
9,973 domains, Tranco list N26XW
Updated
A monthly pulse, a full report each quarter
Author
QueryRev Research

Crawlers covered, by what they are for

What each one does

  • Collects pages for AI training

    • GPTBotOpenAI
    • Google-ExtendedGoogle
    • Applebot-ExtendedApple
    • CCBotCommon Crawl
    • Meta-ExternalAgentMeta
    • BytespiderByteDance

  • Indexes pages for AI search

    • OAI-SearchBotOpenAI
    • PerplexityBotPerplexity
    • AmazonbotAmazon

  • Fetches a page when a user asks

    • ChatGPT-UserOpenAI
    • Perplexity-UserPerplexity

and the catch-all group, User-agent: *No other crawler was measured; “any AI crawler” on this page means any of these 11.

Key figures

The answer, on three denominators.

Studies of AI crawler blocking disagree mostly because they divide by different things. Every share on this page carries the mark of what it is a share of.

What a share is of

  • Sites read6,294 sites whose robots.txt answer we could read
  • Whole panelall 9,973 domains of the panel
  • With AI rules1,339 sites whose robots.txt names at least one AI crawler

For llms.txt, the sites read are the 6,030 that let us ask for one: 264 keep our crawler from /llms.txt in robots.txt.

  • Block at least one AI crawler by name

    14.2%

    of 6,294 sites read
    of the 9,973 in the panel
    9.0%
    of 1,339 with AI rules
    66.9%

    A group naming the crawler closes the whole site to it.

  • Block GPTBot by name

    9.2%

    of 6,294 sites read
    of the 9,973 in the panel
    5.8%
    of 1,339 with AI rules
    43.3%

    OpenAI’s crawler for model training.

  • Block every crawler with the catch-all rule

    3.0%

    of 6,294 sites read
    of the 9,973 in the panel
    1.9%
    of 1,339 with AI rules
    3.7%

    User-agent: * with Disallow: /. Counted apart from rules that name AI crawlers.

  • Publish an llms.txt

    14.1%

    of 6,030 sites asked for one
    of the 9,973 in the panel
    8.5%
    of 1,271 with AI rules asked
    19.7%

    A text file at /llms.txt that opens with a Markdown title; a web page served there is not counted.

Beside two other studies

Each pair is on one denominator. Their figures are from their own pages, read on the day given; ours are from this edition’s counts.

AI Crawler Check, robots.txt study published 5 October 2026: Tranco top 10,000, list of 4 October 2026. Read 16 October 2026.

  • GPTBot kept from the whole site, of the robots.txt files read

    Theirs
    14.3%710 of 4,962
    Ours
    13.0%730 of 5,637

  • The same, by a group that names GPTBot

    Theirs
    11.1%549 of 4,962
    Ours
    10.3%580 of 5,637

  • GPTBot kept from the whole site, of every domain on the list

    Theirs
    7.1%710 of 10,000
    Ours
    7.3%730 of 9,973

SEOmator, llms.txt adoption study published 13 September 2026: Cloudflare Radar top 100,000. Read 16 October 2026.

  • Publish an llms.txt, of every domain on the list

    Theirs
    8.6%8,598 of 100,000
    Ours
    8.5%848 of 9,973

  • Publish an llms.txt, of the domains that answered (theirs) or that we asked (ours)

    Theirs
    11.4%8,598 of 75,353
    Ours
    14.1%848 of 6,030
Where the rest of the difference comes from
  • What counts as blocked. We count a crawler as blocked when its whole site is closed and no Allow opens a path again. AI Crawler Check counts any rules that keep it from the home page, /. A site closed except for a few paths is blocked there and partly blocked here.
  • What counts as a file read. Both read robots.txt as RFC 9309 says. A web page, a bot check or a refusal is not a file in either; how each sorts the answers in between gives 4,962 files there and 5,637 here.
  • The list. Their Tranco list is of 4 October 2026, ours of 14 October 2026. We also leave out the 25 domains of companies that build language models.
  • What counts as an llms.txt. We count a file only when /llms.txt answers 200 with text that is not HTML and opens with a Markdown title. SEOmator counts a 200 whose body is not HTML, JSON or JavaScript, and checks the title apart. It found 20,171 domains answering 200 and 8,598 real files: counting every 200 would have overstated adoption 2.35×.
  • Who is asked. SEOmator’s panel of 100,000 domains reaches far smaller sites than our 9,973. Its second denominator is every domain that answered; ours is the sites whose robots.txt we read and that let us ask.

The panel

One fixed panel of 9,973 domains, read again every month.

The first 10,000 places of Tranco list N26XW, created 14 October 2026 from rankings between 15 September 2026 and 14 October 2026. Each reading measures the same domains, so a change is a change on the web, not in the list.

  1. A monthly pulse

    robots.txt and llms.txt read again on the same panel; what changed goes into the change log on this page.

  2. A full report each quarter

    Every figure again, by crawler, rank and ending, with dated data files.

  3. Within 48 hours of crawler news

    A re-run when a crawler or a common robots.txt template changes, added to the same change log.

This edition is the baseline. The per-domain file lists the 9,973 domains of the panel with what we read (a site that opts out is left out of it), so anyone can compare the next reading with this one.

By crawler

Training crawlers are blocked by name more than search crawlers: 7.9% against 4.8%.

Most often: Bytespider, CCBot and GPTBot. The colour is what a crawler is for. A solid bar is blocked: a group naming the crawler closes the whole site. An outlined bar is partly blocked.

How this was counted
For each crawler, the share of sites whose robots.txt has a group naming it that closes the whole site (blocked) or closes paths the catch-all group leaves open (partly blocked). Rules a crawler only inherits from the catch-all group are not counted here; they are shown on their own row, last. The figure for a purpose is the average of its crawlers’ shares of the sites read, blocked by name: 7.9% for AI training, 4.8% for AI search, 3.6% for user fetches.
AI crawlers blocked by name in robots.txt
Share of
AI trainingAI searchUser fetchBlockedPartly blocked

  1. Bytespider

    AI training · ByteDance

    9.5%+1.5% partly

  2. CCBot

    AI training · Common Crawl

    9.4%+2.4% partly

  3. GPTBot

    AI training · OpenAI

    9.2%+3.3% partly

  4. Google-Extended

    AI training · Google

    6.5%+2.5% partly

  5. Amazonbot

    AI search · Amazon

    6.5%+1.6% partly

  6. Meta-ExternalAgent

    AI training · Meta

    6.4%+2.0% partly

  7. Applebot-Extended

    AI training · Apple

    6.3%+1.8% partly

  8. PerplexityBot

    AI search · Perplexity

    5.1%+2.9% partly

  9. ChatGPT-User

    User fetch · OpenAI

    4.2%+3.2% partly

  10. Perplexity-User

    User fetch · Perplexity

    3.0%+1.3% partly

  11. OAI-SearchBot

    AI search · OpenAI

    2.7%+3.0% partly

  12. Every crawler (catch-all *)

    User-agent: *, the group any crawler without its own follows

    3.0%+1.1% partly

The table gives the counts. Blocked by any rule adds the sites whose catch-all group closes the whole site to a crawler that has no group of its own.

Show the numbers as a table
CrawlerPurposeVendorBlocked by namePartly blocked by nameNamed at allBlocked by any rule
BytespiderAI trainingByteDance59592742770
CCBotAI trainingCommon Crawl594152847763
GPTBotAI trainingOpenAI5802101,019730
Google-ExtendedAI trainingGoogle411157789571
AmazonbotAI searchAmazon406102596573
Meta-ExternalAgentAI trainingMeta401125600573
Applebot-ExtendedAI trainingApple394111590563
PerplexityBotAI searchPerplexity323181747479
ChatGPT-UserUser fetchOpenAI262199688416
Perplexity-UserUser fetchPerplexity18681367359
OAI-SearchBotAI searchOpenAI172188591326
Every crawler (catch-all *)19069190

llms.txt

14.1% of the sites we asked publish an llms.txt. 59.0% of those files are complete.

A 200 answer is not enough: many sites answer every address with a web page. We count a file only when it is text, not HTML, and opens with a Markdown title.

How this was counted
An llms.txt is counted only when /llms.txt answers 200 with a text file whose first non-empty line is a Markdown title (# Name); an HTML page, or a robots.txt, served at that address is not a file. We then read its structure as llmstxt.org describes it and checked up to five of the links it lists on the site’s own host.
Median file
7 KB
Median file
38 links

264 sites keep our crawler from /llms.txt in their robots.txt; we did not ask for theirs, so adoption is a share of the 6,030 sites we asked.

How complete llms.txt files are · 848 files found

  1. A file with a title

    100.0%

  2. A summary under the title

    82.5%

  3. At least one section that lists links

    87.1%

  4. Every link checked answers

    84.9%

  5. All of these

    59.0%

Shares of the 848 files found. Links checked: 3,298, of which 92.6% answered with a page. Every link answers is counted over the 684 files with at least one link checked.

Show the numbers as a table
A file with a title848100.0%
A summary under the title70082.5%
At least one section that lists links73987.1%
Every link checked answers58184.9%
All of these50059.0%

robots.txt and llms.txt

8.8% of the sites with an llms.txt keep an AI crawler from reading it.

An llms.txt is written for AI tools to read; robots.txt says which of them may. On the same site the two do not always agree.

  • 8.8%

    llms.txt sites that keep at least one AI crawler from it

    75 of 848: robots.txt disallows /llms.txt to at least one crawler measured.

  • 0.2%

    llms.txt sites that keep every AI crawler from it

    2 of 848: none of the crawlers measured may read the file.

  • 20

    robots.txt mentions llms.txt, but there is none

    Of 125 robots.txt files that name llms.txt, these had no llms.txt we could find.

llms.txt adoption, by what robots.txt says

  1. Block an AI crawler by name

    8.1%

  2. Block none by name

    15.1%

Shares of the sites in each group that we asked for an llms.txt: 862 block an AI crawler by name, 5,168 block none.

Show the numbers as a table
Block an AI crawler by name70 of 8628.1%
Block none by name778 of 5,16815.1%

Who wrote the rules

2.9% of robots.txt files carry a Content-Signal line.

Some robots.txt files are not written by the site. We looked for fixed signatures, each with one rule, in the 5,637 robots.txt files we read.

  • 2.9%

    Use a Content-Signal line

    164 of 5,637 files. A line that starts Content-Signal:, which states search, ai-input and ai-train preferences (contentsignals.org).

  • 1.0%

    Carry the Content Signals Policy preamble

    59 of 5,637 files. The comment that opens with “As a condition of accessing this website, you agree to abide by the following”, which Cloudflare writes above a managed file. The text is free to reuse, so it shows the template, not who served the file.

  • 0.4%

    Carry Cloudflare’s managed block

    22 of 5,637 files. The comment “BEGIN Cloudflare Managed content”, which opens the rules Cloudflare writes when a site turns its managed robots.txt on.

By rank

Places 1 to 1,000: 17.6% block an AI crawler by name. The rest: 13.9%.

The same figures for the top of the panel and for the rest, each as a share of the sites read in that band; llms.txt as a share of the sites we asked for one.

By place in the panel
Places 1 to 1,000 (590 read)Places 1,001 to 10,000 (5,704 read)

  1. Block an AI crawler by name

    17.6%

    13.9%

  2. Block GPTBot by name

    10.2%

    9.1%

  3. Block every crawler (*)

    6.8%

    2.6%

  4. Publish an llms.txt

    19.1%

    13.6%

Show the numbers as a table
Places 1 to 1,000Places 1,001 to 10,000
Block an AI crawler by name17.6% (104)13.9% (792)
Block GPTBot by name10.2% (60)9.1% (520)
Block every crawler (*)6.8% (40)2.6% (150)
Publish an llms.txt19.1% (103)13.6% (745)

By country-code domain

From 44.6% on .it to 3.7% on .cn.

The share that blocks an AI crawler by name, for each country-code ending (.de, .co.uk, .com.br) with at least 50 sites read. An ending is not a country: some, such as .io or .co, are used worldwide.

Block an AI crawler by name, by country-code domain

  1. .it

    56 read

    44.6%

  2. .fr

    79 read

    38.0%

  3. .uk

    57 read

    33.3%

  4. .de

    139 read

    19.4%

  5. .br

    63 read

    15.9%

  6. .jp

    123 read

    15.4%

  7. .in

    57 read

    10.5%

  8. .tv

    54 read

    7.4%

  9. .ru

    195 read

    6.2%

  10. .io

    116 read

    4.3%

  11. .cn

    54 read

    3.7%

Show the numbers as a table
EndingSites readBlock an AI crawler by nameBlock every crawler (*)Publish an llms.txt
.ru1956.2%1.0%12.0%
.de13919.4%1.4%9.7%
.jp12315.4%0.0%5.9%
.io1164.3%6.0%30.3%
.fr7938.0%5.1%4.0%
.br6315.9%0.0%20.6%
.in5710.5%3.5%14.5%
.uk5733.3%7.0%7.7%
.it5644.6%0.0%5.6%
.cn543.7%0.0%1.9%
.tv547.4%3.7%0.0%

Method

How we measured, and what it cannot tell you.

One run, from one place, with our own crawler, which reads robots.txt as the standard says (RFC 9309).

  1. The panel

    The first 10,000 places of Tranco list N26XW, less the domains of companies that build language models: 9,973 domains.

    Details

    Tranco ranks registrable domains, combined from several sources over 30 days; list N26XW was created 14 October 2026. 2 places were not a valid domain and were left out. We also leave out the domains of companies that build language models (25 domains), so the study measures how the rest of the web treats their crawlers. The list includes domains that serve no website, such as networks and APIs.

  2. What was fetched

    robots.txt, then /llms.txt unless robots.txt keeps us from it, then up to five links an llms.txt lists. Nothing else.

    Details

    For each domain, https://domain/robots.txt; when the domain did not resolve, or the connection to it failed or timed out, the www. host instead, if it answered (319 sites). Then /llms.txt on the same host, unless robots.txt keeps our crawler from it, and up to five links listed in a found llms.txt, on the site’s own host and allowed to our crawler, without reading their pages. Requests identified themselves as QueryRevBot, with a page that says how to stop it; at most 16 were in flight, one at a time per site, with a 10-second limit and redirects followed only to public addresses. Names were looked up through public resolvers. A domain that failed for network reasons was tried once more at the end.

  3. How a rule was read

    Blocked: the whole site closed to the crawler. Partly blocked: some paths closed beyond the catch-all group’s.

    Details

    The group that applies to a crawler is the one naming its product token, any case; failing that, the catch-all group. Within it the longest matching rule wins and Allow wins a tie. Lines other than rules, such as Sitemap, Crawl-delay or Content-Signal, do not split a group. Blocked means a rule for the whole site disallows / and no narrower Allow opens anything; partly blocked means a group naming the crawler closes paths that the catch-all group leaves open, closes the whole site with Allow exceptions, or closes only the home page (Disallow: /$). Paths a group repeats from the catch-all, such as /search, do not count. A 404 means no restrictions.

  4. What counts as an llms.txt

    A 200 answer is not enough. A web page served at /llms.txt, often with status 200 (a soft 404), is not counted.

    Details

    We count an llms.txt only when /llms.txt answers 200 with a text type, the body is not an HTML page and not a robots.txt served at that address, and its first non-empty line is a Markdown title (# Name), the one part llmstxt.org requires. Many sites answer every address with a web page; counting every 200 would count those pages as files.

  5. What counts as read

    6,294 of the 9,973 domains (63.1%) gave a robots.txt answer we could read.

    Details

    Of those, 5,637 had a file and 657 had none. A bot check or a web page served in place of the file, or a 202 answer, is not a file we could read. 0 sites opted out of the study and count only as opted out. The other 3,679 were not read: their counts by cause are below.

The crawlers measured

These 11 product tokens of 8 vendors, and the catch-all group. Other crawlers were not measured, and every “any AI crawler” figure means any of these.

  • Collects pages for AI training

    GPTBot
    OpenAI: Collects pages for model training
    Google-Extended
    Google: Opts pages out of Gemini training and grounding
    Applebot-Extended
    Apple: Opts pages out of Apple model training
    CCBot
    Common Crawl: Builds the open web archive used in many training sets
    Meta-ExternalAgent
    Meta: Collects pages for AI training and products
    Bytespider
    ByteDance: ByteDance crawler; purpose not documented

  • Indexes pages for AI search

    OAI-SearchBot
    OpenAI: Indexes pages for ChatGPT search
    PerplexityBot
    Perplexity: Indexes pages for Perplexity answers
    Amazonbot
    Amazon: Crawls for Amazon products and AI

  • Fetches a page when a user asks

    ChatGPT-User
    OpenAI: Fetches a page a user asks about
    Perplexity-User
    Perplexity: Fetches a page a user asks about

Domains not read, by cause

The name did not resolve1,38513.9%
Refused our crawler (401 or 403)1,06510.7%
A bot check or a web page, not the file4534.5%
No answer within 10 seconds3833.8%
Connection or TLS failed2412.4%
Server error (5xx)590.6%
Resolved to a private or reserved address520.5%
Redirects that could not be followed210.2%
Asked us to slow down (429)200.2%

Limits

  • robots.txt is a request, not an enforcement. A site may block a crawler at its CDN or firewall without saying so in robots.txt, or allow it in robots.txt and refuse it there; we read what the file says, not what the server does.
  • A site that refuses our crawler, or serves a bot check in place of robots.txt, is counted as not read, not as having no file.
  • We keep to our own robots rules, so a site whose robots.txt keeps our crawler from /llms.txt, every site that closes itself to all crawlers included, was not asked for one. The llms.txt figures, and their comparison with robots.txt, cover the sites that let us read it.
  • One vantage point: a site that answers differently by country or by network was measured as it answered us.
  • A rule naming a crawler is counted for that crawler only; similar tokens of the same vendor are separate rows or not measured.

Other studies, and how this one differs

This study keeps one panel and reads it again every month with a change log; reads robots.txt and llms.txt of the same site against each other; measures how complete an llms.txt is, not only whether it exists; looks for rules written by a template; gives each figure on three denominators; and keeps the catch-all group apart from rules that name AI crawlers.

Data

Download, cite or embed it.

The data is free to use under CC BY 4.0: copy, share and adapt it, with a credit and a link to this page.

The same data, mirrored

The same files under the same licence, on three research data hosts. The DOI is the dataset’s permanent address.

DOI10.5281/zenodo.23424275

Licence: CC BY 4.0

Cite this study

QueryRev Research. “State of AI crawler access, October 2026 (dataset).” QueryRev AI, 15 October 2026. https://queryrev.com/research/ai-crawler-access. Dataset: https://doi.org/10.5281/zenodo.23424275

The panel comes from Tranco

Le Pochat et al., “Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation,” NDSS 2019. List N26XW: https://tranco-list.eu/list/N26XW

Embed this chart

An image of the chart by crawler, with a link back. No script: paste it where your page takes HTML.

Share of top sites that block each AI crawler by name in robots.txt, October 2026. Source: QueryRev Research.
<a href="https://queryrev.com/research/ai-crawler-access#by-agent"><img src="https://queryrev.com/data/ai-crawler-access/2026-10/block-rates.svg" width="720" alt="Share of top sites that block each AI crawler by name in robots.txt, October 2026. Source: QueryRev Research."></a>
<p>Source: <a href="https://queryrev.com/research/ai-crawler-access">State of AI Crawler Access, QueryRev Research</a> (CC BY 4.0)</p>

Questions

Does blocking an AI crawler in robots.txt stop it?

Not by itself. robots.txt is a request that well-behaved crawlers follow; it does not refuse any request. Sites that want to refuse a crawler do it at their server, CDN or firewall, which this study does not see.

Why do studies of AI crawler blocking give different numbers?

Mostly because they divide by different things: every domain on a list, the domains that answered, or the domains with a robots.txt file. This study gives each figure on three denominators so it can be compared with any of them, and puts its own figures on the denominators of two other studies under Key figures.

Why is the catch-all rule counted separately?

A site that blocks every crawler with User-agent: * is not making a choice about AI crawlers. Counting it as blocking each AI crawler would mix two decisions, so it has its own figure.

How do I keep my site out of this study?

Add User-agent: QueryRevBot with Disallow: / to your robots.txt. We still read robots.txt itself, as every crawler must, but nothing else, and your site is left out of the per-domain data and counted only as opted out.

Keep exploring

Early access

Be first to see what your SEO really earns.

Early access opens in stages. For in-house teams and agencies: leave your work email, we write when your place comes up, and nothing is charged.

optional

optional

You workoptional

We use these details only to tell you about early access, as our Privacy Policy describes. Every email has a link to leave the list.

Next: confirm your email with the link we send. We write when your place comes up, and nothing is charged.