AI-Search Readiness of Top Sites What the top 9,973 sites say about themselves, and whether a crawler that runs no scripts can read it.

• Describe the organization in JSON-LD

  40.0%

  of 4,480 home pages read

  Counted

  1,793

  of 2,203 pages with schema.org JSON-LD

  81.4%

  An Organization, or one of its schema.org subtypes, anywhere in a JSON-LD block of the home page.

• Link outside profiles with sameAs

  28.6%

  of 4,480 home pages read

  Counted

  1,280

  of 1,793 pages describing an Organization

  71.4%

  The organization’s sameAs holds at least one web address outside the site’s own domain.

• Carry their main text in the HTML

  80.7%

  of 4,480 home pages read

  Counted

  3,615

  At least 150 words of main text in the HTML as served, before any script runs.

• Have a valid sitemap

  55.9%

  of 6,082 sites asked for a sitemap

  Counted

  3,401

  The sitemap the site names, else /sitemap.xml, is a sitemap with at least one address on the site’s own domain. 260 of the 6,342 sites read were not asked: their robots.txt keeps our crawler from the sitemap.

• Publish an llms.txt

  13.0%

  of 6,066 sites asked for an llms.txt

  Counted

  787

  of 6,342 sites read

  12.4%

  The crawler study’s rule: a text file at /llms.txt that starts with a Markdown title. 276 of the 6,342 sites read were not asked: their robots.txt keeps our crawler from /llms.txt.

• Have all five

  7.8%

  of 4,480 home pages read

  Counted

  350

  Organization in JSON-LD with sameAs, main text in the HTML, a valid sitemap and an llms.txt. A count, not a grade.

• 49.2%

  Any JSON-LD

  A block that parses as JSON with a schema.org context. 2,203 of 4,480 home pages read.

• 1.1%

  A block that does not parse

  Among pages with a JSON-LD script, at least one script that fails to parse. 24 of 2,222 pages with a JSON-LD script.

• 34.5%

  WebSite

  A WebSite node. 21.9% of the home pages read also describe a SearchAction. 1,544 of 4,480 home pages read.

• 3.6%

  FAQ

  A FAQPage node, in JSON-LD or microdata. 163 of 4,480 home pages read; 7.1% of 2,203 pages with schema.org JSON-LD.

• 0.8%

  Organization in microdata only

  An Organization type in microdata and none in JSON-LD. RDFa is not read. 37 of 4,480 home pages read.

• 88.8%

  Name and web address

  A name and an absolute web address. 1,593 of 1,793 pages describing an Organization; 35.6% of 4,480 home pages read.

• 86.8%

  Logo

  A logo as an address or as an image with an address. 1,556 of 1,793 pages describing an Organization.

• 5

  Profiles listed

  The median number of distinct sameAs addresses among the Organizations that list any (1,280); the quartiles are 3 and 6 (nearest rank).

• 15.3%

  Little text, no shell marks

  Under 150 words of main text, without the marks of an empty app shell. 685 of 4,480 home pages read.

• 4.0%

  Served as an empty app shell

  Over 5 KiB of HTML, under 50 words of main text, and an application root element or a notice to enable JavaScript. 180 of 4,480 home pages read.

• 1.4%

  Home page says noindex

  A robots meta tag or an X-Robots-Tag header with noindex and no agent prefix. 62 of 4,480 home pages read.

• 71.5%

  Text in the HTML, and open to AI search crawlers

  Text in the HTML, and robots.txt lets every covered crawler whose purpose is search or user fetch read the home page. 3,205 of 4,480 home pages read.

Home pages over 2 MiB, read as far as the cut and flagged: 49 of 4,480.

• 67.6%

  robots.txt lists a sitemap

  At least one Sitemap line in a robots.txt file. 3,845 of 5,691 sites with a robots.txt file; 60.6% of 6,342 sites read.

• 58.7%

  A sitemap index

  The valid file is a sitemapindex. Its children are not read. 2,142 of 3,651 sitemaps found.

Valid sitemaps cut at 5 MiB and counted as valid as far as read: 39 of 3,651 sitemaps found. Sites whose robots.txt keeps our crawler from /llms.txt were not asked for one (276). The trend of llms.txt over time is in the crawler study.

1. 1

   The panel

   The first 10,000 places of Tranco list N26XW, less the domains of companies that build language models: 9,973 domains.

   Details

   It is the panel of the crawler study. Tranco ranks registrable domains, combined from several sources over 30 days; list N26XW was created 14 October 2026. 2 places were not a valid domain and were left out, and 25 domains of companies that build language models are not requested at all. The list includes domains that serve no website, such as networks and APIs; they are counted as not read.

2. 2

   What was fetched

   robots.txt, the home page, /llms.txt and one sitemap, each only if robots.txt allows it. Nothing else.

   Details

   For each domain, in this order and one after the other: https://domain/robots.txt (the www. host instead when the domain did not connect); the home page; the robots.txt of the host the home page landed on when that is another host; /llms.txt; and the sitemap, which is the first Sitemap line of robots.txt on the same registrable domain, else /sitemap.xml. No inner page, no script run, no cookie, and no link of an llms.txt is followed. A redirect to another registrable domain is not followed. Requests identified themselves as QueryRevBot, with a page that says how to stop it; at most 16 domains were in flight, requests to one host were at least a second apart, a Crawl-delay was kept up to 10 seconds, and nothing more was asked of a host after a 429 or 503. A domain that failed for network reasons was tried once more at the end.

3. 3

   What counts as read

   6,342 of the 9,973 domains gave a robots.txt answer we could read; 4,480 of them gave a home page we could use.

   Details

   A home page is read when it answers 200 within the redirects with an HTML body that is not a bot check (a 202, or a page with a challenge’s marks) or a placeholder (a parking page or a default server page, by a fixed list of marks). A body over 2 MiB is read as far as the cut and flagged; when the part read holds under 150 words of main text, the page is not counted as read at all, because the cut may have hidden its text and its structured data. A page is decoded by its own character set. Sites that opted out of the study (0) count only as opted out. The causes of every domain not read are listed under Panel.

4. 4

   How structured data was read

   JSON-LD blocks that parse and name schema.org, walked to any depth; microdata types; RDFa is not read.

   Details

   A block of type application/ld+json counts when it parses as JSON, after a wrapping comment or CDATA is removed, with a top-level @context that names schema.org. The reader is bounded: at most 50 blocks, 512 KiB each and 1 MiB in all, depth 16 and 5,000 nodes; past a bound the rest is not read and the page is flagged. An Organization is any node whose @type is Organization or one of its schema.org subtypes, taken from the schema.org vocabulary at release 30.1. Only type names of that vocabulary are published; any other name is published as other. Organization names and sameAs addresses are read but never published: only counts and the kind of profile are.

5. 5

   How text was counted

   Words of main text in the HTML as served, at least 150 for text in the HTML.

   Details

   Main text is the text of main (or an element with role main) when there is one, else of the body without header, nav, footer and aside. Script, style, noscript, template and svg are never counted. Words are counted with Intl.Segmenter, so pages in Chinese or Japanese are counted too. A page of over 5 KiB with under 50 words and an application root element or a notice to enable JavaScript is an empty app shell; the rest under 150 words is little text.

6. 6

   Sitemaps and llms.txt

   A sitemap is valid when it is XML or text with addresses on the site’s domain; an llms.txt by the crawler study’s rule.

   Details

   A sitemap is valid when the file answered 2xx and is XML without a DOCTYPE or ENTITY whose root is urlset or sitemapindex with at least one loc that is an absolute web address on the site’s registrable domain, or a text sitemap of such addresses. A web page served at /sitemap.xml is not a sitemap. A file cut at 5 MiB counts as valid as far as read. An llms.txt counts when /llms.txt answers 200 with a text type, the body is not an HTML page or a robots.txt, and its first non-empty line is a Markdown title.

7. 7

   The network-failure retry

   1,004 sites whose first try ended in a network failure were asked again from the start, from a connection in the United States, on 24 October 2026.

   Details

   A network can fail to reach a site that answers from another. Every site whose result stood on a network failure (no answer, a timeout, a failed connection, an address refused as private or reserved, or a home page, sitemap or llms.txt that did not answer) was asked again from the start, as above, from a connection in the United States, on 24 October 2026, with the same limits and politeness. For each, the better of the two tries is kept: 294 of the 1,004 read more the second time. A name that did not resolve, a refusal (401 or 403), a bot check, a 429 and a 5xx were not a reason to ask again, because they are what the site answered.