AI Search
llms.txt
llms.txt is a proposed convention for a plain Markdown file at the root of a website, at /llms.txt, that gives large language models a short description of the site and a curated list of links to its most useful pages. It was proposed in 2024 and is not an adopted web standard.
- Industry term
By QueryRev ResearchPublished Updated5 min read
Why it matters.
A language model reading a site at answer time has limited room and must work through navigation, scripts and layout. The proposal behind llms.txt, published in September 2024, is to hand it a clean map instead[1]: a title, a short summary, and sections of links to the pages that hold the detail[2]. It is cheap to publish and easy to keep accurate.
It is not robots.txt. robots.txt follows the Robots Exclusion Protocol, published by the IETF in 2022, which tells crawlers which paths they may access[3]; even that is a way to manage crawling rather than to keep a page out of results, as Google notes[4]. An llms.txt grants and denies nothing.
Adoption is small and uneven. In QueryRev’s study of the top sites, 848 of the 6,030 sites asked for one publish an llms.txt, 14.1%, and 500 of those files, 59.0%, are complete: a title, a summary, at least one section of links, and every checked link answering[5].
At the time of writing, October 2026, no major AI search provider has publicly committed to using the file when choosing or ranking sources. Treat it as a low-cost courtesy, not a lever.
The file is written for two readers at once. A person can open it and see in a minute what the site offers; a program can parse it, because the proposal fixes its order: a title, a summary, optional notes, then sections of links. That plainness is most of its value.
How it is measured.
There is little to measure directly. Check that the file exists, is valid Markdown, lists live canonical URLs and matches the site as it is. Server logs show whether AI crawlers request it. Whether it changed any answer is measured the usual way: through citations and mentions on tracked prompts.
Check the two files against each other, and the links. Of the 848 sites with an llms.txt in the study, 75, 8.8%, keep at least one AI crawler from reading it in their own robots.txt, and of the 3,298 links checked in the files found, 92.6% answered with a page[5].
Look at the median as well as the share. In the study, the median file found was about 7 KB with 38 links[5]: a short document, not a copy of the site. A file far larger than that is often a sitemap in another format, which defeats the point of a curated map.
Measured by
- File is valid
- Links are live
- Requests in server logs
A worked example.
Sample data: an invented site, to show the method.
A SaaS documentation site publishes /llms.txt with a one-line summary and three sections, Docs, Pricing and Policies, each a short list of links with a sentence apiece. A dozen lines tell a model where things are.
Every link points to a canonical page that answers 200, and robots.txt lets the AI crawlers the site allows read /llms.txt itself. The file is checked again whenever a page it lists moves.
What to do, step by step.
Decide whether the site has pages a model would need a map to: documentation, policies, pricing, product facts. A small brochure site gains little.
Write the file in Markdown at /llms.txt: a title as the first line, a one-line summary as a quote, then sections of links with a sentence each.
List only canonical pages that answer 200, and keep the list short: the pages that answer the questions people ask about you.
Check robots.txt: if you allow an AI crawler to read your pages, do not disallow /llms.txt to it.
Serve the file as text, not as an HTML page, and check that every link answers.
Review it with each release that moves pages, not once a year.
Mistakes to avoid.
Expecting it to block or to rank. llms.txt has no power to keep crawlers out and no stated role in ranking. Access is set in robots.txt and on the server.
A web page at /llms.txt. Many sites answer every unknown address with a page of HTML and the status 200. A crawler then reads a web page, not a file.
Listing every URL. A map that copies the sitemap gives a model no help in choosing. Curate the few pages that matter.
Letting it go stale. Links to moved or deleted pages turn a map into a list of errors. Check them with each release.
Disallowing it by accident. A robots rule that closes a whole directory, or every path to a named AI crawler, can also close /llms.txt. Read the two files together after any change to either.
How QueryRev treats it.
QueryRev does not treat llms.txt as evidence of AI visibility, and it is not a decision type. What counts is whether answers cite you and describe you correctly, and that is earned on the pages themselves. Where a file would help, publishing it is a small technical task, not a strategy.
Questions people ask.
Do AI search engines use llms.txt?
No major AI search provider has publicly committed to using the file when choosing or ranking sources, as of October 2026. Some tools and documentation platforms read or generate it, which is why it is cheap to publish rather than a strategy.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which paths they may access and is an IETF standard. llms.txt is a proposed map of a site for language models, and it allows or blocks nothing.
Where does the file go?
At the root of the site, at /llms.txt, served as plain text. The proposal also suggests Markdown versions of individual pages at the same address with .md added.
Sources and citation.
Primary sources only: the search engines, the AI vendors, standards bodies and QueryRev’s own research. Each is cited in the text by its number.
/llms.txt: a proposal to provide information to help LLMs use websitesAnswer.AI ·
The /llms.txt file, v2llmstxt.org ·
Introduction to robots.txtGoogle Search Central ·
State of AI crawler access, October 2026QueryRev Research ·
Cite this definition
A suggested citation. Its link goes to the definition, whose anchor does not change.
QueryRev Research. “llms.txt.” QueryRev AI glossary, 18 October 2026. https://queryrev.com/glossary/llms-txt#definition
Related terms