What is llms.txt, and does your site need one?

llms.txt was proposed in September 2024 by Answer.AI: a Markdown file at /llms.txt that gives language models brief background on a site and links to the pages with more detail, so a model knows what to read when it assembles the context for a question about the site^[1]. The proposal, revised since, names the problem it answers: a web page wraps its information in navigation, adverts and scripts, and most sites are too large to fit in a model’s context window^[2].

The file is a proposal, published at llmstxt.org, not the work of a standards body. robots.txt, by contrast, is an IETF standard, RFC 9309, that tells crawlers which addresses they may fetch^[3]. The two are easy to confuse, because both sit at the root of a site and both are plain text, but they do different jobs.

The proposal puts the difference with a sitemap plainly: a sitemap lists every page for search engines, while an llms.txt offers a curated overview for language models^[2]. So an llms.txt is a reading list, not a permission. It cannot keep a crawler out, it cannot let one in, and it does not list every page of a site the way a sitemap does. It says, in a few lines a model can take in at once, where the answers are.

The format is plain Markdown, in a fixed order^[2]:

1. 1An H1 with the name of the site or the project. It is the only part the proposal requires.
2. 2A blockquote with a short summary of the site, holding the key facts a reader needs to make sense of the rest of the file.
3. 3Optionally, paragraphs or lists, but no headings, with more detail about the site and how to read the files it lists.
4. 4Optionally, sections under H2 headings, each a list of links written as Markdown: the page’s name, its address and, after a colon, a short note on what it holds.

The proposal also asks that pages worth reading offer a clean Markdown copy of themselves, so a model can read the text without the page around it. That copy lives at the page’s own address with .md added, or, for publishing tools that work that way, with .md in place of the page’s extension^[4].

To help agents find these files, the revised proposal recommends two standard link relations, written as HTML link elements or as an HTTP Link header: rel="alternate" with type="text/markdown" points from a page to its Markdown copy, and rel="describedby" points to the llms.txt that covers the page^[2]. The header form works for files that are not HTML, and can be added in the server or CDN configuration without changing any page.

The proposal was revised in August 2026, and two changes matter when you write a file today^[4]. A file in a subfolder, such as /docs/llms.txt, now covers the pages under that path, and the most specific file applies. And the section named Optional, which once told tools what they could leave out, no longer has a special meaning: it can still hold secondary links, as a convention.

Most of the confusion about the file comes from expecting it to do the job of another one. It does none of these:

• It does not stop a model being trained on your pages. Whether a crawler may fetch them is set in robots.txt, crawler by crawler.
• It does not give any crawler permission to fetch what robots.txt closes to it.
• It neither helps nor harms visibility or rankings in Google Search, which ignores it^[5].
• It does not replace a sitemap. A sitemap lists the addresses you want crawled; an llms.txt picks the few worth reading first.
• It does not make a weak page stronger. A file that points to thin or outdated pages hands a model thin or outdated answers.

Google says plainly that you do not need an llms.txt, or any similar file, to appear in Google Search or in its generative AI features, because Google Search itself does not use them. It adds that keeping one for other services or systems that do use such files is fine, and will neither help nor harm a site in Google Search^[5].

Chrome’s Lighthouse checks the file in its agentic browsing audits. A site without one is marked not applicable, because the file is optional, and a server error at /llms.txt is flagged^[6]. That is a check that the file is served correctly, not a sign that a search engine reads it.

The revised proposal describes how agents use the file: they read it or search it, then follow its links to the pages written for them. It names where that happens most: software documentation, where coding agents follow the file to API references and tutorials^[2].

For your own site, the server logs answer the question better than any article can: they show which user agents ask for /llms.txt, how often, and whether they then fetch the pages it lists.

QueryRev Research measures this every month in its State of AI crawler access study of the Tranco top 10,000 domains. In the October 2026 edition, 848 of the 6,030 sites asked for the file had one: 14.1%^[7].

Most of those files follow the format. Of the 848, 82.5% have a summary under the title and 87.1% have at least one section that lists links^[7]. Taking every part the study checks together, 59.0% of the files are complete, and the median file lists 38 links^[7].

The study also found a mismatch worth checking on your own site: 8.8% of the sites with an llms.txt keep at least one of the AI crawlers measured from reading it, in their own robots.txt^[7]. A file a crawler is told not to fetch does nothing for that crawler.

Publishing an llms.txt is a small task, and for Google Search it is not a ranking lever^[5]. Treat it as a small piece of technical hygiene inside generative engine optimization, not as the strategy. The kind of site you run decides how much it is worth.

What matters more for AI answers is on the pages themselves: whether crawlers may fetch them, whether they state facts plainly, and whether they answer the question asked. The file only points to that work.

1. 1Write the title and a summary of a sentence or two that says what the site is and who it is for, in the words you would want an answer to use.
2. 2Choose the pages a model should read first: product facts, pricing, policies, documentation. Link each by its canonical URL, never a redirect or a tracking variant.
3. 3Give every link a short note after a colon, so a reader knows what the page holds before opening it.
4. 4Serve the file at the root, as plain text with a 200 answer. A web page served at /llms.txt is not a file: the study counts one only when it is text that opens with a Markdown title^[7].
5. 5Check robots.txt: the crawlers you want to read the file must be allowed to fetch /llms.txt and the pages it lists. Which AI crawlers to allow is a separate decision, made crawler by crawler.
6. 6Update the file when the pages it lists move or change, and check its links on the same schedule as the rest of the site.

There is little to measure in the file itself. Check that it answers, that it parses as Markdown and that every link resolves. Then look in the server logs for requests to /llms.txt and to the pages it lists, by user agent, before and after you publish.

Whether it changed any answer is measured the way all GEO work is: on a fixed set of prompts, asked on a schedule, by whether the answers mention your brand and cite your pages. That is your AI search visibility, and an AI citation is its unit. Compare periods, not single days: answers vary from one run to the next.