Free tool
llms.txt Generator and Checker See what AI crawlers may read, and write the file that guides them.
Enter your site. The checker reads its robots.txt, llms.txt and llms-full.txt, tells you which AI crawlers on our list may read your pages, and checks your llms.txt against the llmstxt.org format. The generator drafts an llms.txt from your own pages.
- Checks three public files
- Your address goes to our server
- Free, no sign-up
Check or generate your llms.txt
How it works
- Enter your siteA domain or a page address. A domain without https:// is read as https.
- We read public filesOur server reads robots.txt, llms.txt and llms-full.txt from your site. The generator also reads your sitemap and up to 10 of your pages.
- Fix what it findsEach finding says what it means and gives you a snippet to copy into robots.txt or llms.txt.
Your report appears here
Pass, warning and fail rows for robots.txt and llms.txt, the AI crawlers on our list, and snippets to copy.
What the checker reads
Three files at your site’s address: /robots.txt, /llms.txt and /llms-full.txt. Nothing else. It does not read your pages or follow the links inside your llms.txt.
For robots.txt it reads the rules the way the standard (RFC 9309) says crawlers do: the group that names a crawler applies to it, otherwise the * group, and the longest matching rule wins. For each crawler it shows the line that decides your home page and the page you entered.
For llms.txt it first checks that the answer is a file at all. A site that answers every address with a web page returns 200 for /llms.txt too, so a 200 alone is not counted. Then it checks the structure llmstxt.org describes: one H1 with the site’s name, a summary blockquote, and H2 sections that list links.
How the generator drafts a file
It reads your robots.txt, then one sitemap (the first Sitemap line, or /sitemap.xml). When that sitemap is an index of other sitemaps, it reads the first one on your site, and goes no further. It takes up to 30 addresses on your site, and reads your home page and up to 9 more of them: 10 pages at most.
It obeys your robots.txt for its own name, QueryRev-AIReadiness, waits at least a second between two requests to your site, and stops if your site answers 429 or 503.
Each page’s title and meta description are used as written, and pages are grouped into sections by the first part of their address (/blog/… under Blog). No AI model writes or rewrites anything. Read the draft and edit it before you publish it.
Limits and what is stored
Each file is read with a time limit (5 seconds to connect, 10 seconds a file) and a size limit (500 KiB for robots.txt, 1 MiB for llms files). Private and reserved network addresses are never read, and every redirect is checked again.
You can run the tool 10 times a minute and 20 times a day, and one site can be checked 30 times an hour. To count that we keep counters under a one-way keyed hash of your network address and of the site’s name, for about a day. The report is not stored, and a refused check is logged with its reason and the site’s host name only.
Which crawlers it covers
It checks the AI crawlers on its list, not every crawler there is. Our study of AI crawler access uses the same list. Read the study
Keep exploring