Robots.txt vs llms.txt: Different Jobs, and What 47 Real llms.txt Files Contain

This post contains affiliate links. If you buy something through one of them, Veravix may earn a commission at no extra cost to you.

Robots.txt and llms.txt do different jobs, and only one of them can say no. Robots.txt is a set of access rules that crawlers are expected to read before they fetch your pages. Llms.txt is an optional Markdown file that points AI tools at the pages you would most like them to read. It cannot block anything, it is not part of any formal standard, and Google’s own documentation says Google Search ignores it. If your question is “how do I keep AI crawlers out”, the answer is robots.txt. If your question is “should I also publish a reading guide for AI tools”, the answer is “it costs little, and the evidence that it does anything is thin”.

I wanted to see what people do with the second file, not just what advice sites say about it. On 11 November 2026 I requested /llms.txt from 69 well-known sites and graded every file I got back against the published format. About two thirds of the sites serve one. Roughly one in four of those files is not a reading guide for a website at all, and most of the links inside the rest lead to ordinary HTML pages.

Two files, two jobs

	robots.txt	llms.txt
Status	Formal internet standard, RFC 9309	Community proposal, written in 2024 and revised in August 2026
Purpose	Tell crawlers which paths they may fetch	Give an AI tool a short, curated map of your site
Who reads it	Crawlers, before they fetch anything else	Tools that choose to look for it, when a user asks about your site
Can it block?	It asks crawlers to stay out, and honest ones comply	No. It has no rules, only links and descriptions
Format	Plain text, User-agent and Disallow lines	Markdown: a title, a one-line summary, then lists of links

The proposal’s own text draws the same line. It says robots.txt tells automated tools what access is acceptable, while llms.txt is used on demand, when an agent needs information about a topic while helping someone. The two are meant to sit side by side. If you want the rules for blocking or allowing individual AI crawlers, Robots.txt for AI Crawlers: What to Block, What to Allow, and What 63 Real Sites Do covers them, and a hard block that does not depend on the crawler’s good manners has to happen at the server.

What Google, Chrome and the AI companies say about llms.txt

Google’s guide to its generative AI features in Search lists llms.txt under things you do not need to do. It says Google Search does not use these files, so you do not need one to appear in Search or in its AI features, and that publishing one for other systems will neither help nor harm your rankings. That is a statement about Google Search only, and it is worth reading that way.

Chrome’s tooling takes a slightly different line. Lighthouse now has an agentic browsing category with an llms.txt check. Its documentation says the audit flags a server error when fetching the file, and marks the audit not applicable when the file is simply missing, because providing it is optional for now. So the same company tells site owners they do not need the file for Search and also checks for it in a test aimed at browser-based agents.

For the AI companies, I read OpenAI’s and Anthropic’s crawler pages on 8 November and found no statement that their crawlers fetch llms.txt from your site. The proposal’s authors say OpenAI, Anthropic and Gemini publish llms.txt files for their own developer documentation, which shows they find the format useful as publishers. It does not say what their crawlers do with yours. Nobody outside those companies can confirm what a given tool does when it meets the file, which is why I would treat any promised benefit as unproven.

What 47 real llms.txt files look like

I used the same list of sites as the robots.txt survey: WordPress hosts and plugin makers, ecommerce platforms and shops, software companies and a few publishers. Sixty-nine sites, not counting this one and Repairpicks. The result for each site fell into one of four groups.

Result of requesting /llms.txt	Sites (of 69)
A real file (status 200, plain text or Markdown)	47
Not found (404)	16
Status 200 but the body is an ordinary web page	2 (Huel, Trello)
Refused the request or never connected	4

The two sites in the third row matter if you ever test your own file. A page that returns 200 and shows a “not found” design looks fine in a browser and fails for a tool that expects Markdown, so I counted them as having no file. Hostinger, the host this site runs on, is one of the sites with a real file: a 14 KB llms.txt with a 1.1 MB llms-full.txt beside it.

The 47 real files were not all alike:

• Shape. 44 start with the required top-level title. 33 add the short quoted summary the format calls for. 46 have at least one section of link lists, and 9 use the “Optional” section the proposal reserves for links an agent can skip.
• Size. The median file is about 10 KB, and 33 of the 47 are about 20 KB or smaller. Five are very large: Dropbox at 204 KB, WPBeginner at 496 KB, Mailchimp at 515 KB, Zendesk at 724 KB and SendGrid at about 2.5 MB with nearly 10,000 links. The proposal says the file should stay small enough to fit in a model’s context, so those five cannot be read in one go.
• A second file. 18 sites also serve /llms-full.txt, a longer version that some documentation platforms generate.
• A discovery header. Only 2 sites, Cloudflare and Intercom, send the HTTP Link header the proposal recommends for pointing clients at the file.

Eleven of the 47 are not a guide to the website

Eleven shops in my list, among them Allbirds, Gymshark, Glossier and Rothy’s, serve a file of about 5.6 KB built from one template: the same “Agent Instructions” title, the same sections and the same eight links, with the shop’s own name and address filled in. It describes how an AI agent can shop at the store, through a shopping skill and a checkout protocol, not what the site contains. It has no summary quote and no list of pages. A template shared by eleven separate shops points to a platform default, not eleven people writing a file. That is my reading of the evidence; I have not seen the platform say so.

Shopify’s own site, by contrast, has an ordinary 12.5 KB reading guide. So a fair count of real llms.txt files is 36 of 69 sites, not 47, and nearly a quarter of the “adoption” in my list is one platform’s setting. That also fits the proposal’s own list of tools that generate the file automatically, including the Yoast and All in One SEO plugins for WordPress, Wix and documentation platforms. When a site has the file, it is often because its platform wrote it.

Where the links go

The second version of the proposal asks that the links in the file point to clean Markdown versions of pages, so an agent does not have to untangle HTML. I took the first ten links from each file, skipped any address I had already tested, and requested 341 distinct links from 35 files.

• 318 loaded. 3 returned not found: a PayPal shopping page, a WP Engine API address and a Zendesk blog post. The other 20 refused an automated request or errored; Bluehost alone accounted for 11 of them.
• Of the 318 that loaded, 241 returned an HTML page, 60 returned Markdown and 17 returned something else, mostly JSON.
• Only 10 of the 35 files link to any Markdown at all. Cloudflare, Intercom, Node.js and SendGrid link only to Markdown and Ubuntu links mostly to it, while the other five link mostly to ordinary pages.

So the file exists on most big sites, and the second half of the recommendation, serving a Markdown copy of each page, is the part almost nobody has done.

Does having llms.txt go with having strict robots.txt rules?

Of the 63 sites whose robots.txt I could read, 17 name at least one AI crawler. Twelve of those 17 publish an llms.txt, and so do 33 of the 46 that name none. Having crawler rules does not predict having the file. The five sites with rules and no file are the BBC, the New York Times, the Guardian, GoDaddy and Moz. The BBC and the New York Times are the two publishers that block OpenAI’s training crawler completely, and neither serves llms.txt. I would not make much of five sites, but it fits the idea that a publisher who wants AI tools to stay out has no reason to hand them a map.

Should you publish one

For a small site, my view is that it is a cheap file with no proven return. Google says it does nothing for Search. No AI company I read says its crawler uses yours. What it can do is give tools that look for it a clean starting point, and it costs a few minutes. This site and Repairpicks both serve one. Ours is 11.3 KB, follows the format, and its 37 links point to ordinary HTML pages, so by the second version’s standard it is half done.

If you publish one, these are the points the survey makes worth checking:

1. Start with the site name as a top-level title, then a one-line quoted summary. 14 of the 47 real files skip the summary, and it is the part a tool reads first.
2. List the pages you most want read, each with a short description, in a handful of sections. Most real files stay under 20 KB. Do not paste in your sitemap.
3. Serve it as plain text or Markdown at the root, with status 200. Fetch it in a private window and confirm it is not your theme’s “not found” page.
4. Check the links. Three of the 341 I tested were already gone.
5. On WordPress, a generator plugin can write the file for you. If you add it by hand, a physical file in the site root is the simplest route.

A quick test after you save it:

curl -sI https://yoursite.com/llms.txt | head -n 5

You want a 200 status and a content type of text/plain or text/markdown. A content type of text/html means the file is not being served.

What to keep in mind

Treat the two files as separate decisions. Robots.txt is where you decide who may read your pages, and it is the one with a formal standard behind it. Llms.txt is a reading guide that, as of today, Google says it ignores and the AI companies have not said they use. Whichever you choose, check the files again in a few months, because Chrome’s llms.txt check is new and the vendors’ positions are still moving.

Checked against Google’s generative AI guide, Chrome’s Lighthouse documentation, the llms.txt proposal and a live fetch of 69 /llms.txt addresses and 341 of their links on 11 November 2026.