Blog · Access and retrieval

Does llms.txt do anything yet

Published 8 October 2026 · 7 min read

Does llms.txt do anything yet
Photograph: Mathyas Kurmann / unsplash

The short answer: Yes in one narrow lane, no in the lane most people mean. Coding agents and documentation tooling fetch llms.txt deliberately, and that is a real use. The assistants writing the answers you want to appear in do not consult it, and Google states in its own documentation that no new machine readable file is needed to show up in AI features. Publish one if you sell to developers. Do not budget it as a visibility lever.

The argument here is noisy because two questions keep getting answered as one. Does anything read llms.txt. Does reading it change whether a brand gets named in an answer. The first has a yes attached. The second does not.

What the file is, stripped of the marketing

The specification lives at llmstxt.org, and the current revision was modified on 10 August

  1. It is short enough to read in five minutes. A conforming file is markdown at the root of

    2. a host or at any path within it, covering the pages under that path. An H1 carrying the site or project name is the only required element. Everything after that is optional: a blockquote summary, some prose, then H2 sections holding annotated links to the pages that matter.

Note what the spec omits. There are no directives. Nothing in the file can permit or forbid a fetch, which is why the robots.txt comparison misleads people into expecting it to control crawler behaviour. Access control still lives in robots.txt, and the decision about who you let in is covered in should you block AI crawlers.

A better mental model: llms.txt is a pull resource. It sits there. Something has to choose to request it, parse it, and act on what it found. The whole question of whether the file works collapses into whether anything in the pipeline makes that choice.

Who actually fetches it

Two populations do, and they are both on the agent side of the line rather than the search side.

The first is documentation platforms and the agent tooling built on them. Cloudflare's guidance for agents, last updated 29 September 2026, publishes a site index at /llms.txt, a per-section index at /docs-for-agents/llms.txt, a bulk file at /llms-full.txt, a markdown version of any page via /index.md, and the same markdown through an Accept: text/markdown request header. The response carries x-markdown-tokens and x-original-tokens headers so a calling agent can see what it saved. That is a system built around the file.

The second is the AI labs, for their own developer docs. On 8 October 2026 we requested the file from several of them. docs.anthropic.com/llms.txt returned 200 with roughly 83 KB of markdown. docs.perplexity.ai/llms.txt returned 200 with about 37 KB. The apex domains behaved differently: openai.com/llms.txt returned 403, anthropic.com/llms.txt returned

  1. The pattern is consistent and it tells you the intended audience. These files live

    2. where a developer agent goes looking for an API reference, not where a shopper's question lands.

To see how far the practice has spread, we checked /llms.txt on 50 well known domains the same day. Thirty four served a 200 with a text content type. Of those, 31 opened with the required H1, so three were serving something the spec would not recognise as conforming. Adoption is real and it clusters hard in software. Every retailer, airline and news site in our sample returned a 404 or an HTML error page.

Why the engines that matter skip it

Retrieval, as the engines document it and as our own logs show it, has no step where a root file gets read. The loop is a query, a search index lookup, a set of candidate URLs, a fetch or a cached copy, passage selection, then generation. A file at your root is not a candidate URL for any user question, so nothing surfaces it. Adding one does not insert a new step into someone else's pipeline.

Hold that at the right width. Google states the file is not required for its AI features, and the request logs we can see carry almost no traffic to it. Neither of those is proof that no assistant, agent or future product reads it; absent requests in one place are evidence about that place. What they do not support is paying for it as a visibility lever today.

Google says this plainly. The Search Central page on AI features, last updated 10 December 2025, states that you do not need to create new machine readable files, AI text files or markup to appear in those features, and that a page needs to be indexed and snippet eligible to be used as a supporting link. Read that as a routing statement rather than a snub. The engine is telling you the gate is the index, and the index is fed by crawlable, parseable HTML. Whether yours qualifies is the subject of can AI crawlers actually read your site, and that is the work llms.txt gets mistaken for.

Two failure modes that come with publishing one

Size. The spec says nothing about length, so tooling generates whatever the docs contain. In our 50 domain sample the median file was about 11 KB, which is sensible, but six files ran past 100 KB and the largest was roughly 2.3 MB. A 2.3 MB markdown file is not read. It is truncated, and you have no say over where the cut lands. If your file is large, the first few hundred lines are the only part that reliably exists.

Drift. A markdown mirror of your site is a second copy of your pricing, your policy and your claims. Copies go stale on different schedules from the pages they came from. An agent that ingests both now has two versions of a fact and no principled way to choose, and the quiet failure is a confidently wrong answer sourced from your own stale file. One mildly reassuring result: across 44 links sampled from a dozen published files, 42 still resolved with a 200, so link rot was not the problem that day. Content accuracy is harder to test than a status code, and nothing checks it for you.

A check worth fifteen minutes

Run these before and after, not instead of each other.

  1. curl -sI yourdomain.com/llms.txt. Confirm a 200 and a text content type. An

    2. HTML error page served with a 200 is common and counts as broken.

  2. curl -s yourdomain.com/llms.txt | head -1. The first line should be `# Your

    3. Name`. Three of 34 live files in our sample failed this.

  3. curl -s yourdomain.com/llms.txt | wc -c. Past about 100 KB, treat everything

    4. below the fold as decorative.

  4. Check the paths it links to against your robots.txt. A file pointing at URLs your own

    5. rules disallow is worse than no file.

  5. Pick ten claims in it and verify each against the live page. That is the step that

    6. catches drift.

If you want a broader version of this exercise that is not about one file, the self-check is ten things you can test in an afternoon with no tooling.

The honest limits of what we can tell you

We measured availability and conformance on one day across a small sample we chose. That is not adoption across the web, and a 50 domain sample weighted toward software overstates how common the file is in general.

We cannot see inside anyone's retrieval stack. The claim that consumer engines ignore the file rests on vendor documentation and on the absence of requests in server logs, which is evidence of behaviour, not of architecture. A lab could start reading these files next quarter without announcing it.

And when someone tells you their citations rose after they shipped an llms.txt, the number is usually the weak part of the story. Answers are regenerated per run and vary between identical prompts, so a before and after taken from single runs measures noise. The sample size question is covered in how many runs per prompt before a number means anything. If your site has deeper access problems, this file is far down the list, which is roughly the argument in when answer engine optimisation is not worth doing yet.

FAQ

Should I publish an llms.txt at all? Publish one if you have developer documentation or an API, because the agent tooling in that lane genuinely reads it. For a shop, a clinic or a local service business, the hour is better spent on whether your pages render server side and your facts agree with each other.

Is llms.txt the same thing as robots.txt? No. robots.txt carries directives that crawlers obey or ignore. llms.txt carries no directives at all and cannot grant or deny access to anything.

What about llms-full.txt? It is the bulk variant holding full page content rather than links. Useful for an agent doing offline indexing, and the size problem above applies to it twice over.

Does having one help my Google AI Overviews visibility? Google's own documentation says no new machine readable file is required for those features. Treat anyone claiming a direct effect as owing you a run count and a method.

Could this change? Yes. Agent traffic is growing and the file is cheap to maintain if it is generated rather than hand written. A reasonable position is to ship a small correct one and keep it honest, without expecting it to move anything this quarter.

Sources

Related reading

Can AI crawlers actually read your siteShould you block AI crawlersHow many runs per prompt before a number means anythingWhen answer engine optimisation is not worth doing yetPaywalls, logins and what retrieval can reachHow to write robots.txt for AI crawlers