Blog · Machine readability

What an agent-readable page looks like

Published 1 September 2026 · Updated 8 October 2026 · 9 min read

What an agent-readable page looks like
Photograph: Albert Stoynov / unsplash

The short answer: An agent-readable page puts its answer in the HTML the server returns, before any script runs. It states one fact once, in text a machine can lift without guessing. It names itself: what the page is, who publishes it, what the thing costs, whether it is in stock. And it survives being cut into a four hundred word passage, because that passage, not the page, is what the model actually reads.

That is the whole brief. The rest of this post is the detail, the ways pages fail it, and three checks you can run in the next ten minutes.

The page a model sees is not the page you see

Open your product page in Chrome. You are looking at the output of a render: HTML, plus however many scripts hydrated into it, plus whatever a tag manager injected afterwards.

Several crawlers never get that far, and for some of them we can say which. A study by MERJ and Vercel, published 17 December 2024 from logs for nextjs.org and two job board sites on that network, tested rendering per crawler. It found that the OpenAI, Anthropic, Meta, ByteDance and Perplexity crawlers did not execute JavaScript, while Google's Gemini renders through Googlebot's infrastructure and AppleBot renders through a browser-based crawler. On nextjs.org in that period, ChatGPT's crawler spent 11.50 percent of its requests on JavaScript files it was not executing, and Anthropic's 23.84 percent. That is one study, on a handful of sites, nearly two years before this post. Check your own logs for the crawlers you care about rather than assuming the split still holds.

So the practical test is not "does my page work". It is "what is in the first response". Run this and read what comes back:

`` curl -sL -A "GPTBot" example.com/your-page | sed -e 's/<[^>]*>//g' | head -60 ``

If your price, your product name and your first paragraph are not in that output, schema is unlikely to rescue the page for any fetcher that does not render. The markup is describing content that fetcher cannot see.

A second number from the same December 2024 study is worth keeping, with the same caveat that it describes those sites in that month: ChatGPT's crawler spent 34.82 percent of its fetches on 404 pages and Anthropic's 34.16 percent, against 8.22 percent for Googlebot. The authors did not establish why, and said so; what they could see was repeated requests for retired assets under /static/. On those sites, stale internal links and silently retired URLs were expensive. Whether that holds on yours is a question your access logs can answer and this post cannot.

Shape one: the answer sits above the argument

An agent does not read your page top to bottom and form a view. A retrieval pipeline splits the document into passages, scores each one against the query, and keeps two or three. The passage is the unit. Whatever is not inside the retained passage may as well not exist.

That changes layout. A heading that states a question, followed immediately by two or three sentences that answer it, survives chunking. A heading that promises insight, followed by four hundred words of context before the number appears, produces passages that score well on topic and contain nothing useful. The engine keeps one, finds no answer, and cites someone who was blunter.

Tables appear to survive well, because a row carries its own labels. Long bulleted lists with the subject only in the stem above them appear to survive badly, because a chunk boundary falling mid-list leaves every bullet an orphan fragment. We cannot watch the chunking happen, so this is an inference from which of our own passages get quoted back, not a measured result. Repeating the subject in the row costs nothing either way.

Shape two: the page names its own entity

Here is the part most sites skip. A page can be perfectly readable and still leave an agent unable to say which company it belongs to.

Google's own guidance on AI features is blunt about this: a page qualifies for AI Overviews and AI Mode if it is indexed and eligible to show with a snippet, and there is, in their words, no special schema.org structured data you need to add for those features. Read that carefully. It says schema is not an entry ticket. It does not say schema is useless, because schema still does the job it always did, which is to state unambiguously that the string "Citrel" on this page refers to this organisation, with this URL, selling this product, at this price, in this currency.

That is the difference between legibility and resolution. Structured data for AI search: what to mark up first covers which types pay first. The identity half of the job, the links that let an engine match your page to your profiles elsewhere, is in sameAs and the identity links engines check.

One rule beats all of the type-selection advice: never mark up something that is not visible in the text. If the JSON-LD says 1,249 rupees and the rendered page says 1,399, you have not added signal. You have added a contradiction, and in the contradictions we have put in front of engines during audits the usual outcome is a hedge or a competitor's figure. We have not run that as a controlled test, so read it as what we see, not as a law.

Shape three: one fact, one place, one spelling

Agents fail on ambiguity more than on absence. A missing return policy produces "I could not find it". A return policy that says 30 days on the product page, 14 days in the footer and "two weeks" in the FAQ produces a confident wrong answer, which is worse, because nobody reports it.

Walk your own site for the six facts a buyer asks about: price, availability, delivery time, return window, what is included, who it is for. Grep for each one. Count the versions. We run this grep on client sites during audits and it is rare to find only one live answer for all six, usually because an old landing page never got retired. We have not kept those counts in a form anyone could audit, so there is no rate here to quote, only the check and what it tends to turn up.

For catalogue pages the stakes rise, since an agent may be transacting rather than summarising. Agentic commerce: what an agent needs from your catalogue goes through the fields that decide whether your product is comparable at all.

What does not work, and the file everybody adds anyway

There is a persistent belief that agent readability is a file you publish. It is not.

An adoption study tracking more than 3 million websites counted 4,088 llms.txt instances in June 2025 and 36,120 by May 2026, about 8.8 times growth in twelve months. A separate server log study of 137,000 domains, published June 2026, found that 97 percent of those files received zero requests in May 2026. In the logs it could see, the largest identified shares of requests came from SEO audit tools at 21.7 percent and unidentified bots at 14.9 percent, with retrieval bots at 1.1 percent. That is a sample of 137,000 domains for one month, not a census of the web, and server logs cannot show you a file read from a cache or a crawl you are not hosting.

Publish it if you like. It costs an hour. Just do not count it as the work, and do not let it displace the thing that is actually measured, which is whether your HTML carries the answer.

The same caution applies to serving a different, tidier page to agents. Google's spam policies define cloaking as presenting different content to users and to search engines with the intent to manipulate rankings and mislead users, and give serving text only to a search engine user agent as an example. Intent is part of their definition, so an accessibility fallback is not automatically cloaking. Other engines publish less about it, so treat parity as a documented Google rule and an open question elsewhere rather than a settled rule everywhere. Either way a divergence found later is expensive to unwind.

Checks you can run this afternoon

  1. The curl test. Fetch the page with scripts disabled, as above. Your answer should be

    2. in the plain text. If it is not, that is your first fix, ahead of everything else.

  2. The paragraph test. Copy any 300 word window from the middle of the page and paste it

    3. alone into a model with the question your buyer asks. If it cannot answer from that window alone, the window is padding.

  3. The contradiction test. Pick one fact. Search your own domain for it. If you find more

    4. than one answer, the engines have found it too.

  4. The log test. Separate crawler hits from agent hits in your access logs before you draw

    5. any conclusion about demand. Agent traffic is not bot traffic explains why the two behave differently and why mixing them produces a flattering, useless number.

The self-check is a longer version of the same idea, ten things you can test without tooling.

Honest limits

This post does not establish that the shape it describes causes citations. It establishes that specific crawlers, on specific sites, in specific months, could not see content that only existed after JavaScript ran.

The chunk sizes and overlaps used by production retrieval stacks are not published, so everything here about passages, tables and orphaned bullets is inferred from which of our own passages come back quoted. It is a working theory, not a spec, and we have run no controlled test against it.

The rendering and 404 figures are one December 2024 study of a few sites on one network. We have not repeated that measurement ourselves, and crawler behaviour has had nearly two years to change. The llms.txt figures are one month of logs from 137,000 domains; absence of requests in that sample does not prove the file goes unread elsewhere, only that these logs show almost none.

We also have no count to offer for the contradictory-facts check. We run it on client sites and describe what it tends to turn up, and until we publish a method and a sample size that is an impression rather than a finding.

FAQ

Does an agent-readable page need JavaScript removed? No. It needs the answer present before JavaScript runs. Server-side rendering, static generation or a prerendered fallback all satisfy that. Interactive extras on top are fine.

Is llms.txt worth publishing? It is cheap and, where anyone has measured, mostly unread. A server log study of 137,000 domains found 97 percent of llms.txt files took zero requests in May 2026. Those are the logs that study could see, not every log there is. Publish it as a convenience, not as a strategy.

Should I serve a simplified page to AI crawlers? Google's spam policies define cloaking as showing users and search engines different content with intent to manipulate rankings, so a tidier page served only to crawlers is the example they give. We have not found an equivalent published rule at most other engines, which means it is documented for Google and unknown elsewhere rather than forbidden everywhere. Divergence is also hard to unwind once found.

How do I know if my markup is helping? Change one page, wait for a recrawl, then re-run the same prompts on more than one engine and compare what gets cited. A single run proves nothing, because answers vary between runs.

Do images and video need anything? Give them real alt text and real captions in the HTML. A model reading a passage cannot see the image, so anything the image conveys must exist in text somewhere nearby.

Sources

Related reading

Structured data for AI search: what to mark up firstsameAs and the identity links engines checkAgentic commerce: what an agent needs from your catalogueAgent traffic is not bot trafficYour API and MCP server as a visibility surfaceAuthor and publisher markup that survives scrutiny