Blog · Foundations

Eight claims about AI search that do not survive a test

Published 17 July 2026 · Updated 8 October 2026 · 9 min read

Eight claims about AI search that do not survive a test
Photograph: Vova Pintia / unsplash

The short answer: Eight claims get repeated as settled fact, and each one narrows or breaks once you open the documentation it is supposed to rest on or run the check yourself. Special AI markup and llms.txt files, schema as a citation lever, a one line opt out, rank one as a free pass, a single prompt run as evidence, a before and after as proof, other people's sites as the only source of errors, and the updated date as a fix. Here is what is true instead.

None of these are stupid. Most were sensible guesses in 2024 that nobody went back and tested. The cost of carrying them is that teams spend a quarter on the wrong layer of answer engine optimisation and then conclude the field is noise.

Claims one and two: the things you are told to build

The first claim is that generative search needs its own artefacts. An llms.txt file, pages re-chunked into AI friendly blocks, a parallel set of markup. Google's guidance on AI features says the opposite in plain words, for Google's own surfaces: there is no special schema.org structured data you need to add, and you do not need to create new machine readable files, AI text files or markup to appear in AI Overviews or AI Mode. To be eligible as a supporting link, a page must be indexed and eligible to be shown in Google Search with a snippet, and the page states there are no additional technical requirements. That document was last updated on 10 December 2025. It covers AI Overviews and AI Mode only. ChatGPT, Perplexity, Copilot and Claude publish their own crawler documentation and none of it describes a required artefact either, but the absence of a published requirement is not the same as a tested one, so the honest version of this claim is narrower: nobody documents needing these files, and we have not found a case where adding one changed retrieval.

The second claim is the inverse of the first and it is more seductive, because schema does do something. Marking up a product with a price does not make an engine quote you. What it does is let a machine resolve a fact without parsing your sentences, which matters when your page says one thing and a review site says another. Treat markup as identity plumbing rather than a lever. The order of work is in structured data for AI search, and the short version is that the markup must describe what a human can see on the page, or you have added a contradiction rather than a signal.

Claim three: the opt out that is really four switches

People say they have blocked AI. Usually they have blocked one agent out of several and have no idea which behaviour they changed.

OpenAI's crawler documentation lists four user agents with four separate jobs. GPTBot gathers content for foundation models. OAI-SearchBot surfaces sites in ChatGPT's search features, and OpenAI says an opted out site will not surface in ChatGPT's search answers, while still being reachable as a navigational link. ChatGPT-User covers actions a person initiated, and OpenAI warns that your robots file may not govern a fetch of that kind, because a human asked for it. OAI-AdsBot only visits pages submitted as ads. Within OpenAI's products, then, disallowing GPTBot does not remove you from ChatGPT's search answers, and disallowing OAI-SearchBot does not stop training. Other engines split their tokens differently; check each one's own documentation rather than assuming this shape carries across.

Google splits its own tokens for a different reason. Google-Extended governs whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and Google states that it does not impact a site's inclusion in Google Search and is not used as a ranking signal there. That entry was last updated 14 July 2026. AI Overviews and AI Mode run on the Search index, reached by Googlebot. So the common defensive move, disallowing Google-Extended, does not take you out of AI Overviews. It takes you out of Gemini's training and grounding and leaves the surface most people were worried about untouched.

Go and read your own robots.txt before you argue about this. Most of the arguments end there.

Claim four: ranking first means being named

Being the first blue link and being the brand in the sentence are different outcomes produced by different selection steps. On Google's surfaces the index is shared: AI Overviews and AI Mode draw supporting links from pages already indexed and snippet eligible. How passages are then chosen is not something the engines document in detail, so treat the rest as inference from what we see rather than documented mechanism. In our own audit runs a page well below the first result is sometimes the one quoted while the first result is not cited at all. We have not measured how often that happens.

One traffic consequence has been measured, on Google, in the United States. Pew Research Center analysed 68,879 Google searches drawn from the tracked browsing of 900 US adults on a survey panel during March 2025, and published the results on 22 July 2025. A plain search link was clicked on 8% of the visits that carried an AI summary, and on 15% of the visits that carried none. Sources inside a summary were clicked on 1% of such visits. Visits that ended the browsing session: 26% with a summary, 16% without. That is one engine, one country, one month, and an opt-in panel rather than a census of searchers, so it is not a reading of what AI assistants in general do to traffic, and it is eighteen months old.

Read those three numbers together and the ranking frame stops making sense. A correct mention inside the answer is worth more than a position below it. Why AI search is not really a ranking game works through what replaces position as the unit of measurement.

Claims five and six: what counts as evidence

Claim five is that a prompt run once tells you something. A preprint submitted on 18 May 2026 evaluated 13 language models across four benchmarks and found that meaning preserving rephrasings of the same question changed the answer, with mismatch rates reaching more than 23% on the worst cases rather than on average. Those benchmarks are factual question answering and mathematical reasoning with no retrieval attached, so they bound instability in the model alone, before a live index and a rotating set of candidate pages are added on top. That is a reason to expect single runs to be noisy, not a measurement of how noisy brand answers are. Run the prompt five times. If your name appears twice, record 40% presence rather than a yes or a no.

Claim six is the expensive one, because it feels like science. You edit a batch of pages, wait a fortnight, see citations rise, and write it up. The design is the problem, not the effort: with nothing held back, a rise caused by an index refresh, a model update or ordinary week to week movement is indistinguishable from a rise caused by your edit. We have seen untouched pages move over the same window in our own monthly re-runs, which is why our audits hold out a set of pages and measure them alongside the edited ones. We are not publishing a figure for that drift, because we have not run it as a controlled study with a stated sample. The design lesson stands on its own: without untouched comparisons, a before and after on these surfaces supports no causal claim.

The self-check is ten things you can run this afternoon with a spreadsheet, and it is deliberately built around repeated runs rather than single readings.

Claims seven and eight: where wrong facts live, and the date field

Claim seven is that inaccurate statements about your brand come from elsewhere. Competitors, review aggregators, an old news piece. Figures circulate putting most of the blame on a brand's own pages instead, but the ones we can find are vendor estimates with no openable method, so we are not repeating them here. What we can say is that the mechanism is checkable on your own site in an hour, and that contradictory pricing is the recurring finding in the audits we run. Open your pricing page, your plans page, the pricing section of three old blog posts and any PDF the sales team still sends. Count the versions of the number. If there is more than one, an engine cannot tell which one you meant, so it picks, and it may pick the oldest.

Claim eight is that refreshing solves this. Bumping dateModified, adding the current year to a title, republishing with no substantive change: these move a field, not a fact. What gets quoted is text from the page, so a passage with an old price in it is still an old price after you edit the timestamp above it. And a stale copy that sits on a page you do not own is not reachable by republishing yours at all. How often to refresh content for AI search draws the line between a refresh that changes retrievable text and a refresh that only changes metadata.

Honest limits

What this post does not establish, stated plainly. The documentation above is Google's and OpenAI's, and it describes their own surfaces only, so nothing here should be read as a rule covering engines that publish no equivalent guidance. The Pew figures are one engine, one country and one month of panel browsing from March 2025, which makes them a dated sample rather than a current measure of AI search traffic. Where we describe what we see in our own audits, the prompt runs, the drift on untouched pages, the contradictory pricing, those are observations across client work and not a controlled study, which is why they carry no percentages. We cannot attribute a mention in an answer to revenue, because most of these surfaces pass no usable referrer, and we do not know how long a fix lasts, which is why our prompt sets are re-run monthly.

FAQ

Is llms.txt worth adding anyway? It costs almost nothing, and we have not found an engine that documents reading it; Google's AI features page says such files are not needed for its own surfaces. We cannot see other engines' retrieval, so we cannot say the file is ignored, only that nothing we can open says it is used. Add it if it helps your own team keep an inventory. Do not count it as AI search work, and do not let it displace fixing whether your pages render for a crawler at all.

If blocking Google-Extended does not remove me from AI Overviews, what does? Snippet controls. Google's AI features guidance points at nosnippet, data-nosnippet, max-snippet and noindex, which are the same levers that governed featured snippets. They reduce what can be shown from your page, and they reduce your presence in ordinary results too, so it is a trade rather than a switch.

How many runs make a prompt result trustworthy? In our own audits we use five runs per prompt per engine, across a prompt set of about 45 prompts and five engines, and record presence as a fraction. Five is a working compromise, not a statistical threshold. The point is that a fraction forces you to report uncertainty, and a single run lets you hide it.

Does any of this mean structured data is pointless? No. It means schema resolves identity and keeps facts machine readable, which is a different job from winning a citation. We have seen pages with clean markup and contradictory prose summarised wrongly all the same.

Sources

Related reading

What is answer engine optimisationWhy AI search is not really a ranking gameStructured data for AI search: what to mark up firstHow often to refresh content for AI searchDoes SEO still matter for AI searchHow an AI answer is assembled, step by step