Blog · Measurement

How to measure AI visibility without fooling yourself

Published 2 September 2026 · Updated 12 October 2026 · 9 min read

How to measure AI visibility without fooling yourself
Photograph: AFINIS Group ® - AFINIS GASKET® Production / unsplash

The short answer: Measure a frozen set of buyer questions, several runs each, on more than one engine, and report a proportion with an error bar rather than a score. Record what was said about you, not only whether you were named. Keep the evidence. Anything that gives you one number from one run on one day is a mood ring, and the mood belongs to the model.

The trap is not that these numbers are hard to collect. They are easy, which is the problem. A spreadsheet of mention rates looks like measurement well before it is any.

The engine is noisy, and the noise is often bigger than your gains

Start with the mechanism, because it decides the method. Two identical prompts, sent minutes apart, can return different sources and a different roster of brands. People usually blame temperature. That is not the main cause.

Work published by Thinking Machines on 10 September 2025 traced it to batch invariance. Inference kernels reduce floating-point numbers in an order that depends on the batch they land in, and batch size depends on how busy the server is when your request arrives. Their test is the useful part: 1,000 completions from a single open-weight model, Qwen3-235B-A22B-Instruct-2507, self-hosted on vLLM at temperature zero with the same prompt every time, produced 80 distinct outputs, with the first divergence appearing at the 103rd token. Greedy decoding, no sampling randomness, still 80 answers.

That is one model on one serving stack, not a measurement of ChatGPT, Perplexity, Gemini, Claude or AI Overviews. We cannot see inside a hosted engine's serving stack, so we cannot say this is the cause of the variation we observe there. What we can say is that the variation is there, it survives repeated identical prompts, and a mechanism that produces it without any sampling randomness has been demonstrated in public.

So a single run tells you what one roll of the dice said. If your report compares last month's one run against this month's one run, every number in it is noise with a date attached.

The fix is sample size and honest arithmetic. Run each prompt several times per engine, and treat the result as a proportion. A reasonable audit shape is around 45 prompts run five times on each of five engines. Here is the arithmetic that people skip, worked below on a hypothetical 40 per cent to show the shape rather than to report any client's number. If 40 per cent of those responses name you, the binomial standard error across 225 observations is 3.3 points, so you might quote plus or minus 6 points. Runs of the same prompt are correlated, though, so the independent unit is closer to the prompt than the run. Recompute on 45 and the standard error is 7.3 points, so the 95 per cent interval is roughly plus or minus 14 points. Which means a move from 38 to 44 per cent is not a result. It is Tuesday.

That calculation takes one spreadsheet line and ends most victory laps.

Your prompt set is the denominator, and it is doing all the work

Visibility percentages have no meaning without the question list underneath them. Change the list, change the number, with nothing about the business changed at all.

Two ways teams fool themselves here. The first is seeding the set from their own marketing language, so every prompt uses the category name they invented and the score comes back flattering. The second is quietly editing the set between reporting periods, usually by dropping prompts that embarrass the brand. Both produce a chart that climbs.

Freeze the set and version it. Put it in git, tag it, and when you add prompts, add them as a second cohort rather than merging them into the trend line. Building a prompt set that represents real demand covers where the questions should come from in the first place, which is sales calls and support tickets, not a keyword tool.

One more denominator check. Sort your prompts by buying stage. A brand that scores well on "what is X" and badly on "best X for a mid-market team" has a visibility problem exactly where the money is, and a blended score hides it perfectly.

Named is not cited, and both are worth counting separately

An engine can describe you in a sentence without linking to you. It can also link to you while recommending somebody else. These are different outcomes with different causes and different fixes, so collapsing them into one metric throws away the diagnosis.

Count four things per response: whether you were named, whether your domain was cited, whether the claim made about you was accurate, and which other brands shared the answer. The accuracy column is the one we rarely see built and the one that changes behaviour, because a correct mention and a wrong mention score identically in the reporting we have been shown by clients arriving from other arrangements. That is what has crossed our desk, not a survey of the category. Citation share vs mention share goes further into why the two diverge.

The attribution gap is real, and pretending to close it is the biggest lie

Here is where confident reporting becomes fiction. Google's own documentation states that sites appearing in AI features, including AI Overviews and AI Mode, are folded into overall search traffic in Search Console and reported inside the Web search type. The dedicated Search Generative AI performance reports that shipped on 3 June 2026, and finished rolling out worldwide on 31 August 2026, list impressions broken down by page, country, device and date, and list nothing else. No queries, no clicks, no click-through rate and no position appear in that launch note, which also says Google may add metrics over time. Read it as what the report carries today, not as a permanent ceiling.

Outside Google it is thinner still. Cloudflare's published crawl-to-refer ratios for the first week of August 2025, in the unfiltered default view across the sites Cloudflare observes, recorded roughly 50,000 pages crawled per referral sent for Anthropic's crawler, 887 to 1 for OpenAI's, and 118 to 1 for Perplexity. Those are not your site's ratios, and they vary sharply by vertical: the same post puts news and publications sites at 2,500 to 1 for Anthropic and 152 to 1 for OpenAI. The methodology is Cloudflare's, not ours. The direction of the gap is the part that travels.

What follows is uncomfortable and correct. AI appearances are countable. AI-sourced revenue is modelled. If a deck shows a rupee figure next to an AI visibility chart, ask which assumption produced it and what happens to the number when that assumption moves ten per cent.

The proxies worth running anyway, labelled as proxies: a free-text box on the enquiry form asking where the buyer first came across you, branded search volume over time, direct traffic to deep pages that nobody bookmarks, and server logs filtered to assistant user agents so you at least know who fetched what. None of these is attribution. Together they are a tripwire.

Keep the evidence, or you are grading your own homework

Every finding should have the raw response stored next to it, with timestamp, engine, prompt and the sources the engine named. Without that, a disagreement in a meeting becomes an argument about whose dashboard is right, and the dashboard always wins because it is on the screen.

Evidence also protects the time series. When a report says visibility fell, you want to open three responses and see whether you vanished, got demoted, or got described with a stale price from a page you deleted last quarter.

That last one has a specific cause worth naming. Re-platform a site, change URL structure, and the citations pointing at old paths go stale faster than the engines re-crawl them, so visibility appears to collapse while the content itself never changed. Migrating a site without losing citations is the protocol for not doing that to yourself.

If you are buying this work rather than building it, the questions to ask a supplier are in how to buy an AEO audit without being sold a dashboard. The short version: ask to see one finding, with its evidence, before you look at a single chart. Our own sample audit report shows the shape we hold ourselves to, evidence attached per finding.

Honest limits

We cannot connect a mention to revenue: we have not found a way to, and the surfaces we work with do not emit the per-query click data that would make it possible, so this is our own ceiling rather than a claim about everyone's. We have run no controlled durability study, so when the same frozen prompt set moves between monthly re-runs we can name the cause only sometimes, and we cannot put a half-life on a fix. The batch-invariance result above was measured on one open-weight model on one self-hosted serving stack, which makes it a plausible mechanism for the variation we see in hosted engines, not a measurement of them. The error arithmetic assumes each prompt is an independent draw, and prompts on the same topic move together, so a real interval is wider than the figures above suggest. Five engines is a judgement about what matters to our clients, not a sample frame for the market.

FAQ

How many runs per prompt is enough? Five is the floor we work to, and it is a compromise between cost and noise. Fewer than three and you cannot tell a change from a coin flip. If a specific prompt is commercially important, run that one twenty times and stop pretending the blended average speaks for it.

Is a visibility score from a tool useless? No, as a trend on a frozen prompt set with the runs underneath it. It becomes useless the moment the prompt set changes silently, or the score is quoted without the interval, or nobody can open the response that produced it.

Can I use Search Console to measure AI visibility? Partly. The generative AI reports give impressions by page, country, device and date, which tells you a URL appeared. Google's launch note lists no queries, no clicks and no position among the metrics, so as the reports stand today they answer whether, never what happened next.

Should average position still be in my reporting? Treat it with suspicion. Google's Search Console documentation states that an AI Overview occupies a single position and every link inside it is assigned that same position, so a site pulled into those surfaces often can see its average position chart drift for structural reasons rather than performance ones. That rule is documented for AI Overviews specifically; Google documents AI Mode as following the ordinary results-page methodology.

What is the cheapest honest version of this? A frozen list of 20 questions, run three times each on two engines, logged in a spreadsheet with the date and the response pasted in. It costs an afternoon a month, and unlike most of what it replaces, the evidence is right there to open.

Sources

Related reading

Building a prompt set that represents real demandCitation share vs mention shareHow to buy an AEO audit without being sold a dashboardMigrating a site without losing citationsReporting AI search to the people who fund itTesting with retrieval on and off