Blog · Measurement

What AI visibility tools can and cannot see

Published 23 April 2026 · Updated 12 October 2026 · 9 min read

What AI visibility tools can and cannot see
Photograph: Matt LaVasseur / unsplash

The short answer: AI visibility tools see one thing well: the text of answers they themselves generated, by sending prompts they chose to engines they have access to. None of the engines we checked publishes real user prompts, how often a question is asked, what an individual user was shown, or why one source was picked over another, so a tool working from the outside cannot report those either. Where a tool documents its method, the rest of what it reports is an inference built on top of that single observation.

That is not a complaint about the category. It is the shape of the problem. An answer is generated per request, for one person, inside a session nobody outside the engine can read. Any outside observer has to recreate a plausible request and report what came back.

The one thing they actually observe

Where a product documents its method, the mechanism is a loop. It holds a list of prompts. It sends each prompt to each engine, through an API, a partner endpoint or a browser session. It parses the returned text for brand names and for linked sources. It writes a row.

In that design every number downstream is arithmetic on those rows. Mention rate is the fraction of rows containing your name. Citation share is the fraction of linked sources that point at your domain. Sentiment is a classifier applied to the sentence your name sat in. The rows are real, and they are evidence. What they are not is a measurement of what buyers saw, which is the thing everyone quietly assumes a dashboard is reporting. That assumption is the subject of How to measure AI visibility without fooling yourself.

The five blind spots, in order of how much they cost you

Demand. We have found no public counter for how often a given conversational prompt is typed, and none of the engines we checked publishes one. Keyword tools had a search-volume signal, flawed but real, derived from advertising systems. Engines may well hold prompt-frequency data internally; what we can say is that it is not published, not that it does not exist. When a report weights prompts by importance, that weighting came from a human guess, from search volume for a loosely related keyword, or from a model's opinion. Ask which, and check whether your top three prompts are ones your customers would really type.

Personalisation and memory. In our own audit runs the same prompt returns different answers for a logged-in session with history, a session geolocated elsewhere, and a clean API call with no session. That is an observation from those runs, not a measured effect size, and we have not published a controlled comparison. A tool runs the clean version because it is the only reproducible one. Reproducibility is a virtue here, and it also means the measured population is a population of one synthetic user.

Routing. Assistants decide per request whether to retrieve at all, how many searches to issue, and which internal query strings to use, and those strings are not the user's words. Microsoft's AI Performance report in Bing Webmaster Tools, opened in public preview on 10 February 2026, is the only mainstream surface we have found that exposes them: it calls them grounding queries, and defines them as the key phrases the AI used when retrieving content that was referenced in AI-generated answers. Microsoft also says plainly that what it shows is a sample of overall citation activity, not a census. That much is documented, for one engine. For the others we have no comparable documentation, so read rewritten retrieval queries as confirmed at Microsoft and inferred elsewhere. Worth reading next to Microsoft Copilot and the Bing index, because the report is the clearest public window into retrieval that site owners currently have.

The rest of the answer. Being named is binary in a spreadsheet and graded in reality. First sentence versus a parenthetical in paragraph six are different outcomes with the same row value. Microsoft is explicit that its citation count does not describe placement or prominence either. If a tool gives you a share-of-voice percentage, ask whether position inside the answer changes it at all. We have not tested how each product handles that, so ask rather than assume. It is the limitation pulled apart in Share of voice in AI answers, and its limits.

Attribution. In the client analytics we have looked at, a large share of assistant-driven clicks arrive with no usable referrer and land as direct. We cannot see how each surface sets its referrer, so treat that as what those analytics show us rather than a rule we can extend to surfaces we have not looked at. Google counts a click from an AI Overview or an AI Mode answer only when someone follows a link to an external page. Those clicks sit in the overall Web search type in Search Console, and since June 2026 their impressions are also reported on their own in a dedicated Generative AI performance report, rolled out to all sites worldwide on 31 August 2026. Reading an answer and typing your brand into a browser two days later is invisible to both the tool and the analytics stack.

What they see better than you do

Two things, and they are worth paying for when the manual version stops scaling.

The first is repetition at volume. Answers vary run to run, so a single result is an anecdote and forty results are a distribution. Running forty prompts five times across five engines by hand is roughly a day of work, every week, forever. Machines are good at exactly that.

The second is competitor coverage. You can read your own site. You cannot easily notice that a particular review roundup is being cited in answer after answer across your category while your own comparison page is not cited at all. Pattern-finding across source lists is genuinely hard by hand and genuinely easy to automate.

Server logs: the half of the picture no dashboard holds

Here is the check that separates people who know what is happening from people who are reading a chart. Your own access logs record the fetches. Pull a week and group by user agent.

OpenAI's crawler documentation, checked on 12 October 2026, describes four user agents, and three of them are the ones that matter here. GPTBot collects content for model training. OAI-SearchBot builds the index behind ChatGPT's search. ChatGPT-User fetches a page live because a specific person asked a question just then, and OpenAI's documentation notes that these are user-initiated actions where ordinary robots rules may not apply in the usual way. The fourth, OAI-AdsBot, only visits pages submitted as ads. Those three hitting your site mean three completely different things, and we have not seen a dashboard that separates them for you, though we have not tested every product in the category.

An illustration rather than a case study, to show the shape of the gap: a tool reports your mention rate fell. Your logs show OAI-SearchBot stopped fetching a template of pages three weeks earlier, right after a deploy moved the main content behind a client-side render. The dashboard saw the symptom at week three. The log saw the cause at week zero. One of those is actionable on a Tuesday.

Run this pair quarterly and keep the output:

`` grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|ClaudeBot|bingbot' access.log \ | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head -40 ``

Then fetch one important page with each agent string yourself and diff the HTML against what a browser receives. If the two differ by more than boilerplate, the engines are reading a different site from the one you published.

What to ask before you buy anything

Four questions, each with a right answer that a vendor can give you in writing.

Which engines are covered by an official interface, and which are covered by a browser session that can break without notice. How many runs per prompt per period, because the answer determines whether the trend line is signal or noise. Is the raw answer text stored and retrievable, because a score without the text underneath it cannot be argued with. And does the prompt set belong to you if you leave.

None of this replaces owning the inputs. The sources an engine trusts, the pages it can actually parse, the consistency of your own facts: those are yours, and the split between them and the measurement layer is the point of Rented visibility vs owned visibility in AI search. If you want to see what evidence-backed findings look like before committing to any tooling, the sample audit report lays out the shape, with the observation attached to each conclusion.

Honest limits of this post

Our audits run a fixed set of about 45 prompts across five engines, with retrieval on and off. We have not published a run-to-run variance measurement, so when this post says single-run numbers are unstable, that is what we see in those runs and not a figure you can reproduce from anything we have released. The same applies to the referrer and personalisation observations above: they come from client analytics and audit runs we have looked at, not from a sample we have sized, dated and published. We have not benchmarked commercial tools against each other and will not, because a fair test would need identical prompt sets, identical run counts and identical time windows, and we do not have that, so every statement here about what dashboards do or do not show is bounded by the products we happen to have seen. Engines also change retrieval behaviour without announcement, so a check that held in July can be wrong by October. Re-run it.

FAQ

Can any tool see the prompts real users type? We know of no tool that sees user prompts, and none of the engines we checked publishes them. The closest public signal we have found is Bing's grounding queries, which are the retrieval phrases the AI generated internally, not what the person typed, and Microsoft describes the data as a sample rather than a complete count.

Why does the same prompt give different answers in a tool and in my browser? Your browser session carries login state, history, location and often a different model routing decision. A tool deliberately runs without those so results are reproducible. Both are correct, and they are measuring different things.

Is Search Console enough on its own? It is necessary and not sufficient. Google includes AI Overviews and AI Mode inside overall Web search data and, since June 2026, also reports their impressions separately, but it registers a click only when a link to an external page is followed, so mentions that generated no click leave no trace.

What is the cheapest honest setup? A prompt list you wrote from real sales calls, five runs each across two or three engines, the answer text saved verbatim, and a quarterly look at your access logs by user agent. The discipline matters more than the software.

Sources

Related reading

How to measure AI visibility without fooling yourselfShare of voice in AI answers, and its limitsRented visibility vs owned visibility in AI searchMicrosoft Copilot and the Bing index: what it means for visibilityVariance, sample size and the honest error barHow to track AI referral traffic in your analytics