Blog · Running the work

Choosing an AI visibility tool

Published 8 October 2026 · Updated 12 October 2026 · 8 min read

Choosing an AI visibility tool
Photograph: Voxogos / unsplash

The short answer: Choose on method, not on engine count. Ask how the tool gets its numbers. In every tool we have been shown a demo of, the number comes from one of three places: simulated prompts, bought panel data, or your own server logs. Make the vendor state runs per prompt, region, login state and whether retrieval was on. Then subtract what your own free first-party reports already show, and pay only for the gap that is left.

Most buying processes for this category go wrong in the first meeting, because the comparison is drawn on features. Engine coverage, prompt volume, a share-of-voice chart. None of that tells you whether the number on screen would survive being re-run tomorrow.

Start by pricing the free half

Before any trial, find out what you already have. Google's generative AI performance reports in Search Console were announced on 3 June 2026 and reached all sites worldwide on 31 August 2026. They give impressions, the URLs that appeared, country, device (Search results only) and dates down to hourly granularity for AI Overviews, AI Mode and the generative features in Discover. That is first-party data about Google's answer surfaces, from Google, at no cost.

Analytics covers a second slice. GA4 ships a default channel group called AI Assistant, which fires when the medium is ai-assistant, set automatically when the referrer matches Google's list of assistants. Google's own definition excludes AI Overviews and AI Mode, which land in Organic Search, so this channel is about assistant referrals and not about Google's answer surfaces. Imperfect, but free, and it tells you whether anybody is arriving.

A third free source sits with the other search engine. Microsoft's assistant leans on the Bing index, which makes Bing Webmaster Tools a free first-party source for a surface most vendors only sample, as Microsoft Copilot and the Bing index sets out in more detail.

Write those three down. Whatever remains is the honest scope of a purchase. Usually it is ChatGPT, Perplexity, Claude and Grok, where we have not found a publisher-side console of the Search Console kind as of October 2026. If one exists we have not seen it, and until it does, every number about those engines is produced by someone running prompts and counting.

The three data shapes, and what each one cannot see

Each of the products we have evaluated or seen documented is built on one of three sources, or a blend of them. We have not surveyed the whole category, so read this as a map of the shapes we have met rather than a census.

Simulated prompts. The tool runs a prompt list against engines on a schedule and parses the answers. It sees exactly what it asked, and nothing else. It cannot tell you what real buyers typed, and its prompts are its own invention until you replace them with yours.

Panel or clickstream data. The tool buys aggregated browsing data to estimate real prompt volume. It sees demand shape, which simulation cannot. It is thin on small markets, thin on logged-in enterprise usage, and the panel composition is rarely published.

Crawler and log analysis. The tool reads your access logs and reports which AI agents fetched which pages. This is the only one of the three grounded in your own servers, and the only one that catches an access failure the day it starts.

Each shape answers a different question. A tool built on the first will never explain a traffic drop. A tool built on the third will never tell you what the answer said about you. If a vendor cannot say in one sentence which of the three their headline number comes from, that is the finding.

The sampling questions that decide everything

A visibility percentage is a sample statistic. Sample statistics have error bars, and this category is unusually noisy because the engines are stochastic and personalised. Ask five things, and get the answers in writing.

  1. Runs per prompt per period. One run per prompt per week is a coin flip plotted as a

    2. trend. Ask how many runs sit behind each point, because a single run cannot separate a real change from ordinary engine variance, and we have no published figure for how many runs are enough.

  2. Retrieval on or off. An answer written from model weights and an answer written from

    3. freshly fetched pages are different experiments. Mixing them produces a number that belongs to neither.

  3. Region and language. Many engines route by location. A Bengaluru query and a New York

    4. query are not the same query, and a single global score hides that completely.

  4. Logged in or logged out, and with what history. Several engines personalise answers,

    5. and nobody outside those engines can measure how much that moves citations. Running logged out is defensible, but it is a choice, and the vendor should say which one they made.

  5. What counts as a mention. Brand named in prose, brand named with a link, domain cited

    6. in the source list. These are three different metrics and they move independently.

A vendor who answers all five plainly has thought about measurement. A vendor who answers none is selling a chart.

A test to run during the trial

Here is a check that takes an afternoon and has settled several of these decisions for us.

Pick ten prompts your buyers really ask. During the trial, run each of them through the tool, then run the same ten yourself, by hand, in a clean browser session, three times each. Record whether you were named, whether the claim was true, and which domains were cited.

You are looking for two things. First, agreement: does the tool's answer for a given prompt resemble what you saw, or is it reporting a world you cannot reproduce? Second, spread: count how many of your thirty manual runs returned a different citation set. If the spread across your own runs is wider than the week-on-week movement the tool is charting, the chart is measuring noise. That finding is worth more than any feature matrix.

The self-check lists ten of these tests you can run without buying anything, and it is a reasonable baseline to hold a trial against.

A second check applies to log-based claims. Some engines publish their crawler identities and IP ranges, which makes that agent's traffic verifiable rather than inferred. OpenAI is one. Counting the prefix entries in OpenAI's three published JSON files on 8 October 2026, the live-browsing agent ChatGPT-User listed 234 IP prefixes, with a creation time stamped the previous day, the search crawler OAI-SearchBot listed 39, and the training crawler GPTBot listed 18. Those counts are a single reading on a single day and the files change. Where an engine publishes such a file, a tool reporting that engine's crawler activity should be verifying against it rather than trusting the user-agent string, which anyone can spoof. Ask them which engines they verify this way, and which they only infer.

Buying one tool instead of four

Sprawl in this category is usually an accident of sequencing. Someone buys a prompt tracker in a panic, someone else buys a crawler monitor, a content team buys an optimiser, and a year later three dashboards disagree about the same brand and nobody can say which is right.

Two rules keep that from happening. Name the decision each tool informs before you buy it, and if no decision changes based on its output, do not buy it. Second, keep the evidence layer in something you own, because a tool subscription is rented capability in exactly the sense described in rented visibility vs owned visibility. Your prompt list, your run records and your fix log should survive a vendor change.

Timing matters too. Buying tooling in week one is usually premature, since the first phase of work is access and identity, both of which are diagnosed from logs and your own pages. The first 90 days of AEO work puts measurement after the first round of fixes for that reason. If you are still deciding who runs this at all, the tool question is downstream of in-house or agency for AEO, because an agency often brings its own licence and the in-house version has to budget for one.

Honest limits

We cannot tell you which tool is most accurate. We are not aware of a shared public benchmark with an open prompt set, and a comparison run by the party being compared is not independent evidence either way. The three-shapes taxonomy above is our own, drawn from the tools we have been shown rather than from a survey of the market, so a product built some other way would not be wrong, only absent from our map. The crawler counts are one reading of OpenAI's files on one day, they say nothing about engines that publish no such file, and absence from our logs or from a published list is not proof that a thing is not happening. We also do not know how stable any of these methods are over a year, since engines change retrieval behaviour without notice.

FAQ

Do I need an AI visibility tool at all? Not to begin. A prompt list, a spreadsheet and a disciplined weekly run cover the first phase. Tools earn their price when repeating the manual version costs more than the licence.

What is a fair prompt count to start with? Forty to sixty prompts that map to real buying questions beats several hundred generated ones. Our own audit uses a set of about 45 across five engines, run with retrieval on and off, because that is the smallest set that has given us arguable findings.

Why do two tools report different numbers for the same brand? Different prompts, different run counts, different regions, different definitions of a mention, and genuine engine volatility. Treat large gaps as a prompt-and-method difference until someone shows you both methods side by side.

Should I pick the tool with the most engines? Engine count is cheap to add and tells you little. Depth of sampling on the two or three engines your buyers actually use is worth more than shallow coverage of ten.

Sources

Related reading

The first 90 days of AEO workIn-house or agency for AEORented visibility vs owned visibility in AI searchMicrosoft Copilot and the Bing index: what it means for visibilityGovernance, and one source of truth for brand factsWhen the model makes something up about you