Benchmarking your answers against competitors

The short answer: Benchmarking against competitors means running one fixed prompt set, scoring every brand that appears in each answer by the same rule, and repeating it often enough that the gaps you see survive arithmetic. The last part is the one most often left out. At 45 prompts, a mention rate near a third carries a 95% margin of error of roughly fourteen points, computed in session from the standard binomial formula, so a seven point lead over a rival is not a lead at all.

That single fact changes what the exercise is for. You are not producing a leaderboard. You are producing a measurement with error bars, and then arguing about whether the error bars are small enough to act on.

Score every brand in the answer, not just yours

A benchmark starts one step before the spreadsheet. When an engine writes an answer to "best payroll software for Indian startups", it names some set of companies. Record all of them. Your own mention rate, read alone, tells you almost nothing, because it moves with how broad the prompt set is and how generous the engine feels that week. Read against the other names in the same answers, it becomes a comparison that holds still.

Two rules make the comparison honest. Score a brand as mentioned only when it is named in the answer body, not merely present in a footnote link, and keep that rule identical for every brand including your own. Second, record the claim attached to the name. Being named with a wrong price is a different outcome from being named correctly, and a benchmark that collapses the two will congratulate you for a bad answer. The same discipline runs through how to measure AI visibility without fooling yourself.

The arithmetic nobody runs

Here is where most competitive dashboards quietly break. Mention rate is a proportion, and proportions measured on small samples wobble. Using the standard binomial formula, computed in session, a brand sitting near 33% has these 95% margins of error:

• 45 observations: about 14 points
• 135 observations: about 8 points
• 225 observations: about 6 points
• 450 observations: about 4 points

Take a worked case, the arithmetic again done in session. You appear in 14 of 45 answers, a rival in 17 of 45. On a slide that reads 31% against 38%, and somebody will ask why you are losing. Run a two proportion test on it and the z score is 0.67, nowhere near significance. Nothing happened. To detect a genuine ten point gap in that range, 35% against 45%, at 80% power and the conventional 5% threshold, you need roughly 375 observations per brand.

Observations, not prompts. A 45 prompt set run across five engines with three runs each gives 675 observations, and that is a usable benchmark. The same 45 prompts run once gives 45, and that is a conversation starter. Repeat runs are the cheapest statistical improvement available to you, because answers to the same prompt are not fixed. Google documents this for its own surfaces: AI Mode and AI Overviews "may use different models and techniques, so the set of responses and links they show will vary", and AI Overviews "often don't trigger" at all. We see the same instability in our own audit runs across the five engines we test, and we have not measured how large it is engine by engine.

Choosing the comparison set is the political part

Every benchmark embeds a decision about who counts as a competitor, and that decision moves the numbers more than any content fix will. Three sets are worth keeping separate.

The set you sell against. The names your sales team hears on calls. This is the set leadership cares about and the one most likely to flatter you, because it excludes the aggregators and the incumbents that actually crowd the answer.

The set the engine uses. Whoever the model names when asked your category question, cold, with no brand in the prompt. In the audits we have run these are often not the sales competitor list, and the ordering can change as the category question is narrowed. We have not measured that across a sample large enough to quote a figure, so treat it as a reason to run the cold prompt rather than as a published result. Measure the set the engine names, then compare it to the set you believe in. The difference is usually the interesting finding of the whole exercise.

The set that gets cited. Not brands at all, but the review sites, listicles and forums the answers draw from. You cannot outrank these, and you are not trying to. They are the supply chain, and a benchmark that ignores them explains the score without explaining the cause.

Three numbers worth comparing, and two that mislead

Worth tracking, per brand, per engine:

1. Mention rate. Share of answers in which the brand is named at all. Blunt, stable,

and the one number a benchmark can carry alone.

2. Correct mention rate. Same denominator, but only counting answers where the claim

attached is true. The gap between this and the first number is your correction backlog.

3. Position in list. For answers that name several brands, whether you appear first,

middle or last. Order carries real weight when the answer is short, and on one engine we know it often is: Pew's browsing study of 68,879 Google searches collected in March 2025 found that 12,593 of them produced an AI summary, that the median such summary ran to 67 words, and that 88% of those summaries cited three or more sources. That is Google AI summaries in one month in one country, and says nothing about the other engines.

Misleading, in the two ways teams most often get caught:

A single blended score. Rolling five engines into one visibility index hides the fact that Gemini, ChatGPT and Perplexity disagree about who the category leaders are. Engine level numbers are harder to present and far more useful, and how Gemini answers brand questions is a different problem from how a retrieval-first engine does.

Share of voice without a denominator. When your share rises because a competitor dropped out of the prompt set, nothing improved for you. Normalise against a fixed brand list or do not quote the number. The failure modes are set out in share of voice in AI answers, and its limits.

A check you can run this week

Pick ten questions a buyer genuinely asks. Run each three times on two engines, which is sixty answers and, on our own runs, somewhere around ninety minutes of clicking. For every answer, write down the brands named, in order, and whether the claim about each is true. Count. You now have an honest, small benchmark, and you know its error bars are wide.

Then do the part that pays. Take the five answers where a rival was named and you were not, open the cited sources, and ask whether your own site contains an equivalent passage that an engine could lift. Usually the answer is no, or it is buried. That failure is reproducible and fixable, unlike a percentage.

If you want to see what this looks like at full size, with the evidence attached to each finding, the sample audit report shows the shape of a benchmark that can be argued with.

What benchmarking cannot tell you

It cannot tell you why a competitor leads. Correlation between their mention rate and their backlink profile, their Wikipedia entry or their review volume is visible, causation is not, and every published estimate of what moves generative visibility comes with large confidence intervals attached. The KDD 2024 paper that introduced the GEO-bench benchmark reports that its methods boost visibility by up to 40% in generative engine responses, and says in the same breath that the efficacy varies across domains. That is a best case on a research benchmark, not a forecast for your site.

It cannot give you a clean before-and-after either, because the engines change underneath the measurement. Re-baseline every quarter, keep the old prompt set running alongside the new one for a cycle, and expect the shape of the timeline described in how long AEO takes to show up in answers.

And it cannot tell you about revenue. In the analytics we are shown during audits, traffic from these surfaces often arrives without a referrer that identifies the engine, and an answer read without a click leaves no trace at all, so we cannot see what we cannot see. The honest framing is that a benchmark measures presence in a consideration set, which is a leading indicator and not a sale.

Honest limits

The arithmetic here is the reliable part: the margins of error, the z score and the 375 observations per arm follow from the standard binomial formulas and can be recomputed by anyone. The judgements around it are weaker. The claim that the engine's competitor set differs from the sales team's comes from audits we have run, not from a counted sample, and we have no published figure for how often it holds. The same goes for how far answers move between identical runs: Google documents that its own answers vary, we observe variation on the other engines we test, and neither of those is a measured volatility rate. The ninety minute estimate for the weekly check is our own experience of clicking, not a timed study.

FAQ

How many prompts do I need for a defensible benchmark? Count observations, not prompts. Prompts multiplied by engines multiplied by runs. Around 375 observations per brand gets you reliable detection of a ten point gap. Fewer than a hundred and you should present ranges rather than positions.

Should I benchmark against the market leader or my real competitors? Both, kept in separate columns. The leader tells you what the ceiling looks like in your category. Your real competitors tell you what is winnable this quarter.

How often should a competitive benchmark be re-run? Monthly is enough for trend, weekly only if you are mid-launch. Daily re-runs mostly measure engine volatility and will have you reacting to noise.

Can I use a vendor dashboard instead of running this myself? Yes, if it shows you the raw answers behind the score. A number you cannot trace back to a specific answer on a specific date cannot be argued with, and benchmarks exist to be argued with.

Sources

• Pew Research Center, Google users are less likely to click on links when an AI summary appears in the results, 22 July 2025, browsing data from 900 US adults covering 68,879 Google searches collected in March 2025, of which 12,593 produced an AI summary. Checked 12 October 2026: www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/
• Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan, Deshpande, GEO: Generative Engine Optimization, KDD 2024, abstract reports a visibility boost of up to 40% in generative engine responses and notes the efficacy varies across domains. Checked 12 October 2026: arxiv.org/abs/2311.09735
• Google Search Central, AI features and your website, source of the quoted lines that AI Mode and AI Overviews "may use different models and techniques, so the set of responses and links they show will vary" and that AI Overviews "often don't trigger". Checked 12 October 2026: developers.google.com/search/docs/appearance/ai-features
• Margins of error (95%, p near 0.33), the two proportion z score (14 of 45 against 17 of 45) and the per arm sample size (35% against 45%, 80% power, 5% threshold) were computed in session from the standard binomial formulas on 12 October 2026.