Variance, sample size and the honest error bar

The short answer: An honest error bar on an AI visibility score is computed across prompts, not across answers. Pool every answer into one binomial interval and the bar comes out too narrow, because repeat runs of the same prompt are not independent observations. In a simulation we ran on 12 October 2026, 45 prompts run ten times each, the naive bar was about 1.6 times too small: 40 percent plus or minus 4.5 points on paper, and plus or minus 7.2 points in fact. Those are simulated numbers, not a measurement taken on live engines.

That gap is the whole subject. A score with a dishonest bar is worse than a score with no bar, because it invites a decision the data cannot support.

Variance lives in three places, and only one of them is cheap

Pull apart any portfolio number and you find three separate sources of wobble stacked on top of each other.

Run variance is the engine answering the same question differently on two consecutive calls. Prompt variance is the fact that your 45 tracked prompts are a sample of the thousands of questions a buyer might type, and a different 45 would have produced a different score. Occasion variance is the index, the model weights and the retrieval layer, any of which can move underneath you between Tuesday and Friday.

Repeat runs attack the first of those and leave the other two untouched. This matters because of what dashboards in the category tend to print: a score, often to one decimal place, with no prompt count and no interval beside it. We cannot see how any given tool computes its number, so we do not claim the clustering was ignored. We claim the narrower thing, that a score published without its sample size cannot be checked by the person reading it. The point is developed further in how to measure AI visibility without fooling yourself, and it is the reason a share of voice figure carries less precision than one decimal place suggests.

What pooling does to the bar

Here is the mistake, stated plainly. You run 45 prompts ten times each, collect 450 answers, count 180 mentions, and compute a binomial interval on 180 out of 450. The arithmetic is correct. The assumption underneath it is not, because those 450 answers came from 45 clusters, and answers inside a cluster are correlated. A prompt your brand owns tends to return a mention on most runs. A prompt you have lost tends to return nothing on most runs. That correlation is exactly what the pooled interval assumes away.

We simulated this on 12 October 2026, in a script written for this post rather than on client data. Per-prompt mention rates were drawn from a Beta(2,3) distribution averaging 40 percent, answers drawn as Bernoulli trials at each prompt's rate, each design repeated 20,000 times. We compared the standard deviation of the simulated score against the naive binomial standard error a dashboard would print. Figures are in percentage points and rounded to two decimals, so a re-run on a different seed moves the last digit:

| Design | Answers | Naive bar | Honest bar | Understated by | | --- | --- | --- | --- | --- | | 45 prompts, 1 run | 45 | 7.30 | 7.28 | 1.00x | | 45 prompts, 3 runs | 135 | 4.22 | 4.91 | 1.16x | | 45 prompts, 10 runs | 450 | 2.31 | 3.67 | 1.59x | | 135 prompts, 1 run | 135 | 4.22 | 4.21 | 1.00x |

Two things fall out of that table. The lie gets worse as you run more repeats, because every extra run adds to the denominator without adding information about the prompt population. And a single run across 135 distinct prompts beats three runs across 45, at exactly the same cost, with an honest bar instead of a flattering one.

This is not a new result in statistics. Clustered standard errors were built for this shape of data, and one published analysis of language model evaluations measured the effect on three public benchmarks, scoring Anthropic models: on DROP the clustered standard error came out 3.05 times the naive one, on MGSM 1.88 times, on RACE-H 1.10 times (Table 4 of the paper cited below). Note the spread. The multiplier is a property of how clustered that particular dataset is, not a constant you can borrow, and none of those three is a brand prompt set run against an answer engine.

Turning the temperature down makes it worse

A tempting shortcut is to set sampling temperature to zero, watch the run variance collapse, and declare the measurement clean. The paper cited below argues against it in its Section 3.3, with two worked examples that are sharp enough to be worth repeating. Both are constructed examples, not measurements.

Take a binary outcome whose per-prompt probability is spread uniformly between 0 and 1 at temperature one. The variance of those per-prompt means is 1/12. Drop to temperature zero and each prompt rounds to a hard yes or no, which is a coin flip with probability one half, and the variance becomes 1/4. The cheap variance went away and the expensive kind tripled. In the second example, where the spread runs from 1/3 to 1, the rounding also moves the expected score from 2/3 to 3/4, so the estimator comes out biased as well as noisier.

Whether the same thing happens to your prompt set depends on how your per-prompt rates are spread, which we have not measured on live engines. The arithmetic shows it can happen, not that it must. What does hold by definition is the shape of the problem: variance you can shrink with more calls is not the variance that decides whether a portfolio score is true, because that one sits in which prompts you chose, and determinism does not touch it.

Forty-five prompts, forty-five chances to be fooled

Most reports do not stop at a portfolio number. They break it out per prompt, mark the movers in green and red, and hand the list to someone who will act on it. That is where a second, quieter error enters.

Test 45 prompts week over week at the usual 5 percent threshold and, with nothing at all changing, the chance that at least one prompt crosses it by luck alone is 90.1 percent, and the expected count of false movers is 2.25 a week. That is one line of arithmetic, 1 minus 0.95 to the power 45, and it assumes the 45 tests are independent and each carries a true 5 percent false positive rate. Prompt results are not perfectly independent, so treat it as the order of magnitude rather than the figure. Even halved, a monthly report with three highlighted changes and no correction for multiple testing is reporting some noise as findings.

The fix is cheap. Decide in advance which five or six prompts a decision depends on, apply a correction across the family you are testing, and treat the rest as a background series rather than a list of findings. Our own sample audit report marks prompts as won, lost or contested rather than ranking them by weekly delta, for this reason.

The recipe for a bar you can defend

Four steps, and none of them need a tool.

Collapse first. For each prompt, average its runs into one number between 0 and 1. You now have 45 observations, not 450, which is the honest sample size.

Compute the standard error across those 45 prompt-level scores, not across answers. Multiply by 1.96 for a 95 percent bar. If you prefer an interval on a single prompt, the Wilson formula is four lines of code, and the NIST handbook recommends it over the textbook normal approximation for proportions.

Report the prompt count next to the score. "41 percent, n equals 45 prompts, plus or minus 14 points" is a sentence a sceptical executive can argue with. "41 percent" is not. The 14 points there is the bar our simulated 45-prompt design produced; yours follows from your own spread, not from ours.

Compare like with like. When you are testing a change, difference each prompt against its own earlier score and put the bar on the differences. Pairing cancels the prompt-sampling variance that usually dominates, so when the two measurements are positively correlated across prompts, a before-and-after on a fixed list detects a smaller real effect than two independent measurements of the same size.

None of this is specific to answer engines. It is ordinary survey arithmetic applied to a surface that happens to be new, which is the recurring theme of what answer engine optimisation actually is and of the naming argument in AEO vs GEO vs SEO.

Honest limits

Every number in this post is either arithmetic or simulation. The design table comes from a model of prompt behaviour, Beta(2,3) rates with Bernoulli answers, not from engine logs, so the 1.6 times understatement is what that model implies and not a figure we have measured on ChatGPT, Perplexity, Google AI Overviews, Gemini or Copilot. We have no published figure for the intra-cluster correlation in real brand prompt sets, and the true design effect depends on how polarised a given set is: a set full of branded prompts will behave differently from a set of unbranded comparison queries. The three benchmark ratios we quote range from 1.10 to 3.05 on reading and maths evals, which is evidence that clustering matters and not evidence of how much it matters for you. We also cannot see inside any visibility tool, so nothing here says what other tools compute; it says what a score printed without a prompt count lets a reader verify, which is nothing.

Occasion variance is the harder gap. We can bound run and prompt variance with arithmetic. We cannot yet separate a genuine week-over-week move from an index refresh without a control prompt whose answer should not change, and we do not have enough months of control data to publish a number on how often that control drifts.

FAQ

Why is my sample size the number of prompts and not the number of answers? Because answers to the same prompt are correlated. Treating 450 answers from 45 prompts as 450 independent draws inflates the denominator without adding information about the prompt population, and the interval shrinks for a reason that is not real.

Does running more repeats ever help? Yes, for one prompt you are about to make a decision on. Repeats pin down that prompt's rate. They do nothing for a portfolio score, where the dominant uncertainty is which prompts you chose to track.

What is a reasonable error bar to expect? In our simulated 45-prompt design, measured once each on a score near 40 percent, the 95 percent bar was about plus or minus 14 points. Your own set can be tighter or wider depending on how spread its per-prompt rates are. A much narrower bar on the same budget is worth questioning, and the first thing to ask is whether answers were pooled instead of collapsed to one score per prompt.

Should I set temperature to zero to reduce noise? No, unless zero is the setting you actually care about measuring. It can shift variance into the part you cannot shrink and bias the estimate at the same time.

Sources

• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, arXiv:2411.00640, submitted 1 November 2024, checked 12 October 2026. Clustered versus naive standard error ratios of 3.05 (DROP), 1.88 (MGSM) and 1.10 (RACE-H) are Table 4, computed on those benchmarks using Anthropic models; the temperature argument, the 1/12 to 1/4 variance example and the 2/3 to 3/4 bias example are Section 3.3: arxiv.org/abs/2411.00640
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.4.1, confidence intervals for a proportion, source for the Wilson interval being preferred to the normal approximation, checked 12 October 2026: www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm
• The four-row design table is simulated, not observed: 20,000 iterations per design, 45 or 135 prompts, per-prompt mention rates drawn from Beta(2,3) with mean 0.4, answers drawn as Bernoulli trials at each prompt's rate, run 12 October 2026. The 90.1 percent false-mover figure is arithmetic, 1 minus 0.95 to the power 45, assuming independent tests at a 5 percent threshold.