How many runs per prompt before a number means anything

The short answer: For one prompt, five runs tell you whether a mention is possible, and about thirty runs pin the rate to roughly plus or minus 17 points. Single-digit precision on one prompt needs several hundred. For a portfolio number, stop there and spend the budget sideways instead: at a fixed call count, more prompts run once each beats fewer prompts run many times.

That is the whole answer, and most of the argument that follows is about why the second half matters more than the first.

Why a run is a coin flip and not a reading

An engine answering the same question twice is not misbehaving. The clearest published account of why is a September 2025 write-up from Thinking Machines Lab, which traces it to batch invariance: floating point addition is not associative, common GPU kernels give slightly different sums depending on how many requests share the batch, and batch size depends on who else is using the service at that second. That paper argues the same mechanism applies to nearly all hosted inference endpoints, including ones served from CPUs and TPUs. We cannot verify it from outside a given engine, because the batch conditions inside a hosted engine are not visible to us.

The size of that effect has been measured once, carefully, on one model. In that experiment the same prompt went to Qwen3-235B-A22B-Instruct-2507 a thousand times at temperature zero and produced 80 distinct completions. All thousand agreed for the first 102 tokens, then 992 of them said one thing and 8 said another, and the texts fanned out from there. Rewriting the kernels to be batch invariant collapsed all 1,000 back to a single output. That is an open weights model on the lab's own server, not ChatGPT, Perplexity or AI Overviews, so read it as a demonstrated mechanism rather than a measured rate for the engines you track.

So a single run gives you one draw from a distribution you have not seen. On the Wilson interval below, a brand absent once is compatible with a true mention rate anywhere up to 79 percent, and present once is compatible with anything from about 21 percent upward. This is the mechanism behind why an AI answer changes between two identical runs, and it is why a seed parameter does not rescue you where one is offered at all. OpenAI's own documentation for reproducible outputs in the Chat Completions API says determinism is best effort and not guaranteed, and that responses can still differ when the seed, the parameters and the backend fingerprint all match. The consumer answer engines expose no such control to begin with.

The arithmetic for one prompt

Treat each run as a biased coin. Count mentions out of runs, then put a Wilson interval around the result. The formula is in the NIST statistics handbook and takes four lines of code. Here is what it gives at a true rate near 40 percent, computed with that formula:

| Runs | Observed | 95 percent interval | Width | | --- | --- | --- | --- | | 1 | 0 of 1 | 0 to 79 | 79 points | | 5 | 2 of 5 | 12 to 77 | 65 points | | 10 | 4 of 10 | 17 to 69 | 52 points | | 30 | 12 of 30 | 25 to 58 | 33 points | | 100 | 40 of 100 | 31 to 50 | 19 points | | 200 | 80 of 200 | 34 to 47 | 13 points |

Read the last column, not the middle one. At ten runs, a prompt that really sits at 40 percent and a prompt that really sits at 65 percent can produce the same count. At thirty runs you can tell a quiet prompt from a loud one and nothing finer. The precision a dashboard implies when it prints 42 percent next to a prompt is not available until the hundreds, per prompt, per engine, per day.

One useful asymmetry: zero mentions is cheaper to establish than a rate. Twenty clean runs with no mention puts the upper bound near 16 percent. That is enough to call a prompt lost and move on, which is often the only decision you needed.

Why more prompts beats more runs

Your portfolio score has two independent sources of wobble. Run-level noise is the coin flipping. Prompt-level noise is the fact that your tracked prompts are a sample of the questions buyers actually ask. Extra runs shrink only the first. Extra prompts shrink both.

We did not measure this on live engines. We simulated it in October 2026: a fixed budget of 45 engine calls, per-prompt mention rates drawn from a Beta(2,3) distribution averaging 40 percent, each run treated as an independent coin flip at its prompt's rate, and each design repeated 4,000 times. Standard deviation of the reported portfolio score across those 4,000 simulated portfolios:

• 45 prompts, 1 run each: 7.2 points
• 15 prompts, 3 runs each: 8.6 points
• 9 prompts, 5 runs each: 9.6 points

Same cost, worse answer, in every design we simulated. Tripling the budget to 135 calls across all 45 prompts gets you to about 4.8 points. Going to 450 calls gets about 3.7, because by then you are grinding against prompt sampling, which more runs cannot touch. Those figures describe the model, not the behaviour of a named engine, and they move if real prompts are less variable than Beta(2,3) assumes.

The practical rule falls out of it. Decide how many engine calls a day you can afford, then buy prompts with the budget until the prompt list genuinely covers your category, and only then add runs. The exception is a single prompt you are about to make a decision on, such as the one comparison query a sales team keeps losing. That one deserves thirty runs and an interval, not a daily single sample.

What this does to a week-on-week comparison

A report can measure carefully and still fail at the comparison. If Monday reads 41 percent on 45 single runs and Thursday reads 46 percent, the simulated standard deviation above says a gap that size is indistinguishable from noise. Two intervals that overlap are not a result.

Three things make comparisons survivable. Hold the prompt list fixed, because adding prompts mid-month moves the score by itself. Separate branded prompts from unbranded ones, since branded prompts sit near a ceiling and drag the average into looking stable while the prompts that matter move underneath. And accumulate over time rather than averaging within a day: fourteen days of 45 prompts is 630 observations of the thing you care about, and it also lets you see a step change when a model updates. The same discipline is the spine of how to measure AI visibility without fooling yourself.

The metric you choose changes the arithmetic too. Mention is a clean yes or no per run, so the table above applies directly. Citation share is a ratio over a varying number of sources, so the denominator moves as well as the numerator, which is one more reason to hold citation share vs mention share apart rather than blending them into one headline figure.

A check you can run this afternoon

Pick your five most commercially important prompts. Run each one twenty times on one engine, in one sitting, recording only whether your brand name appears. Compute the Wilson interval for each. Then do the same thing tomorrow and ask whether any interval has moved clear of yesterday's.

At twenty runs the interval around a rate near 40 percent is still roughly 40 points wide, so on the arithmetic alone you should expect most of those intervals to sit on top of yesterday's. If that is what you find, the six point move your tool reported was inside the noise the whole time. That result is worth having. It tells you to stop reading the daily line and start looking at the inputs you can actually change, crawler access, markup and corroboration. The self-check covers the ten non-statistical checks that sit alongside this one.

It also deflates a common experiment. Teams add a file like llms.txt, watch the score rise four points the next week, and conclude it worked. Four points on 45 single runs is nothing, and the separate question of whether engines read that file at all is covered in does llms.txt do anything yet.

Honest limits

This post establishes arithmetic, not engine behaviour. The interval table is the Wilson formula applied to assumed counts, and the portfolio figures come from a simulation we ran in October 2026, not from a census of live engines. Neither tells you the true mention rate of any real prompt.

Our simulation assumes runs within a prompt are independent. They probably are not, because engines cache, and two runs a minute apart may share a retrieval set in a way two runs a day apart do not. If that correlation is strong, close-together runs are worth less than the table suggests, and the case for more prompts gets stronger rather than weaker. We also chose Beta(2,3) for the spread of per-prompt rates because it is a plausible shape, not because we have fitted it to measured prompt data.

The nondeterminism mechanism is documented for one open weights model on one lab's hardware. We cannot observe batch conditions inside ChatGPT, Perplexity, AI Overviews or Gemini, so we do not claim to know that the same cause is at work there, only that repeats on the engines in our own prompt set have not come back identical. And we cannot separate model drift from sampling noise inside a single day without a control prompt whose answer should not change. We now keep two in every prompt set for that reason, and we do not yet have enough months of them to publish a figure.

FAQ

Is one run a day ever enough? For a portfolio score tracked over weeks, yes, provided the prompt list is wide and fixed. Daily single runs accumulate into a usable series. What they cannot support is a statement about one prompt on one day.

How many runs do I need to prove a fix worked? Enough that the before and after intervals do not overlap. At a rate near 40 percent that usually means a few hundred observations on each side, which you reach faster by running many prompts for two weeks than by hammering one prompt for an afternoon.

Does setting temperature to zero remove the need for repeats? No. The one published experiment we rely on here, a thousand temperature zero completions from a single open weights model in September 2025, still produced 80 distinct outputs, because the cause sits in the kernels and the batching rather than in sampling. Whether hosted engines behave the same way is something we cannot see from outside.

Should branded and unbranded prompts be averaged together? Keep them in separate buckets. Branded prompts tend to sit high and move little, so mixing them in makes a portfolio look steadier than it is and hides movement on the prompts you can actually influence.

Sources

• Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, September 2025: thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.4.1, confidence intervals for a proportion (Wilson/Agresti-Coull): www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm
• OpenAI Cookbook, reproducible outputs with the seed parameter: developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter
• Interval widths in the table are the Wilson formula from the NIST handbook applied to the counts shown, computed in session on 12 October 2026.
• The budget figures come from a simulation run in October 2026: per-prompt rates drawn from Beta(2,3), each run an independent draw at its prompt's rate, 4,000 repetitions per design. It is a model of the arithmetic, not a measurement taken from live engines.