Why an AI answer changes between two identical runs

The short answer: Three separate things move, and only one of them is the model being creative. The arithmetic inside the server is sensitive to how many other people are asking at that moment. The retrieval step can fetch a different set of pages from one run to the next. And the platform itself changes under you on its own schedule, usually without notice. Two of the three survive even when the temperature is set to zero.

That last point is the one that catches people out. Set temperature to zero, pin a seed, send the same bytes twice, and you can still get two different answers back. The reason is not mysticism. It is floating point arithmetic and server load.

The arithmetic reason, with a number attached

Addition on a GPU is not associative when the numbers are stored in sixteen bits. Add three values in a different order and you can land on a different result in the last bit. That is fine in isolation. It stops being fine when the model is choosing the next token from a probability distribution, because a last-bit difference can flip which token wins, and every token after that one is conditioned on the flip.

What changes the order of addition? Batch size. Inference servers group concurrent requests into batches, and the reduction kernels inside the model split their work differently depending on how large that batch is. Your request does not control the batch. The other people hitting the same endpoint at the same second do.

Thinking Machines Lab published a direct demonstration of this on 10 September 2025. They sampled one prompt a thousand times at temperature zero against the open-weight model Qwen3-235B-A22B-Instruct-2507 and got 80 distinct completions, the most common of which appeared 78 times. Every one of the thousand was character-for-character identical for the first 102 tokens. Divergence started at the 103rd token and then compounded. After they replaced three kernels, RMSNorm, matrix multiplication and attention, with batch-invariant versions, all one thousand completions came back identical.

Their timing test is a separate and smaller experiment, one GPU serving Qwen3-8B with a thousand short sequences: 26 seconds in the default configuration, 55 seconds with the deterministic kernels, and 42 seconds once the attention kernel was improved. That is a 1.6x to 2.1x slowdown, a ratio we worked out from their table (42 and 55 seconds against 26), on code they say they did not optimise. No hosted engine publishes whether it runs anything of this kind, so read the cost as a reason the technique is not free, not as evidence that nobody ships it.

A paper on numerical sources of nondeterminism, submitted 11 June 2025 and revised 24 October 2025, puts a size on the downstream damage for one model. Changing only the evaluation batch size, the GPU count or the GPU version, with greedy decoding and bfloat16 held constant, moved the accuracy of DeepSeek-R1-Distill-Qwen-7B by up to 9 per cent and its response length by around 9,000 tokens on the benchmarks the authors ran. Same weights. Same prompt. Different hardware queue. Whether the effect is that large on a frontier model behind a commercial API is not something that paper measures, and not something we can measure from outside it.

So when an API offers you a seed, read that vendor's own wording rather than a general rule. OpenAI's cookbook recipe for the seed parameter, dated 6 November 2023 and now carrying an archived notice, promises a best effort to sample deterministically and pairs it with a system fingerprint that signals when the backend configuration changed. It states in as many words that it does not assure determinism, and that responses can differ even when the request parameters and the fingerprint both match. That is one vendor describing one parameter on two named models. Other engines word it differently, or expose no seed at all, so check the API you are calling.

Retrieval moves even when the model does not

The second mechanism has nothing to do with sampling. Before the model writes anything, the system turns your question into search queries, runs them, and keeps a handful of passages. That step has its own variance. We cannot tell you whether it is larger than the token-level kind, because none of the engines we test expose the passages they considered and discarded.

Google's documentation on AI features says that AI Overviews and AI Mode may use a query fan-out technique, issuing multiple related searches across subtopics and data sources, and that further supporting pages are identified while the response is being generated. That is Google describing Google. No other engine we use publishes an equivalent account of its own retrieval step, so do not assume the same machinery sits behind the rest of them.

The worked example that follows is ours, not Google's. If the candidate pool is assembled fresh each time, a page that ranked eleventh for one of those sub-queries this morning can rank ninth this afternoon, and ninth is inside the window while eleventh is not. The same Google documentation says that AI Mode and AI Overviews may use different models and techniques, so the set of responses and links they show will vary, and that indexing and serving are not assured even for a page that meets every requirement.

This is part of why two engines answering the same question cite different pages. How each AI engine picks the sources it cites walks through the per-engine differences, and how Google AI Overviews chooses its sources covers the fan-out behaviour in more detail. The practical consequence: if you were marginal in the candidate set, you will appear in some runs and not others, and nothing about your site changed between them.

Platform drift, the part that does not average out

The third mechanism is the one people forget to budget for. Platforms update model versions, re-tune inference stacks, change the system prompt, roll out a tool, and swap which search backend serves the retrieval step. Some of that reaches a release note. Much of it does not, and from outside we cannot tell which is which. All of it shifts the baseline.

Drift is different in kind from the first two. Token-level randomness averages away if you run the prompt more times. Retrieval variance looks like it mostly averages away too, though that is an assumption we have not been able to test directly. Drift does not, because it is a step change in the ground truth, not noise around a fixed value. Running a prompt a hundred times on Tuesday tells you nothing about what it will do on Friday if the model version changed on Wednesday night.

That distinction is the whole argument for measuring on a cadence rather than measuring hard once. It is also why a visibility number without a date and a model version attached is close to meaningless.

A check you can run this week

Thirty minutes, no tooling:

1. Pick one prompt a buyer would actually type. Not your brand name.
2. Run it ten times in ten fresh sessions on one engine, within an hour. Record whether you

were named, and list every cited domain.

3. Count how many of the ten named you. That fraction is your signal.
4. Count how many distinct domains appeared across all ten runs, and how many appeared in

every single run. The ones in all ten are the stable backbone of that answer. The ones appearing twice or three times are the churn.

5. Repeat the identical exercise seven days later and compare the backbone, not the fraction.

If your brand sits in the churn band, the fix is corroboration, not copy. If the backbone itself turned over between the two weeks, you have watched platform drift happen, and you should stop reading any single-run report as evidence. Our self-check has nine more tests in the same spirit.

What this means for the number in your report

A single run is an anecdote with a screenshot attached. The question of how many runs you need before a percentage means anything is a sampling question with a real answer, and how many runs per prompt before a number means anything works through the arithmetic. The short version: the smaller the change you want to detect, the more runs you need, and detecting a five point move needs considerably more runs than a single daily check provides.

One thing worth separating out. Volatility in which sources get cited is not the same as volatility in what gets said about you. The second is more damaging and more fixable. If three of ten runs describe your pricing wrong, the first place to look is your own site and structured data for a contradiction, rather than writing it off as sampling noise. That is where we have found the cause in client work, and it is not a measured base rate. Our audit finds that class of problem by running the prompt set with retrieval on and retrieval off and comparing the two. FAQPage and HowTo after the rich-result changes covers one place those contradictions commonly hide.

Honest limits

We cannot separate retrieval variance from sampling variance inside a hosted product, because none of the engines we test expose the candidate passages they rejected. We infer it from citation overlap across runs, which is indirect, and our claim that retrieval variance averages out over repeated runs is an assumption rather than something we have measured. Every number in this post was measured on open-weight models running on hardware the researchers controlled, Qwen3-235B-A22B-Instruct-2507 and DeepSeek-R1-Distill-Qwen-7B, and we have no way to confirm that the same magnitudes hold inside the commercial endpoints you actually query. We re-run prompt sets for clients on a monthly cadence and the numbers move, but we are not publishing a method or a sample size for that here, so treat it as our experience and not as a measurement. We also cannot predict drift, which is why we would treat any figure quoted for a single engine without a date and a model version attached as unverifiable.

FAQ

Does setting temperature to zero make the answer repeatable? It removes deliberate sampling randomness and nothing else. Batch-size effects inside the inference server and variation in what retrieval fetched can both survive temperature zero, which is why a thousand runs of one prompt at temperature zero produced 80 different completions in the September 2025 experiment on Qwen3-235B. That was one open-weight model on hardware the researchers controlled, so treat it as a demonstration of the mechanism rather than as a figure for the engine you use.

Is the variation a sign that something is wrong with my site? Usually not, if the variation is only about which sources get cited. Being named in four runs out of ten suggests you sit at the edge of the candidate set. If what is said about you changes run to run, that is a different problem, and it is worth checking your own pages for contradictory facts before you blame the engine.

How many runs do I need before a number is trustworthy? More than one, and the answer depends on how small a change you want to detect. Ten runs will tell you roughly whether you are present, absent or marginal. Detecting a few points of movement reliably needs considerably more, plus the same prompts and the same engine each time.

Should I report an average or a range? Report both, with the date and the engine. An average hides the thing a reader most needs to know, which is whether you were named nine times out of ten or five times out of ten.

Sources

All four were opened and checked on 12 October 2026.

• Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, 10 September 2025. Source

for the 1000 completions, the 80 distinct results, the 102 identical tokens, and the 26, 55 and 42 second timings on Qwen3-8B from which the 1.6x to 2.1x range is calculated: thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference,

arXiv:2506.09501, submitted 11 June 2025, revised 24 October 2025. Source for the 9 per cent accuracy and 9,000 token figures on DeepSeek-R1-Distill-Qwen-7B: arxiv.org/abs/2506.09501

• OpenAI developer cookbook, reproducible outputs with the seed parameter, 6 November 2023,

marked archived. Source for the best-effort wording, the system fingerprint, and the statement that determinism is not assured: developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter

• Google Search Central, AI features and your website. Source for query fan-out, for the

statement that AI Mode and AI Overviews may use different models and techniques so their responses and links will vary, and for indexing and serving not being assured: developers.google.com/search/docs/appearance/ai-features