Blog · Engines

What a model update does to your visibility

Published 28 August 2026 · Updated 12 October 2026 · 9 min read

What a model update does to your visibility
Photograph: Miha Meglic / unsplash

The short answer: A model update can re-draw the shortlist. The engine keeps answering the same questions, but it may retrieve a different set of pages and name fewer brands per answer. Some of what you see the week after an update is a real change in how you are selected. A good deal of it is ordinary run to run variance that was always there and only became visible because somebody started watching.

Both things are true at once, and telling them apart is most of the job.

Two different events get called a model update

The first kind is a swap of the model that writes the answer. A new default lands, an old one is retired, and the prose on top of the retrieved passages is produced by different weights. OpenAI publishes minimum notice periods for its API: at least six months before a generally available model is retired, at least three months for specialised variants, and as little as two weeks for anything with "preview" in the name, unless safety or compliance forces a faster timeline. That is one provider's published commitment, and it covers API models. It does not bind the consumer assistants, and we have not found an equivalent published schedule for them. The API dates you can put in a calendar. The chat interface dates you cannot.

The second kind is a change in retrieval or in the surface, and it rarely gets announced as a model change at all. Google's own documentation says AI Mode and AI Overviews use query fan-out, issuing several related searches across subtopics before writing, and that the two surfaces "may use different models and techniques, so the set of responses and links they show will vary". On Google's own description, a change to that fan-out would change which pages are eligible to be cited before any weights move. Google does not publish when it tunes fan-out, so this is a mechanism you can reason about, not an event you can date. Nor do we know that the other engines work this way, because they document far less.

The second kind is the easier one to miss, because the public conversation is always about the first. We cannot tell you which kind is more common, because only the first is announced.

Most of the swing you measured was already there

Before you attribute anything to an update, you have to know how much your numbers move when nothing changes. They move a lot.

The cleanest public demonstration of why is a study of inference nondeterminism published on 10 September 2025. It ran one prompt one thousand times against one open weights model at temperature zero and got 80 distinct completions. The first 102 tokens were identical in every run. Divergence started at token 103, where 992 runs said one thing and 8 said another. The cause is not randomness in the sampler. It is that the numerical result of the kernels depends on the batch the request happens to land in, and batch size varies with server load.

Note what that does and does not establish. It was one prompt, one model, one server setup, not the hosted assistants you are tracking, and we have found no published account of how any of those assistants batch requests. It shows that temperature zero is not a promise of a repeated answer. It does not give you a variance figure for ChatGPT or for AI Mode, and we have not seen one published.

Stack a retrieval layer on top of that and the spread widens. A paper on what AI benchmarks leave unmeasured, posted 6 August 2026 and revised 9 October 2026, ran 401 prompts from two public benchmarks three times each across ChatGPT's chat interface and OpenAI's API, with and without web search, 4,812 responses in total. Repeated runs of the same prompt disagreed on up to 21 per cent of prompts. Turning web search on reduced accuracy by up to 8 percentage points and changed which sources the answer was grounded in. Those were safety and bias benchmark items, not brand or category questions, and the audit covers one model family, so read it as evidence that repetition and search condition matter rather than as a variance rate for your prompt set.

So a brand that appears in four of five runs on Monday and three of five on Friday may not have lost anything you can act on. Neither study measured brand mentions, so neither hands you a threshold. The rule we work to is our own, and it is a convention rather than a measurement: with five runs per prompt we do not treat a shift of one run as a signal, and we do not act until a shift holds across repeated runs on more than one engine. If your reporting treats a single run as a reading, your dashboard will manufacture an update every week.

What a real update actually changes

When the shift is genuine, the pattern we look for first is compression rather than a targeted demotion: answers get shorter, name fewer options, and lean on a narrower set of domains. We cannot show you that this is what updates usually do, because we have no way to observe the shortlist and have not found an engine that publishes one. It is the shape we check first, because it explains a drop with nothing having changed on your side. Nobody removed you. The list you were on got shorter.

That is why two sites with similar pages can come out of the same week in opposite places. Our working explanation is corroboration: the site whose claims are repeated on sources the engine already retrieves gives the selection more to hold on to than the site whose only evidence for its own positioning is its own marketing page. We cannot see inside that selection, so treat it as a hypothesis we act on, not as documented engine behaviour. What we do assume is that the mechanics of selection are much the same before and after an update, which is the argument for how each AI engine picks the sources it cites being the thing you study rather than the release notes. On the Google side specifically, the account of how Google AI Overviews chooses its sources sets out the reason fan-out widens the eligible set and narrows the cited one in the same step.

There is a second failure mode worth naming. Compression also compresses the context around your name. A brand that used to get a clause of qualification can be reduced to a category label, and if that label is wrong you now have a different problem. If the sentence next to your name went from accurate to roughly accurate after an update, treat it the way you would treat the model making something up about you, not as a visibility problem.

The check to run, in order

Run this before and after, not only after.

  1. Fix a prompt set and leave it fixed. Changing the questions between runs makes every

    2. comparison meaningless.

  2. Run each prompt at least five times per engine, on at least two engines, with retrieval on

    3. and retrieval off. The off runs tell you what the weights believe without the web. Our own audit is the same shape at a larger size, about 45 prompts across five engines with retrieval on and off, and it is still a sample rather than a census of how an engine answers.

  3. Record four fields per run: whether you were named, whether the claim was true, which

    4. sources were cited, and the model label the interface shows. That last field is the only evidence you will have about which version answered you.

  4. Compare the distribution, not the best run and not the worst. Report it as three of five,

    5. not as a score.

  5. Keep the answer text. When something changes, the diff between two stored answers tells

    6. you more than any aggregate number.

The self-check is a shorter version of this you can do in an afternoon with nothing but a spreadsheet.

What you can do about it

You cannot schedule a model release and you cannot petition for a different shortlist. Three things stay under your control.

Keep access boring. A page an engine cannot fetch is a page it cannot cite, and that failure is invisible in every report that only counts mentions.

Keep identity tight. One name, one description, one set of facts, repeated consistently across your site, your schema and your profiles, so that a shorter answer has less about you to get wrong.

Keep corroboration going. Third party sources that repeat your claim are the part of the system that does not reset when a model does, which is the whole case for digital PR when the audience is a model.

One new instrument is worth setting up. On 3 June 2026 Google announced a Search Console control over whether a site may appear in and ground responses in its generative AI features, along with reporting on impressions and on which pages appear in AI responses. It began as a test with a subset of sites in the UK, and the same announcement, updated 31 August 2026, says it has since rolled out to all websites worldwide. That is first party data about one surface. We have not found anything comparable published for the other engines.

Honest limits

We cannot tell you how large a swing has to be before it is real for your category. The two studies cited here measured benchmark prompts on one model family and one open weights model, not brand mentions in the engines you care about, and our own working threshold of more than one run in five is a convention we have not published a measurement for. The compression pattern and the corroboration explanation above are what we look for and act on, not documented engine behaviour, because we have not found an engine that publishes how its shortlist is cut. We also cannot connect a mention to revenue with any confidence, since these surfaces send little usable referral data. And an update that changes nothing for you this quarter may change a lot next quarter, because the retrieval layer moves on its own schedule.

FAQ

How long after a model update should I wait before measuring? Give it two weeks of repeated runs. Rollouts are often staggered: Google's June 2026 Search Console control started with a subset of sites in the UK before reaching all sites at the end of August. While something is rolling out gradually, a single sample mixes old and new behaviour.

Can I pin a model so my measurements stay stable? Through an API, often yes, by requesting a dated snapshot. In the consumer interfaces we test, we have not found a way to pin a version, and the model label the interface shows is the only record we get. That gap is why API readings and chat interface readings disagree, and why both are worth collecting.

Does a drop in mentions mean I did something wrong? Not by itself. If answers got shorter and named fewer brands, everyone's count falls at once. Check whether your share of the brands still named moved before you rewrite anything.

Should I rewrite my pages every time an engine changes? No. Access, identity and corroboration are the layers least likely to be invalidated by a model change, because they live outside the model. The content that keeps earning citations is usually the content that answered a real question clearly, which does not expire on a release schedule.

Sources

Related reading

How each AI engine picks the sources it citesHow Google AI Overviews chooses its sourcesDigital PR when the audience is a modelWhen the model makes something up about youHow Perplexity picks sources, and how to track the URLsAds in AI answers: what they buy and what they do not