Proving a change worked: before and after in AI search

The short answer: A before and after proves something only when three conditions hold at once. The prompt set, the engines and the run count are identical on both sides. The sample is big enough that the two numbers do not overlap once you put error bars on them. And a set of pages you did not touch moved differently over the same window. Miss any one of those and you have a story, not evidence.

The second condition is the easiest one to skip, because nothing in a screenshot of an answer shows how many runs produced it. That is the expensive part, because the fix gets copied to twenty more pages on the strength of a number that may have been noise.

What you are actually comparing

An answer engine does not give you a rank to diff. It gives you a sentence, and the thing you can count is whether your name was in it, whether the claim attached to your name was right, and which sources the engine leaned on. Those are three separate measurements and they move independently.

So fix the unit before you touch the site. The usual choice is mention rate: out of N runs of a fixed prompt set, in how many did the brand appear. Citation rate, meaning how often one of your URLs appears in the source list, is a cleaner signal for retrieval work because it is less sensitive to how the model decides to phrase things. How to measure AI visibility without fooling yourself goes through the trade between them.

One warning on the obvious shortcut. Google's documentation on AI features states that traffic from AI Overviews and AI Mode is included in overall search traffic in the Search Console Performance report rather than broken out, so a flat analytics line after a fix is not proof the fix failed. That is a statement about Google's reporting, checked on 12 October 2026. Other engines publish no comparable breakdown at all, which means the absence of a line for them is not evidence that nothing happened there.

The arithmetic nobody runs

Here is the part that decides whether your result is real, and it takes two minutes.

Mention rate is a proportion. Any proportion measured from a finite number of runs carries a confidence interval, and at the sample sizes people actually use, that interval is wide enough to swallow the result. The intervals below are Wilson score intervals at 95% confidence, computed in session on 12 October 2026 from the counts shown, so any reader can reproduce them with the same formula and the same two inputs:

4 mentions out of 10 runs: the true rate sits somewhere between 17% and 69%.
6 out of 10: between 31% and 83%.
12 out of 30: between 25% and 58%.
18 out of 30: between 42% and 75%.
120 out of 300: between 35% and 46%.
180 out of 300: between 54% and 65%.

Read those pairs. Going from 40% to 60% on ten runs each is indistinguishable from nothing happening, because the two intervals overlap across most of their length. The same jump on thirty runs still overlaps. At three hundred runs a side this particular jump separates cleanly, and only then have you measured something. A smaller real effect would need more runs again, so three hundred is not a threshold that holds everywhere, it is what this 40 to 60 example happens to need.

That is the whole reason a single dramatic screenshot is worthless and a boring repeated harness is not. If three hundred runs per side is out of reach, and for most teams it is, then either narrow the claim to a handful of prompts you can afford to run properly, or report the interval alongside the number and let the reader see how soft it is.

Pooling helps. Ten prompts run thirty times each gives you three hundred observations for the set as a whole, even though no single prompt is well measured. Say which level you are claiming at.

Freeze the harness, move one thing

Borrowed from ordinary experiment design and worth the discipline here. Write down, before you change anything, the exact prompt strings, the engine list, the number of runs, the time of day, whether retrieval is on or off, and the metric. Those are the harness. The harness does not change during the experiment.

Then change one thing on the site. If you ship a schema fix, a rewrite and a speed improvement in the same week, you have learned that the week worked. You have not learned which of the three did it, and you will be unable to decide what to do next quarter.

Keep a control set. Pick five or ten pages that target comparable prompts and leave them alone. If the untouched pages climbed by a similar amount, the likelier explanation is the engine, not your site. What makes a result credible is not the size of the headline number but the gap between the treated set and the control over the same window.

The confounders that will eat your result

Four of these come up again and again in the audits we run, and three are invisible unless you look for them.

Propagation delay. Access changes do not take effect the moment you deploy them. OpenAI's crawler documentation states that for search results it can take about 24 hours from a site's robots.txt update for their systems to adjust, checked on 12 October 2026. That is OpenAI's published figure for OpenAI, and the other engines do not publish one, so treat a day as the one number you can cite and the rest as unknown. If you measure the morning after a deploy, you may be measuring the old state.

Model updates underneath you. Engines update their models and their retrieval stacks on their own schedule, and we have no visibility into when. A step change in your numbers that lands on the same day across every prompt and every brand you track is more likely theirs than yours. This is what the control set catches.

Retrieval that never reached the page. A fix to page content cannot show up if the page is not being fetched inside the engine's time budget. Check origin response time before you conclude the content change failed, because slow pages get dropped before they get read. The same goes for a rewrite that reads beautifully as a whole page and loses the answer once it is cut up, which is the mechanism in chunking: how your page is cut up before it is read.

Denominator drift. If you add prompts to the set between the before and the after, or if one engine starts refusing some of your prompts, the two sides are no longer comparable. Count the completed runs on each side and make sure they match. A quiet drop from 300 runs to 240 because an engine rate-limited you will move the percentage all by itself.

Share of voice deserves its own warning here. It is a ratio, so it moves when a competitor disappears as readily as when you improve, and the two look identical in a chart. Share of voice in AI answers, and its limits covers where it misleads.

A before and after you can run this month

Ten prompts a buyer would really type. Three engines. Thirty runs per prompt per engine, spread across several days rather than fired in one burst. Record for every run: date, engine, whether retrieval was on, brand named yes or no, claim correct yes or no, and the source URLs listed. Leave five comparable pages untouched as the control.

Run the baseline. Ship one change. Wait a full crawl cycle. Run the identical harness again. Compute the interval on both sides and on the control. If your interval moved and the control's did not, you have something worth acting on. If both moved, you learned about the engine.

The self-check lists ten things you can verify by hand before you spend a month measuring something that was broken at the access layer all along.

What this method cannot tell you

It cannot tell you about revenue. Referrer data from these surfaces is incomplete, and a visit that arrives without a usable referrer is one we cannot see at all, so a measured lift in mention rate connects to pipeline by argument rather than by attribution.

It cannot tell you how long the effect lasts. Nothing in a single before and after speaks to durability, so re-running the same harness on a schedule is the only honest answer we have.

It cannot separate two changes you shipped together. If the schedule forces a bundle, say so in the write-up.

FAQ

How many runs do I need for a credible before and after? More than ten and probably more than thirty per prompt. At thirty runs a side the 95% confidence interval on a 40% to 60% swing still overlaps. Pooling across a prompt set is the practical route to a sample size that separates.

Why does a control group matter if I only changed my own pages? Because the engine changes too. A control set of untouched but comparable pages is the cheapest way we know to tell an engine-side shift from your own work.

How long should I wait between the change and the second measurement? At least one crawl cycle. Access and robots changes can take about a day to register with some engines and longer to show up in answers, so a week is a safer floor than a day.

Does Search Console show whether my AI search fix worked? Not directly. Google's documentation says AI Overviews and AI Mode traffic is included in overall search traffic rather than reported as its own line, so you cannot isolate the effect there. You need your own prompt harness.

Is a published lift figure something I can expect? Treat any single published figure as one result from one site with one prompt set. Ask for the run count and whether a control group moved. Without both, the number is a headline.

Honest limits

This post establishes the arithmetic, not a benchmark. The intervals are Wilson intervals computed from the counts printed beside them, and they describe sampling error only: they say nothing about an engine that is drifting between your two measurement windows, which is a real effect the maths cannot price. The claims about propagation and reporting come from two engines that publish documentation, Google and OpenAI, and we do not know the equivalent figures for engines that publish none, so the advice to wait a crawl cycle is an extrapolation rather than a measured interval. The confounders listed here are the ones we keep meeting in our own audit work, not the output of a survey of the field, and the list is unlikely to be complete.

Sources

Google Search Central, AI features and your website (traffic from AI features is included in overall search traffic in Search Console): developers.google.com/search/docs/appearance/ai-features
OpenAI, Overview of OpenAI crawlers (for search results it can take about 24 hours from a site's robots.txt update for their systems to adjust): developers.openai.com/api/docs/bots
E. B. Wilson, "Probable Inference, the Law of Succession, and Statistical Inference", Journal of the American Statistical Association, volume 22, 1927, pages 209 to 212. The intervals in this post were computed in session on 12 October 2026 from the counts shown, at 95% confidence, using that method.

Both documentation pages above were opened and re-read on 12 October 2026 and still carry the sentences cited here.