Blog · Running the work

QA for changes meant to move AI visibility

Published 17 August 2026 · Updated 12 October 2026 · 9 min read

QA for changes meant to move AI visibility
Photograph: Hans Westbeek / unsplash

The short answer: QA for an AI visibility change has four gates, and most teams only run the first. Gate one asks whether the change is in the HTML a machine receives, not the page a browser paints. Gate two asks whether robots.txt has caught up with the deploy. Gate three asks whether a crawler actually fetched that HTML, proved by a verified log line. Gate four asks whether the answers moved by more than the noise, which needs more runs than a ten run before and after can supply.

Skip any one of those and you get the same outcome: a ticket marked done, a report that says visibility is flat, and nobody able to say which part failed.

Gate one: did the change reach a machine

The browser is the wrong witness. It runs your JavaScript, it carries your cookies, it negotiates a modern image format, and it paints a page that the bytes on the wire do not contain. Fetch the raw bytes instead and read what a text-only client receives.

`` curl -sS -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \ -D- example.com/pricing -o /tmp/page.html ``

Then grep /tmp/page.html for the literal sentence you shipped. If the sentence is not in there, the change exists only after hydration and the work is not done. The same check catches a second problem that is easy to miss: structured data injected client side. Google's JavaScript SEO guidance, updated 4 March 2026, says you may generate the JSON-LD with JavaScript and inject it into the page. Google's general structured data guidelines, updated 10 July 2026, then say not to mark up content that is not visible to readers of the page, and give the example of JSON-LD describing a performer the HTML body never mentions. A schema block describing a product whose name never appears in the rendered body is therefore a defect by Google's rules even when the validator is green. We have found no equivalent published policy from the other engines, so treat this as Google's rule, not a general one. Run the Rich Results Test and the URL Inspection live test on the deployed URL, not on a staging copy, because staging usually carries a different robots policy.

Keep the raw response headers from that curl. A Vary: User-Agent you did not intend, a Cache-Control: no-store on a page you want cached, a 200 that is actually a soft 404 with your error template inside it. All three show up in the headers and the first two hundred lines of body, and none of them show up in a browser tab.

Gate two: the robots.txt deploy window

Here is the mechanism almost nobody QAs, and it is written into the standard. RFC 9309, the robots exclusion protocol, says in section 2.3.1.4 that when robots.txt is unreachable, which in HTTP terms it identifies as a status in the 500 to 599 range, a crawler must assume complete disallow. Section 2.4 says a crawler should not reuse a cached robots.txt for more than 24 hours unless the file is unreachable.

That is what the standard asks for. What a given crawler does is a separate question, and only some of them publish it. Google does. Its robots.txt documentation says that if it has found a robots.txt file and then cannot fetch it, it stops crawling the site for the first 12 hours, then falls back to the last good copy for up to 30 days while it keeps retrying, and assumes there are no crawl restrictions if it has no cached copy at all. We have not found published behaviour of that specificity for the other engines we track, so for those we can quote the standard and not the implementation.

The failure mode falls out of the Google case on its own. Your deploy takes the origin down for ninety seconds. During that window a crawler requests robots.txt and gets a 503. Your site is back in two minutes, and on Google's documented behaviour the crawl pause is twelve hours, not ninety seconds. For an engine that documents nothing, you cannot see from outside what it decided, which is the argument for never letting that one file fail.

The QA step is cheap. Serve robots.txt from somewhere that does not share a fate with the application, usually the CDN edge as a static object, and add a synthetic monitor that alerts on any non-200 for that one path. Then check it immediately after each release. A single curl -I example.com/robots.txt in the post-deploy job costs nothing and closes a hole that no content work can compensate for. This belongs in the first week of The first 90 days of AEO work, before anybody writes a new page.

Gate three: prove the fetch happened

A change that shipped and was permitted still has to be read. The only evidence you can check yourself is a log line, and the user agent string in that line is a claim, not a fact. Anyone can send it.

OpenAI publishes separate IP range files for each of its bots, at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json. Google publishes its crawler ranges the same way, and its verification documentation, updated 20 March 2026, describes both the IP match and a reverse then forward DNS lookup that has to resolve back to the same address. Checking the source IP against the published list turns a user agent claim into a verified fetch. Do that before you count anything, because unverified agent strings inflate your numbers in exactly the direction that flatters the work.

Two details from the same documentation change how you read the logs. OAI-SearchBot is the agent that surfaces pages inside ChatGPT's search features, and it is a different bot from GPTBot, which collects training data. Blocking one does not block the other, and teams routinely block the wrong one. Separately, ChatGPT-User fires when a person asks the assistant to visit a page. OpenAI's documentation warns that the exclusion rules in your robots file might not govern that agent, since nothing automatic is crawling. A spike of ChatGPT-User hits is user demand and tells you nothing about whether the retrieval index has your new copy. Counting them together produces a chart that goes up while nothing improved. The same confusion sits behind Google AI Mode: what changes for a publisher, where the fetch that assembles an answer and the fetch that follows a user click are different events with different meanings.

A useful discipline: timestamp the deploy, then watch for the first verified fetch of the changed URL by each engine you care about. The gap between those two timestamps is your real feedback latency, and it differs a great deal between a page crawled often and one buried deep in the site. We have watched that gap on client sites but never to a stated method, sample size and date, so we are not publishing a range for it here. Measure it on your own verified log lines before you schedule the re-run.

Gate four: measuring the outcome without fooling yourself

Now the uncomfortable arithmetic. Engine answers are sampled, so your before and after is a proportion, and proportions from small samples are almost useless. Run a prompt ten times before the change and get five mentions. Run it ten times after and get seven. That looks like a lift.

Compute the interval and it is not. A Wilson 95% interval on five of ten runs, which we calculated in session for this post, spans roughly 24% to 76%. At forty runs, twenty hits still spans about 35% to 65%. Your two-mention improvement sits inside the noise of both. That is the statistical form of the point made in Why an AI answer changes between two identical runs, and it is why a single screenshot is never evidence of anything.

Three practices make the comparison survivable. Hold the prompt text fixed and version it, because a reworded prompt is a different experiment. Run the same prompt set on an unchanged control page or a competitor term, so engine-wide drift shows up as movement in both arms rather than as your win. And change one thing per release window where the calendar allows it, since a content rewrite and a schema change shipped together cannot be told apart afterwards.

Do not expect Search Console to referee this. Google's own documentation on AI features, updated 10 December 2025, puts appearances in AI Overviews and AI Mode inside the site's overall search totals, reported under the Web search type, with nothing broken out. There is no AI segment to filter on, so the dashboard cannot confirm or deny a change you made specifically for those surfaces. The self-check walks through what you can observe without it.

Honest limits

It cannot attribute. A verified fetch and a measured lift in mention rate still leave you guessing about revenue, because these surfaces mostly send no referrer worth the name.

It cannot isolate your change from the engine's. Models are updated on schedules you do not see, and an update can move results across a whole category within days. The control arm detects that drift. It does not let you subtract it cleanly.

It also cannot be run well by somebody with no access to deploys and logs, which is the practical argument running underneath In-house or agency for AEO. Whoever holds the QA gates needs the log pipeline and the release calendar, or the gates are theatre.

Three more things this post does not establish. The robots.txt behaviour above is the standard's requirement plus Google's published implementation of it, and we cannot tell you what an engine that publishes no such page does with a 503, so the twelve hour figure is Google's and nobody else's. The Wilson intervals quoted here assume runs are independent draws from a stable process, which engine sampling only approximates, so read them as a floor on the uncertainty rather than the whole of it. And the worked examples lean on OpenAI and Google because those two document their crawlers in public, which is a fact about the documentation and not evidence that the rest behave the same way.

FAQ

How many runs do I need before a before and after means anything? More than feels reasonable. At forty runs per prompt a half-and-half result still carries roughly a thirty point spread at 95% confidence, so treat small movements as unresolved and pool across a prompt set rather than chasing one prompt.

Can I QA this on staging? Partly. Staging is fine for checking that the sentence and the schema are in the served HTML. It cannot test robots.txt behaviour, CDN caching or crawler access, because those differ by environment, so the later gates have to run against production.

What is the single highest value check to add first? A post-deploy assertion that robots.txt returns 200. It takes minutes to add. The standard tells a crawler to assume complete disallow while that file is unreachable, and Google documents a twelve hour crawl pause after a failed fetch of it, so a brief outage on a single file can cost more than a quarter of content work earns.

Why did my mention rate drop after a change I was sure was an improvement? Check the order of the gates before rewriting anything. A sentence that moved below a template fold, a page that now renders its answer client side, a robots.txt blip during the release, or an engine update that week. Each produces the same symptom and has a different fix.

Sources

Related reading

The first 90 days of AEO workIn-house or agency for AEOGoogle AI Mode: what changes for a publisherWhy an AI answer changes between two identical runsTool sprawl in AI search, and the cost of itWhat an AEO audit should contain