Testing with retrieval on and off
The short answer: Run the same prompt twice, once with the engine allowed to search the live web and once with search withheld, then compare the two answers. The retrieval-off run shows what the model already believes about you from training. The retrieval-on run shows what today's web and today's index can add. Two different problems live behind those two answers, and a single blended run hides which one you have.
Reporting that blends the two states hides which one moved. A score goes up, and nobody can say whether a page fix did it or whether the model simply happened to search that time.
The two runs isolate two different failures
Switch retrieval off and you are interrogating the weights. Whatever the model says about your category, your pricing, your positioning, comes from text it absorbed during training and from whatever it has generalised since. You cannot patch that this quarter. Publishing changes it slowly, through the next training cut, and only if the new text is widespread enough to matter.
Switch retrieval on and you are interrogating a pipeline: a query rewrite, an index lookup, a fetch, a passage cut, a reranking pass, then the writing. Every one of those steps is something you can affect in days or weeks. A blocked user agent, a client-rendered page, a missing comparison table, a competitor's listicle that answers more cleanly than your own page does.
Both runs matter, and they fail in opposite directions. A brand that wins on retrieval off and loses on retrieval on points you at access and passage first: the model knew the name, and something in the live pipeline dropped it. A brand that wins only with retrieval on has no memory footprint yet, which means its visibility is rented and is exposed the moment the engine decides the question does not need a search. That split is the whole subject of Retrieval vs training data: which one is answering about your brand.
Where the switch actually lives
Consumer apps are the wrong place to run this. The toggles change between releases, we have not found them documented as stable behaviour, and you cannot prove from a transcript which state you were in. The developer APIs are explicit, and the parameter names are published.
On OpenAI's Responses API, web search is a tool you add to the tools array. The documentation is blunt about who decides: "Like any other tool, the model can choose to search the web or not based on the content of the input prompt." To make the state deterministic, the same page tells you that tool_choice: 'auto' leaves search optional while tool_choice: 'required' makes it run. Retrieval off is simply the request with no search tool attached. One more detail is worth knowing before you compare long answers: the same page states that for Responses API web search the search context is limited to 128k tokens even when the model's own context window is larger. Figures here were read from the four pages listed under Sources on 12 October 2026, and vendor documentation changes.
Anthropic's web search tool behaves the same way. Omit the tool and the model answers from training. Include it and the model decides, with max_uses available as a hard cap on the number of searches in a request. Each result carries a page_age field, which the documentation describes as when the site was last updated, a quiet gift for anyone testing freshness. That page also says citations are always enabled for web search, with cited_text of up to 150 characters per citation. It lists web search on the Claude API at $10 per 1,000 searches, plus standard token costs, so a few hundred paired runs is a real but small number rather than a budget conversation.
Google's grounding tool is also an opt-in. The Gemini API documentation describes the model analysing the prompt and deciding whether a Google Search would improve the answer, then generating one or more queries itself. Exclude the tool, and you have the retrieval-off side of the pair.
Across those three APIs the grammar is the same. Retrieval off means the tool is absent. Retrieval on means the tool is present and, where the API allows it, forced, so that run-to-run variance in the model's own judgement does not contaminate the comparison. We have read the switch in those three sets of documentation and nowhere else, so treat it as three worked examples rather than a rule covering the rest of the field.
Reading the four outcomes
Pair every prompt and sort the result into one of four boxes. The box, not the score, is the finding.
Named in both. The strongest position. The model knows you and the live web confirms you. Check only that the claim attached to your name is accurate in both, because a confidently wrong memory is worse than absence.
Named only with retrieval on. Your pages and your third-party mentions are doing the work. There is no residue in the weights. This is what you would expect of a company younger than the training cut, and it is not a crisis, but it does mean your presence depends on the engine choosing to search at all.
Named only with retrieval off. The interesting one, and the one most likely to be fixable inside a quarter. The model remembers you, then searches, then writes you out. Something in the retrieved set outranked your page, or your page never entered the candidate set. Start with access, then the passage. Paywalls, logins and what retrieval can reach covers the first half of that.
Named in neither. No memory, no retrievable evidence. The honest answer here is a publishing and corroboration programme, not a tweak.
Record the four counts per engine rather than a single blended percentage. A percentage that mixes states is the fastest way to fool yourself, which is the argument in How to measure AI visibility without fooling yourself.
A check you can run this week
Take ten prompts a buyer would really type. Run each one five times with the search tool withheld and five times with it forced, on at least two engines. Ten runs per prompt per engine sounds heavy until you realise it is a single afternoon of scripting.
For each run, record four fields: brand named yes or no, the claim made about you, the cited URLs, and the retrieval state. Nothing else. Resist the dashboard urge.
Then add the step that turns the runs into evidence. Open your server logs for the same window and look for the fetch. OpenAI documents OAI-SearchBot as the agent that surfaces sites in ChatGPT's search features, and ChatGPT-User as the agent that visits a page when a user's question sends it there. Because a person starts that second visit, the same page cautions that the rules in your robots file may not govern it. Those two agent names are OpenAI's; other engines publish their own, and each one is worth looking up rather than assuming this list covers it.
Read the logs carefully, because absence in a log is weaker evidence than presence. If your retrieval-on runs cite you and no matching fetch appears, the likeliest reading is that you were served from an index or a cache rather than a live fetch, which tells you how stale the engine's copy of your page could be. It is only the likeliest reading: a fetch can arrive from an agent string you do not recognise, from an IP range you have not attributed, through a CDN that never passes the request to your origin, or outside the window you searched. If no agent you can identify appears and your pages go uncited across the set, access is the first thing to rule out, before rewriting a word of the page.
Counting citation shares on top of this is fine, as long as you keep the two states in separate columns. The caveats are in Share of voice in AI answers, and its limits. Our sample audit report shows what a paired run looks like once the evidence is attached to each finding.
Honest limits
Forcing a search is not the same as the engine's natural behaviour. Consumer products apply their own policy about when to search, and a forced run tells you what retrieval could do for you, not how often it will be invoked. Keep an unforced arm if you want that second number, and expect it to be noisy.
The retrieval-off run is also not a clean read of the weights. Safety layers, system prompts, memory features and personalisation all sit between you and the raw model, and none of them are visible to you. Treat the result as a directional signal rather than a measurement.
Five runs will not settle a close call either. We use larger counts when a decision depends on the gap, and we say so when a difference is inside the noise. An audit here runs about 45 prompts across five engines, paired on and off, with the evidence kept so a finding can be argued with rather than believed. That is a sample of prompts, not a census of what buyers ask.
Two further things this post does not establish. The on and off switch is documented for the three developer APIs named above and was read there on 12 October 2026; we have not verified it for other engines, and consumer products sit outside the method entirely. And a log with no fetch in it does not prove you were not fetched, only that nothing you could identify reached the origin you were watching, which is why the log step narrows a hypothesis rather than closing a question.
FAQ
Can I do this without writing code? Partly. You can approximate retrieval-off by asking an engine to answer without searching, but the model may search anyway or claim it did not. For evidence you can defend, use the APIs, where the tool is either in the request or it is not.
How many runs per prompt are enough? Five per state is the floor for spotting an obvious split. Ten is better if you are going to report the difference to anyone who might act on it. Single runs are anecdotes.
Does a retrieval-off win mean I can stop publishing? No. Training data ages, and the next model version is trained on a newer slice of the web. A memory footprint that is not being refreshed by live mentions decays quietly.
Which engines support the comparison properly? We have verified the switch in three places: the OpenAI Responses API, the Anthropic web search tool and the Gemini API grounding tool, each of which documents web search as a tool you include or omit. Others may offer the same control; we have not checked them. Products with no such switch can still be tested on the retrieval-on side only, and you should label those results as such.
Sources
- Anthropic, web search tool documentation: platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
- OpenAI, web search tool guide: developers.openai.com/api/docs/guides/tools-web-search
- OpenAI, bots and crawler user agents: developers.openai.com/api/docs/bots
- Google, Grounding with Google Search, Gemini API documentation: ai.google.dev/gemini-api/docs/google-search
All four pages were opened and the quoted details confirmed on 12 October 2026.