An audit runs a prompt set of about 45 prompts across five engines, each prompt put twice, once with live retrieval allowed and once with it withheld. Every outcome is sorted into one of four count tiers, every rate is printed with the denominator it was measured against, and engines are never averaged together. This page is the counting rule itself, written out so you can run a prompt yourself, apply our rules, and get the answer we would have got.
Three things define a run. Change any one of them and the number changes, which is why all three are fixed in writing before a single prompt is sent.
Questions a buyer in your category would really type, researched for your category and your buyers' phrasing before the run starts. The list is fixed for the engagement, because adding prompts mid-run moves the score by itself. It is a sample of what buyers ask, not a census of it. Branded and unbranded prompts are kept in separate buckets, since branded prompts sit near a ceiling and flatten the average.
ChatGPT, Claude, Perplexity, Gemini, and Google's AI surfaces, AI Overviews and AI Mode. Each engine is reported on its own denominator. We never pool five engines into one headline percentage, because a brand can sit at a third on one surface and at nothing on another, and the mean hides both facts.
The engine is allowed to search the live web. This interrogates a pipeline: query rewrite, index lookup, fetch, passage selection, reranking, then the writing. Every step there is something a site can affect in weeks. In the developer APIs this is the search tool present in the request and, where the API allows it, forced, so the model's own decision about whether to search does not contaminate the comparison.
The same prompt with the search tool absent. This interrogates what the model already holds from training. It moves slowly, through the next training cut, and no fix ships it this quarter. The split between the two states is the owned and rented reading: whatever disappears when retrieval is withheld is rented.
The on and off switch is documented as a tool you include or omit in three places we have read directly: the OpenAI Responses API web search tool, the Anthropic web search tool and the Gemini API grounding tool, all checked on 12 October 2026. Where a surface offers no such control, it is run on the retrieval-on side only and labelled that way in the report. Consumer app toggles are not used as evidence, because a transcript cannot prove which state produced it.
Our published position is a range, not a single number, and the reason is arithmetic rather than budget.
At a fixed number of engine calls, more prompts run once each beats fewer prompts run many times. In a simulation we ran in October 2026, with per-prompt mention rates drawn from a Beta(2,3) distribution averaging 40 percent and 4,000 repetitions per design, a budget of 45 calls spent as 45 prompts run once gave a standard deviation of 7.2 points, 15 prompts run three times gave 8.6, and 9 prompts run five times gave 9.6. Same cost, worse answer. Those figures describe the model, not a measurement taken on live engines.
Five runs tell you whether a mention is possible at all, and five per retrieval state is the floor for spotting an obvious on-versus-off split. Ten per state is better if the difference is going to be reported to someone who may act on it. Below five, you have an anecdote.
Thirty runs pin a rate to roughly plus or minus 17 points, which is enough to tell a quiet prompt from a loud one and nothing finer. Single-digit precision on one prompt needs several hundred runs, per engine, per day, which is why no report of ours prints a per-prompt percentage to one decimal place. The one comparison query a sales team keeps losing is worth that thirty. A daily single sample of it is not.
Zero is cheaper to establish than a rate. Twenty runs with no mention puts the upper bound near 16 percent, which is enough to write the prompt off and move on. That is often the only decision the question needed.
Two control prompts, whose answer should not move, sit in every prompt set. They are what separates an engine shifting underneath us from our own sampling noise. We do not yet have enough months of control data to publish a figure for how often the control drifts, and we will say so until we do.
An engine answering the same question differently twice is not misbehaving. The clearest published account is Thinking Machines Lab's September 2025 write-up on nondeterminism in LLM inference, which traces it to batch invariance: floating point addition is not associative, and the batch a request lands in depends on who else is using the service that second. In the one careful measurement there, a single prompt sent a thousand times at temperature zero to one open weights model produced 80 distinct completions. That is a demonstrated mechanism on one model on one lab's hardware, not a measured rate for the engines you care about. We cannot see inside a hosted engine, and the engines we run expose no seed control to begin with.
So we treat each run as a draw and do four things with it.
The denominator goes beside every rate. A naming rate of 12 of 45 on one engine is 26.7 percent with a 95 percent interval of roughly 16 to 41 percent. A re-run landing on 31 percent is not evidence of improvement. The rate without its denominator is not a number a reader can check.
Compute the bar across prompts, not across answers. Collapse each prompt's runs to one score between 0 and 1 first, then take the standard error across those prompt-level scores. Pooling every answer into one binomial interval treats correlated answers from the same prompt as independent draws, and the bar comes out too narrow. In a simulation we ran on 12 October 2026, 45 prompts run ten times each, the naive bar was about 1.6 times too small: plus or minus 4.5 points on paper against plus or minus 7.2 points in fact. Those are simulated figures. The same effect has been measured on public language model benchmarks, where clustered standard errors came out between 1.10 and 3.05 times the naive ones depending on the dataset, which tells you clustering matters and not how much it matters for your prompt set.
Do not report every mover. Test 45 prompts week over week at a 5 percent threshold and, with nothing at all changing, the chance that at least one crosses it by luck alone is 90.1 percent, with 2.25 false movers expected a week. That is one line of arithmetic, 1 minus 0.95 to the power 45, assuming independent tests. Prompts are not perfectly independent, so read it as an order of magnitude. It is still why our reports mark prompts as won, lost or contested rather than ranking them by weekly delta.
Cap the confidence to what the run supports. Every number in a report carries one of four labels. Verified: you can re-run it today and get the same answer. Measured: we ran it, and sampling error applies. Indicative: directional, with engine nondeterminism dominating. Judgement: our recommendation, with no confidence claimed for it. A single-day run is capped at indicative however clean it looks, and a composite score inherits the lowest confidence of any input it is built from.
How to read a rate rather than a rank. Read the interval, not the point. Read the denominator beside it. Compare a prompt against its own earlier score rather than against another brand's, because pairing on a fixed list cancels the prompt-sampling variance that otherwise dominates. If two intervals overlap, you do not have a result. A rank reads as though the order is stable and the engine were a league table; a naming rate with its interval reads as what it is, a rate observed on a sample, on a surface that will re-retrieve the next time it is asked.
A brand named in prose, a product named without the brand, a domain used as a source, and a feed carousel placement are four different outcomes with four different causes. Merging them inflates the headline and hides the fix.
An answer that says "single-origin subscriptions like Northbay" counts as Tier A for Northbay, the invented brand our sample report uses. The unit of counting is the run, not the sentence: a brand named four times in one answer counts once. The match list is written down before the run and published in the report's method appendix, covering the brand name and its spaced, hyphenated and possessive variants, plus any product line names deliberately included. Near-collisions, similarly spelled unrelated brands, are excluded by name with the reason written beside each one. A mention still counts when the answer hedges, and it still counts when the brand is named unfavourably, because the headline number measures whether an engine can reach you, not whether it likes you. Accuracy of the claim attached to your name is recorded separately, since a confidently wrong memory is worse than absence.
The answer names the thing you sell and not you. It is recorded, and it is never added to the Tier A rate. The cause is usually an entity problem rather than a content one, and the fix is different.
A citation is a URL on a domain declared in the match list as yours, attached to the answer by the engine itself: an inline footnote, a source list entry, or a link in the answer. It is counted once per run and deduplicated by domain, so a page cited three times in one answer is one citation. A link in a shopping or feed placement is not a citation, it is Tier D. A third-party page about you is not your citation; it is logged as a corroborating source, which is a different finding. Cited-but-not-named is the costliest pattern we see: the page was good enough to use and the brand was not good enough to name.
A carousel showing your product under a generic query is Tier D. It is reported on its own line and never added to the headline naming rate, and it is never a recommendation. It is a feed placement, with a feed fix behind it.
Being named last in a list of eight is not the same result as being named first, and a mean position hides that, so position is recorded as a distribution rather than averaged. Within Tier A, a run is scored as a recommendation when the answer puts the brand forward as something to act on: a ranked or shortlisted item, or a direct endorsement. The edge cases are these.
Conditional recommendations count, with the condition recorded. "If you need same-day delivery, Northbay" is a recommendation on a narrower question than the one asked, and a report that drops the condition is overstating the result.
Context, baselines and warnings do not count. A brand named to define the category, used as a price comparison point, or named as a thing to avoid, is a Tier A mention and not a recommendation.
A source attribution does not count. If the brand appears only as the attribution on a claim, that is Tier C with a name attached, logged as named-and-cited, not as a recommendation.
Position is kept with the count. Named at position 7 of 8 is recorded as such. A brand can be reachable in most answers and never the recommendation in any of them, and that is a specific finding with a specific fix.
Take one prompt. Run it on one engine with search forced on, then again with the search tool absent. For each run record five fields and nothing else: the retrieval state, whether the brand was named, which tier the outcome falls in, the position if it was named in a list, and the cited URLs. Repeat on the engines you care about. Score each run against the four tiers above, never merging D into A and never averaging across engines, and print the count with its denominator. That is our rule, applied the way we apply it. If your count differs from ours on the same answer, one of us has a match-list problem, and the match list is published in the report so the argument can be settled.
Said plainly, because a method is only as honest as the list of things it cannot do.
A naming rate is not a demand figure. No finding here traces a mention in an answer to a sale, a signup or a pipeline number, and we do not model one. Generative referral traffic that already exists is read from your own Search Console, as first-party evidence that the surface sends people at all. That is a separate fact from the naming rate and is kept separate from it.
Nothing in the method makes an engine cite you. The audit finds what is in your control and suppressing you, what is not in your control, and which of the two is doing the damage. Fixing the first makes you eligible. Eligibility is not a placement, and no position is guaranteed by anyone honest.
We cannot see inside a hosted engine's batching, index, ranking or safety layers, and none of them are ours to change. Retrieval-led surfaces tend to move in four to eight weeks once the readable layer is fixed. Anything that has to reach a training corpus is a six-to-eighteen-month question, and the report says which of your prompts sit in which bucket.
About 45 prompts is a sample of the thousands of questions a buyer might type, and a different list would have produced a different score. Forcing a search also tells you what retrieval could do for you, not how often an engine will choose to search. A retrieval-off run is not a clean read of the weights either: system prompts, memory and personalisation sit in between, so read it as direction rather than measurement.
Where server logs are available, we look for the matching fetch, because presence is useful evidence. Absence is weak: a fetch can arrive from an agent string nobody has attributed, from an unrecognised IP range, through a CDN that never reaches your origin, or outside the window searched. A log step narrows a hypothesis. It does not close a question.
We have clients, D2C brands in hair care and in skin care, and this method has been applied with them. We publish none of their results, and there will be no case study. No client has agreed to be named, and a number we cannot hand you the evidence for is not evidence, whichever way it points. So nothing on this site carries a client figure, a client name or a before-and-after. What we publish instead is this method, the sample audit, whose invented sections are labelled as invented and whose worked sections carry only the confidence they earned, and a study of our own on a category we do not sell into, with its prompt set, protocol and scoring rules fixed and published before the run. That study has no results yet, and its protocol is published in full in advance of collection. Where a figure comes from a simulation rather than a live engine, the report and this page both say so.
One thing: the audit report. It arrives as a link and a PDF to the email on the order, within 7 working days of cleared payment, with a 30-minute walkthrough and questions answered by email for two weeks after it lands. Every finding carries how it was measured and the confidence tier it earned, so it can be argued with rather than believed.
What does not arrive is a data product. We do not send the prompt set as a file, raw run exports, ledgers or CSVs, and no part of this page should be read as promising them. The evidence lives inside the report, attached to the finding it supports. Depth varies with the category, because every audit is scoped to the client, so treat a section list as the shape of the work rather than a fixed contract. We do not implement the fixes inside the audit either: that is the build, and the report tags every item with who owns it and how long it takes so you can price it first.
Want to run some of this yourself first? The self-check is ten checks that need no tooling and no email.
About 45 prompts, five engines, retrieval on and off, seven working days. $500.
Read the sample audit, section by section →
Or ask a question through the contact form