The rules, written down before the first answer is collected.

How stable is the brand list an answer engine gives a buyer?

For one public consumer buying category, across three answer engines, over one collection window, the study asks four things. The second is the primary one. The other three are reported because the data is being collected anyway, and publishing only the cut that suits us would be the exact failure we write about.

We publish method posts saying that a single engine answer is one draw from a distribution rather than a reading. We have never published the raw evidence behind that claim, because our audit work belongs to the clients who paid for it. This study is meant to be that evidence, run where we have nothing to gain from the answer.

Robot vacuum cleaners, UK consumer buyer, retail purchase.

We have no commercial interest in this category. We do not sell to it, we have never audited a brand in it, and we have no relationship with any manufacturer or retailer that may be named in the results. Whatever the data says about any brand, we gain nothing either way. A study run on our own category would be marketing wearing a lab coat.

On brand names: our house rule is that we never name a vendor in our own category, the AEO and AI-visibility category, and that rule is untouched here. Robot vacuum manufacturers are the subject of the measurement, not competitors of ours, and a specific-product prompt cannot be written without naming a product.

45 study prompts and 2 controls, frozen on 24 October 2026.

47 prompts in total, version 1.0.0. The set is frozen: prompts are never added, removed, reworded or reordered once collection starts, and a change of any kind ends this version and starts a new study. All 47 are listed in full below, with their ids, so you can read exactly what will be pasted.

They are written the way buyers type, in lower case, with the hesitations and the imprecise model names intact. Where a product name is slightly wrong or out of date, that is deliberate and is left as written, because buyers are imprecise and we are measuring what engines do with real questions. The runner is instructed to paste each one character for character and to fix nothing.

Three engines, two arms, 783 answers.

All three engines are driven through a desktop browser. We run no engine APIs, because the answers a buyer sees come from the consumer products and not from an API. Model versions are not under our control and are not pinned: whatever the surface states about the model is recorded on every row.

The two arms, and why those run counts.

Breadth arm. One run per prompt per engine, on each of three separate days. 47 prompts, 3 engines, 3 days, which is 423 answers. It supports a category-level naming rate with a prompt-level error bar computed across the 45 study prompts.

Depth arm. Twenty runs per prompt per engine, inside a single sitting on day 1, on six prompts declared in advance: rv-002, rv-012, rv-017, rv-026, rv-027 and rv-038. That is 360 answers, and it supports a per-prompt rate and an inter-run agreement figure. Those six were chosen before collection to cover two discovery prompts, two comparison prompts, one specific-product prompt and one troubleshooting prompt, and not on the basis of any observed result, because none exists.

The two arms answer different questions and are never pooled.

The run counts come from our own published position, which we are obliged to follow here even though a bigger headline would be available if we ignored it. At a fixed call budget, more prompts run once each beats fewer prompts run many times for any portfolio-level figure, because extra runs shrink only run noise while extra prompts shrink run noise and prompt-sampling noise together. So the budget buys width, not repetition. Twenty runs is the point we can actually execute by hand across three engines in one sitting: it yields a Wilson interval roughly 40 points wide at a rate near 40 percent, and an upper bound near 16 percent when a brand is named zero times out of twenty. We report the interval, never the midpoint alone, and we do not claim single-digit precision on any prompt. The breadth arm's error bar is computed across the 45 prompt-level scores, not across the 423 answers, because the honest sample size for a category-level figure is the number of prompts, and the script reports both so a reader can see the size of the difference.

Nothing in this design supports a claim that a brand's true naming rate is, say, 42 percent. It supports a claim about an interval, and about how much two runs of the same prompt agree.

Session conditions, on every single run.

A run that breaks any of these is discarded and re-run, and the discard is recorded.

Order and spacing.

The breadth arm runs on day 1, day 4 and day 8 of the collection window, which gives one within-week gap and one across-week gap. Each day has three collection blocks, one per engine, and the block order rotates across days so that the first engine of the day is not always the one run at the freshest hour. Inside a block the 47 prompts run in the fixed shuffled order, the same on all three days and all three engines. Minimum 45 seconds between the end of one run and the start of the next. All three blocks of a day start inside the same three hour window of the local clock, and that window is the same on all three days.

The depth arm runs on day 1 only, after that day's breadth blocks are complete. Within an engine, runs are interleaved by prompt: one run of each of the six prompts in the fixed order, then the next run of each, twenty times round, never twenty consecutive runs of the same prompt, because that would confound the repeat count with the time of day. Minimum 60 seconds between runs. One engine is completed before the next is started. If the depth arm cannot be completed on day 1 it is abandoned and rerun in full on a later single day, because a depth arm split across two days measures something else and is not reported.

One reduction is permitted, and it is declared now rather than later. If the depth arm cannot be executed on three engines, it may be reduced to ChatGPT only, at the full 20 runs on all six prompts. That reduction must be decided and written into the methods section before collection begins, never after seeing a result. No other reduction is permitted.

Every row records the moment the prompt was submitted as an ISO 8601 string with an explicit offset, and the IANA timezone name of the collection machine, because one of those survives a spreadsheet and the other survives a daylight-saving boundary. The planned day, 1, 4 or 8, is recorded separately from the calendar date, so a slipped day is visible as a mismatch rather than hidden. The machine clock is synced to network time before each day, and the runner records that it did.

One of the three engines cannot be run signed out.

Gemini cannot be used signed out, so it runs in a dedicated account created for this study, with personalisation, activity history and any memory feature switched off in settings before collection, and with no other use of that account at any time.

That is not the same condition as the other two. ChatGPT and Perplexity are collected from a signed-out visitor's surface, and Gemini is collected from a clean signed-in account, which is the closest equivalent the surface permits and still not an equivalent. We never compare Gemini's number to the other two without repeating this sentence, and it will be repeated beside every figure in the results when results exist.

A related limit applies to all three engines. The protocol forces logged-out or clean sessions. Real buyers often have history and memory switched on, and personalised answers may differ from these in ways we cannot measure.

An engine that refuses is data. An engine that fails is not.

The two are recorded in separate columns and handled differently, and the difference is decided now rather than in the moment.

A refusal is kept. The engine answers, but declines to give what the prompt asked for: it says it cannot recommend a product, it answers a different question, it asks a clarifying question instead of answering, or it returns advice with no brand at all where the prompt asked for a recommendation. The row is kept with the verbatim text and one of four codes, for a policy refusal, a question asked back, a brand-free generic answer, or an answer to a different question. It is never re-run. A refusal is a result, and re-running until the engine complies would be selecting the answers we like. A refusal counts in the denominator for naming rate, because a buyer who gets no brand has received an answer with no brand in it, and the analysis also prints the rate with refusals excluded so a reader can take the other view.

An error is not an answer. A network failure, a rate limit, a capacity message, a truncated answer, a page that never finished loading, or a sign-in wall appearing mid-block. The row records the error code and whatever text exists, the scoring columns stay empty, and after five minutes the prompt is re-run once as a new row pointing back at the failed one. If the re-run also errors, re-running stops for that prompt in that block and both errored rows stand in the file. Errored rows are excluded from every denominator, and the script prints how many were excluded. If more than 10 percent of a block's runs error, the block is abandoned, the reason recorded, and the whole block rerun on the next day at the same hour.

A silent model downgrade is a variant change, not an error. If a surface switches to a lesser model after a quota, the new variant is recorded from that row onward with the row at which it changed. A block whose variant changed part way through is reported separately and never pooled with blocks on the other variant.

Capture, for every run. The full verbatim answer as text rather than a screenshot, with its formatting preserved and nothing edited, trimmed or tidied. Every URL the answer exposes as a source, from inline citations and from the sources panel expanded fully, recorded as the real URL rather than the display label. And a full-page screenshot, retained with the dataset as evidence and not published if it contains any personal detail.

A mention, a recommendation and a citation are three different events.

They are never blended into one headline figure. A brand can be mentioned without being recommended, recommended without being cited, and a domain can be cited in an answer that names no brand at all. All three are recorded per run. Scoring happens after collection, never during it, from the recorded text rather than from memory of the session, and every scored row carries the ids of the rules that produced its decision, so a reader can see not just what we decided but which sentence decided it.

Where judgement cannot be removed, the rule says which way to resolve it, and the resolution is always the conservative one: the reading that credits a brand with less.

Mentions

What counts. The appearance of a brand's name, in any case, anywhere in the answer text, including in a table, a bullet list, a heading, a caption or a closing caveat. A brand named six times in one answer is one mention for that run: the unit is the run, not the sentence. Surface forms are mapped to a canonical name through a published alias map, and a model name counts as a mention of its manufacturer only when the manufacturer is unambiguous from the model name alone. If the model name alone does not identify a manufacturer and the answer does not name one, no brand is coded. The raw string is recorded either way, so a reader can check the alias map.

Negative, hypothetical and historical mentions count. "Avoid the cheaper ones, the brushes tangle" is a mention, recorded with negative sentiment. "Used to make a good one" and "is not suitable here" are mentions. The buyer has still read the name. Naming rate is a visibility measure, not an approval measure, and we never report it as though it were endorsement.

A brand the buyer named first does not count as a mention. If a brand appears in the prompt text, its appearance in the answer is recorded separately as prompted, and that brand is excluded from that prompt's mention denominator entirely. An engine repeating a word back to the buyer is not evidence that the engine would have reached for it.

Where it appears matters. A brand name inside the title of a cited page, reproduced in the answer, counts, because it is visible to the buyer. A brand name that appears only inside a URL in the sources panel, and never in the answer prose, does not: that is a citation of a domain.

Retailers, publishers, platforms and standards are never coded as brands. They may be cited domains. A retailer's own-label machine, if ever named, is a brand.

The denominator. Every scored run on that engine, excluding errored runs, excluding control prompts, and excluding runs on prompts where that brand appears in the prompt text. Refusals stay in. The rate with refusals excluded is printed alongside, and both are published.

Recommendations

A recommendation is a stronger and rarer event than a mention, and it is the one a buyer acts on, so the rules are deliberately strict.

A brand is recommended when the answer singles it out as the answer to the question asked. "The one to get is X", "for most people, X", "X is the best choice for pet hair", "I would go with X".

Named in a list is not a recommendation. A brand appearing in a list of options, a comparison table, a shortlist or a set of good choices is a mention only. This is the single most consequential rule in the file, and it is the one a flattering study would soften. Lists are the common case; picks are not.

A labelled list produces a recommendation only for the labelled entries. "Best overall", "best budget", "best for pet hair": each labelled entry is a recommendation for its label, and unlabelled entries in the same list are mentions only. An ordered list with no labels and no language of preference is not a ranking and produces no recommendation, because position is not preference and engines order lists for reasons we cannot see.

Conditional picks count, with the condition recorded. "If your floors are mostly hard, get X" is a recommendation for X. A buyer reading that sentence has been told what to buy.

Hedged non-picks and negative picks are not recommendations. "Any of these would be fine", "it depends on your floors" with no condition attached to a brand, and "I cannot recommend a specific product" are all zero recommendations. "Whatever you do, do not buy X" is a mention with negative sentiment and a recommendation for nobody.

A brand named in the prompt can still be recommended. Removing prompted brands from the mention denominator is about discovery. A pick is different: the engine was asked to choose, and it chose. Prompted and unprompted recommendations are reported in separate columns and never summed into one figure. Every recommended brand must also appear as named or prompted, and the analysis script refuses to run on a file where it does not.

Citations

A citation is one URL shown by the engine, in one answer, to one prompt, on one engine, on one date. The unit is the URL instance. Every citation cut is reported twice, as an instance count that counts the same URL again each time it appears, and as a distinct URL count that collapses duplicates within the cut. The ratio between them is itself a result, and we fixed this before collection precisely because choosing afterwards would change the headline.

What is counted. Inline citation links in the answer body, entries in a sources or citations panel attached to the answer, and any other link the surface attaches to that answer. Not links in a "people also ask", a "related" strip, a suggested follow-up or an advertisement, because those are not sources for the answer.

Described without a link is not a citation. "According to a well-known consumer testing organisation", with no URL, is recorded as an unlinked attribution in plain text, and the count of those is published alongside the citation counts, because an answer that cites authority without a link is a different thing from one that links.

A citation is never credited to a brand. A link to a manufacturer's own site is a citation of that domain. It is not a mention of the brand unless the brand name also appears in the answer text. Domains are reported at the registrable level with the host-level data kept in the file so a reader can regroup it, and a redirect or tracking wrapper is recorded exactly as the engine exposed it, with the resolved destination beside it where the runner resolved it.

A sources panel that could not be fully expanded marks the row, and that row is excluded from citation counts while remaining in the mention and recommendation counts. The script prints how many rows were excluded that way.

A second scorer independently scores a random 10 percent of rows. Disagreements are counted and published as a scorer agreement figure, and each is resolved by the rule text. If a disagreement cannot be resolved by the existing rule text, the rule is too vague: we add the case as a new numbered rule, rescore every affected row under it, and say in the methods section that we did.

What would prove our own published position wrong.

These are published before collection so that the result cannot be chosen after the data arrives. Our published position is that one run is an anecdote: that repeats of the same prompt return materially different brand sets, and that a brand absent from one answer may still be named most of the time. The study contradicts that position if any of the following holds.

Any of those outcomes gets published with the same prominence as a confirming one. A study we can only win is not evidence. None of them has been tested, because no data has been collected.

What we will not claim from it, decided now.

Written before collection, because the time to decide what a result cannot support is before you have one.

Everything, which is the point.

The reason to publish a protocol before the run is that it can be held against the result afterwards. This page is the thing to hold us to.

When results exist, the results file, this protocol, the prompt set, the scoring rules and the analysis script are published together, at a stable URL, with the collection window in the page text rather than only in metadata, and the file under CC BY 4.0. Until then, this page is apparatus and should be cited as apparatus. It is not a finding, and there is no number from this study to quote.

The counting rules we apply in client work are set out on the methodology page, and the sample audit shows how a finding is evidenced in a report. Both are separate from this study, and neither carries a client result.

A protocol, published before the run.

45 prompts and 2 controls, three engines, two arms, 783 answers, and the conditions that would prove us wrong. No answers collected, no results published.

How an audit is run and scored →
Or ask a question about the design through the contact form