Blog · Data and trends

Running your own AI search study

Published 19 June 2026 · Updated 12 October 2026 · 9 min read

Running your own AI search study
Photograph: Fulvio Ciccolo / unsplash

The short answer: Running your own AI search study means fixing four things in writing before the first call goes out: the question, the sampling frame, the counting rule, and the analysis you will run on the result. Everything after that is logging. In the studies we have been asked to check, the usual fault is that one of those four was decided after the data arrived, not that the sample was too small.

A study is worth running for two reasons. It tells you something about your own market that no general report covers. And a defensible dataset is one of the few assets engines will quote back, which is why original data earns citations that opinion pieces do not.

Fix the counting rule before you look

This is the decision that changes a headline by a factor of two, and almost nobody writes it down first.

Say you collect two million cited URLs over a month. Do you count every citation instance, or do you collapse duplicates and count unique URLs? Citation sets tend to concentrate on a few reference domains: in the Pew Research Center's April 2025 scrape of Google AI summaries, Wikipedia, YouTube and Reddit together accounted for 15 percent of all sources listed. We have not measured that concentration on the other engines. Counting instances makes repeat domains look dominant. Collapsing to unique URLs makes a long tail of one-off pages look dominant instead. Both numbers are true. They answer different questions, and a study that quietly switches between them in the same article is not a study.

Write the rule as a sentence you could hand to someone else: "a citation is one link shown in one answer to one prompt on one engine on one date, and we report both instance counts and distinct URL counts for every cut." Then publish both columns. The ratio between them is often the most interesting thing in the dataset, because it tells you whether a surface rewards one canonical page or many shallow ones.

The same discipline applies to the brand side. A mention with no link, a link with no mention, and a mention inside a list of five competitors are three different events. Decide which one counts, and stick with it even when the alternative would be a better headline. The gap between these events is what citation share and mention share are separately measuring.

Your sampling frame is the only thing you can generalise to

The prompts you choose are not a window onto AI search. They are the population. Every claim you make applies to that population and nowhere else.

That sounds obvious and it is routinely broken. A study built from an existing keyword tracker inherits whatever that tracker already followed, which in the trackers we have been handed skews commercial, English and high-volume, because that is what people pay to track. A study built from your own support inbox skews to problems rather than purchases. Neither is wrong. Both need a methods paragraph that says so, in the study itself, not in a footnote.

The cleanest frames we have been shown, which is an impression from audit work rather than a measured comparison, are built from an explicit list drawn before any data collection: a fixed set of buyer questions, a fixed set of engines, a fixed window of dates. If you want a worked version of that, the method behind building a prompt set is the same method, scaled down to one company.

One check worth running on your own frame: count how many of your prompts could only have been typed by someone who already knows your category's jargon. If it is most of them, you have measured your own echo, not the market.

Report per engine, always

Pooling engines into one number creates a confound that will eventually embarrass you.

Suppose your overall citation share rises from 11 percent to 14 percent across a quarter. That can happen because your pages got better. It can also happen because the mix of engines in your sample shifted toward the one that already liked you, while nothing about your site changed at all. With a pooled number you cannot tell these apart. With a per engine table you can, in about thirty seconds.

Engines differ enough that pooling can lose real information. We can only speak to the engines we run, and only for the dates we ran them: in our own audit runs, a finding that holds on one engine often fails to appear on another in the same week. We have no visibility into the ranking or sourcing logic behind those results, so we cannot say why, and none of the five engines we run publishes enough for us to call this a permanent property rather than a snapshot. Treat each engine as a separate experiment that happens to share a prompt list, which is what engine overlap and divergence describes in detail.

Effect sizes are conditional, so publish the subgroups

The academic work on this is useful here. The GEO paper presented at KDD 2024 reported visibility gains of up to 40 percent from content-side changes, measured on its own GEO-bench query set rather than on live commercial traffic, and was explicit that the size of the gain varied by domain. That second clause matters more than the headline. An average uplift across all domains is a number no single reader can act on.

So cut your results by the dimensions that plausibly change the mechanism: query intent, industry, prompt length, whether the prompt names a brand. Publish the subgroup sizes next to the subgroup results. A 40 point swing computed on eleven prompts is a story about eleven prompts.

Correlations near zero deserve the same airtime as strong ones. If engagement metrics turn out to have no measurable relationship with citation frequency in your data, that is a finding, and a more useful one than most positive results, because it stops teams spending money.

What a methods section has to contain

Copy the structure from a study that has survived scrutiny. The Pew Research Center's July 2025 work on AI summaries is a good model: it states the panel, the dates, and the exact counts. The researchers tracked browsing for 900 US adults through March 2025, collected 68,879 unique Google searches of which 12,593 produced an AI summary, and scraped the result pages between 7 and 17 April 2025. The headline, that users clicked a traditional result on 8 percent of visits with a summary present against 15 percent without, is checkable because every one of those numbers is on the page.

Your methods section needs the same seven items: the frame, the collection window, the engines, the raw counts at each filtering step, the counting rule, the analysis performed, and what you excluded and why. If a reader cannot reconstruct your denominator, you have published a claim, not a study.

Publish it so it can be quoted

A study nobody can parse gets cited as "one analysis" with no brand attached, which defeats the purpose. Three practical steps.

Put the number in the heading and the sentence under it, in plain text. A figure that exists only inside a chart image cannot be quoted from the page text, and we have no way to confirm whether an engine has read it from the image instead, which is the same problem as content engines cannot read.

Mark the dataset up. Google's Dataset structured data documentation requires only name and description, and recommends temporalCoverage, variableMeasured, measurementTechnique, license and a distribution pointing at the actual file. Google says supporting information of this kind makes a dataset easier to find in its Dataset Search, and it also gives any system that reads the page a machine-readable statement of what you measured and when.

Give it a stable URL and date the updates in the page, not just in a sitemap. Studies get quoted for years, and an undated figure attached to your brand will be repeated back to you long after it stopped being true. Make sure the numbers you publish agree with the numbers elsewhere on your site, because contradictions across your own pages are resolved by the engine picking one, not by it asking you.

Honest limits

This post is a method, not a measurement. It reports no original figures of our own: the only numbers in it come from the Pew Research Center's 2025 panel study and from the GEO paper, and both are scoped to what those authors measured, one panel of 900 US adults on Google in March and April 2025, and one academic benchmark. Neither is evidence about how the engines behave today. Where this post describes our own experience, with prompt frames and with per engine divergence, that is an impression from audit work rather than a controlled comparison, and we have not published a sample size or a method for it, so treat it as a reason to check rather than as evidence. What follows is what we still cannot do cleanly.

We cannot connect a citation in an answer to revenue, because in the client analytics we have access to, assistant traffic mostly arrives with no usable referrer. We cannot see what other analytics stacks record, so we say this about our own view rather than about the surfaces themselves. Any study claiming a conversion rate from AI answers is inferring it.

We cannot hold the engines still. A model update mid-study will move your numbers for causes unrelated to the variable you are testing, which is why a date range is part of every claim rather than a decoration on it. Running the same study twice, a month apart, is the only cheap defence.

And a study of your own market is not evidence about AI search in general, however much the headline wants to be. Resist the generalisation. The specific version is more credible anyway, which is part of why AI search is not a ranking game: there is no league table to top, only a question about whether your evidence holds.

If you want the small version before committing to the large one, the self-check is ten tests you can run this afternoon with a spreadsheet, and it uses the same counting discipline at one hundredth of the scale. The numbers worth lifting from published work are collected in AI search statistics worth quoting, and the demand side of the question, what people are actually asking these systems, is covered in how buyers are actually using AI search.

FAQ

How many prompts does a credible study need? Fewer than people assume, if the frame is honest. Forty to sixty prompts run across several engines will support claims about those prompts. No sample size rescues a frame that was picked to flatter the result.

Should I publish the raw data? If you can, yes, with a licence attached. A downloadable file with Dataset markup is far more quotable than a chart. Strip anything client-identifying first.

Is one month long enough? For a snapshot, yes, if you say it is a snapshot. For a trend claim you need at least three collection windows, because a single move between two points cannot be separated from normal volatility.

What is the most common mistake? Choosing the counting rule after seeing the data. It is rarely deliberate, and it invalidates the result anyway.

Sources

Related reading

AI search statistics worth quotingHow buyers are actually using AI searchContradictions across your own pagesWhy AI search is not a ranking gameWhat changed in AI search this yearZero click, and what replaces the click, checked in your logs