Blog · Running the work

How to buy an AEO audit without being sold a dashboard

Published 27 April 2026 · Updated 8 October 2026 · 7 min read

How to buy an AEO audit without being sold a dashboard
Photograph: Luke Southern / unsplash

The short answer: Buy the diagnosis, not the login. An audit should give you a written report that names specific failures, attaches the evidence behind each one, and ends in fixes an engineer can ticket on Monday. A dashboard gives you a score. Ask to see the report shape before you ask for access, and ask how many runs sit behind every number printed in it.

That one swap in the buying order filters out most of the market. Scores are cheap to produce and expensive to act on. Findings are the opposite.

A dashboard and an audit answer different questions

A monitoring product answers "did the number move". A diagnosis answers "why, and what do we change". Both are legitimate. They are sold at the same price and bought by the same person, which is where the confusion starts.

The test is what you are left holding at the end. If the engagement ends and you keep a document with findings, evidence and a fix list, you bought a diagnosis. If the engagement ends and you lose access to everything, you bought a subscription with an onboarding call attached. Neither is wrong. Only one survives a cancelled contract.

There is a second difference that rarely shows up in a demo. A dashboard is built to be generic, because the same screen has to work for a payments company and a mattress brand. A diagnosis is scoped to your site, your buyers and the twenty questions that actually decide your deals. Generic screens are why so many teams open a visibility tool twice a week for a month and then stop.

Every AI visibility number is a sample, not a measurement

This is the mechanism most buyers never get told, and it changes what a fair audit can honestly claim.

No engine publishes per-query answer logs. Google's own guidance says AI features are not ranked or tracked separately, and that sites appearing in them are folded into overall Search performance reporting rather than broken out. So nobody, at any price, is reading your AI Mode position from an official feed. Every number in this category is produced by asking the engine a question repeatedly and counting what comes back.

That has a consequence worth printing on the invoice. A visibility percentage is an estimate with a sample size behind it, and the sample size is usually hidden. One run per prompt per week is common. Answers regenerate on every request, so a single run is a coin toss dressed as a metric, and a week-on-week delta built from single runs is mostly noise.

Ask the number that nobody volunteers: how many independent runs produced this figure, and what was the spread across them. A vendor who can answer that is measuring. A vendor who cannot is rounding.

The academic work points the same way. The paper that introduced the term generative engine optimisation reported visibility gains of up to 40 percent from content-side changes, measured over a purpose-built benchmark of queries rather than a live dashboard, precisely because repeated controlled runs were the only way to see the effect at all.

Five questions that separate a diagnosis from a subscription

Send these before the call. The answers arrive in writing, which is the point.

1. What exactly do I own when this ends? You want a file. A report, a document, a PDF. If the answer is a seat in a product, the deliverable is access and the access expires.

2. How many prompts, across how many engines, and how many runs each? Numbers, not adjectives. Our own audit runs a set of roughly 45 prompts across five engines, with retrieval switched on and switched off, because the gap between those two conditions is where the interesting failures hide. Any shape is defensible. Vagueness is not.

3. Is retrieval-off tested? With retrieval disabled, the model answers from what it absorbed during training. With retrieval enabled, it goes and fetches. A brand that shows up only with retrieval on has an access-and-content story. A brand that shows up only with retrieval off has a stale-memory story. These need opposite fixes, and a dashboard that reports one blended score cannot tell you which one you have.

4. What evidence is attached to each finding? A finding without an artefact is an opinion. You want the question that was asked, what came back, and the specific thing on your site or off it that explains the result. The sample audit report shows the shape of that, section by section.

5. What does the fix list look like, and who can execute it? Findings that terminate in "improve your content authority" are not findings. Findings that terminate in "this robots rule blocks the search crawler, remove line 14" are. If the fix list cannot be handed to a developer and a writer without a translation layer, the audit has not finished its job.

Run the free checks before you spend anything

Two of the most expensive problems we find are free to detect, and finding one yourself changes what you are buying.

Open yourdomain.com/robots.txt and read it. OpenAI documents its crawlers separately: GPTBot collects training data, OAI-SearchBot is the one that surfaces pages inside ChatGPT search, and the documentation states plainly that sites opted out of OAI-SearchBot will not appear in ChatGPT search answers. Plenty of sites blocked everything with an AI-sounding name in 2024 and never revisited it. If that line is in your file, no audit and no dashboard will move your numbers until it is gone.

Second, disable JavaScript in your browser and reload your three most commercially important pages. If the pricing table, the product description or the comparison content disappears, the passage an engine needs to quote does not exist in the fetched HTML.

The self-check has eight more in the same spirit. Running them first means you arrive at a sales conversation knowing which answers should already be obvious, which is a surprisingly good vendor test on its own.

What the money is actually doing

Price is where buyers lose their nerve, usually because the range in this category spans two orders of magnitude for things with the same name on the invoice.

Our audit is $500 and takes seven working days, and the only deliverable is the report. Not a prompt set, not raw exports, not a login. We say that plainly because the deliverable is the single most-fudged line in this market.

The report is also a sequencing instrument, which is its quieter value. It tells you what to do in the first 90 days of AEO work and, just as usefully, what to leave alone until the access layer is clean. Teams that skip the diagnosis tend to start with content, because content is the part marketing controls, while the actual loss is a bot rule or a client-side render. What an AEO budget actually buys walks through where the money goes once the report exists.

One warning about scope. An audit tells you where you stand and why. It does not create the thing that gets you cited, which is usually a page carrying information nobody else has. Original data gives an engine something only you have covers that work, and it is a build decision, not an audit finding. The same applies to whether an assistant can complete a transaction on your site, which an agent trying to book, buy or fill your form will expose faster than any score.

Where this advice runs out

Three honest limits.

Sampling cuts both ways. Our audit uses repeated runs with the evidence kept, and it is still a sample taken on specific days. A finding is a strong signal, not a permanent fact about the engine.

Attribution stays unsolved. Pew Research Center, reporting in July 2025 on browsing data from 900 US adults collected that March, found users clicked a result link on 8 percent of visits to a page carrying an AI summary against 15 percent without one. Fewer clicks means less of your influence shows up in analytics at all, whoever you buy from.

And an audit cannot make an organisation act. We have watched good reports sit for a quarter because nobody owned the robots file. If you have no route from a finding to a deployed change, buy the diagnosis later and fix the ownership gap first.

FAQ

Is a dashboard ever the right purchase? Yes, after a diagnosis, when you already know which prompts matter and want to watch them over months. Monitoring is a maintenance tool. It is a poor first purchase because it tells you a number without telling you the cause.

How do I know an audit is not just a dashboard export? Look for findings that could only come from your site. A report that would read identically with another company's name swapped in was generated, not investigated.

What sample size should I accept per prompt? Enough that the vendor will state it. Single runs per prompt are not defensible for anything you plan to report upward. Ask for the spread across runs, not just the average.

Should I audit before or after a site redesign? Before, if the redesign is already scoped, so the fixes ride along with work that is happening anyway. After is still useful, and it costs more because the changes become a separate release.

Sources

Related reading

The first 90 days of AEO workWhat an AEO budget actually buysOriginal data gives an engine something only you haveCan an agent book, buy or fill your formChoosing an AI visibility toolGovernance, and one source of truth for brand facts