Blog · Running the work

What an AEO audit should contain

Published 8 October 2026 · Updated 12 October 2026 · 9 min read

What an AEO audit should contain
Photograph: Kelly Sikkema / unsplash

The short answer: An AEO audit should contain five things, each with evidence attached: proof that the assistants can fetch and read your pages, a prompt run repeated across several engines with retrieval on and off, a passage-level test of the pages that are meant to answer, an identity check that shows whether an engine can resolve who you are, and a ranked fix list with effort and owner next to each line. Anything else is commentary.

The word audit has been stretched until it covers a one-page score and a sales call. A real one is closer to a lab notebook. It records what was asked, what came back, and what a second person could repeat next week to check the finding.

Access, proved rather than asserted

Most audits say "your site is crawlable" and stop. The stronger version attaches the request and the response, because user-agent strings are trivially forged and Google's own crawler documentation says so in as many words. A WAF rule that allowlists the literal string OAI-SearchBot is not an access control. It is a welcome mat for anybody who types it.

The checkable version matches the requesting address in your own logs against the published ranges. OpenAI publishes one address file per crawler as plain JSON, and they are small enough to read by hand. We pulled three of them on 12 October 2026: gptbot.json carried 18 IPv4 prefixes, searchbot.json carried 39, and chatgpt-user.json carried 234. Those counts are a reading on a date, not a constant, and the files say so themselves: searchbot.json carried a creation stamp of 2 January 2026 while chatgpt-user.json carried one of 7 October

  1. An audit that lists your firewall rules against those files, on a stated date, tells

    2. you something. A green tick does not. Other engines publish their access information differently, so this check has to be rebuilt per engine rather than assumed.

Two details belong in this section and are usually missing. First, OpenAI's crawlers do different jobs: GPTBot gathers training data, OAI-SearchBot supplies ChatGPT's search results, and ChatGPT-User fires when a person in a chat asks for your page directly. OpenAI's documentation states that a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link, and that disallowing GPTBot is a statement about training rather than about search. That is OpenAI's split, and we do not assume any other engine divides its tokens the same way. Second, the same documentation notes that it can take about 24 hours from a robots.txt update for OpenAI's search systems to adjust, so an audit that re-tests ten minutes after a fix has proved nothing at all.

Google draws its own line, and it is worth writing down. Google-Extended governs Gemini training and grounding, and Google states plainly that it does not impact a site's inclusion in Google Search and is not used as a ranking signal there. We have sat in audits where a team had blocked it believing they were protecting Search rankings, which is a misread of that documentation and costs them Gemini surfaces for no gain.

The prompt run, and why one pass is worthless

The core of the deliverable is the run. Ours is a fixed set of about 45 prompts across five engines, executed with retrieval on and with retrieval off, and the raw answers kept so a finding can be argued with rather than believed.

Retrieval off matters more than most people expect. It separates what the model has absorbed about your brand from what it just read on your site this minute. If you appear only with retrieval on, your presence is rented from a live crawl and will move when a crawl or a competitor moves. If you appear with retrieval off, something durable has settled into the weights. Those two states need different work and different budgets, which is the argument running underneath What an AEO budget actually buys.

Repetition is the other half. The same prompt, fired twice an hour apart, can return a different source list. We see that often enough in our own runs to design around it, though we do not publish a rate, because we have not measured one in a form a reader could repeat. An audit that quotes a single run as a result is quoting something it has not shown to be stable. Every prompt should be run several times, and the report should carry the spread, not just the mean. A brand named in four runs out of ten is a genuinely different situation from one named in all ten, and a single number hides the difference.

Passage tests, because pages are not the unit

Here is the failure mode that catches careful teams. The page is indexed. It ranks. The crawler reaches it. And a thinner competitor page gets quoted instead.

That happens when the system scores passages rather than whole pages. Google documents a passage ranking system that identifies individual sections of a page, and the retrieval-augmented generation literature, Lewis et al. 2020, describes retrieval over passages rather than whole documents. We have not found a published description of how the assistants in our run split documents, so for a particular engine this is the working model we audit against, not something we can verify from outside. A strong page can still hold a weak candidate chunk. The causes we find most often are dull and fixable: the answer sits in paragraph nine under four hundred words of throat-clearing, or every paragraph leans on the one before it with openers like "as described above", so no chunk survives being lifted out.

So the audit should name the single page meant to answer each priority prompt, then test whether a self-contained answer exists inside the first screen of it. The check a reader can run alone: copy one paragraph into a blank document and ask whether it answers the question without the rest of the page. If it does not, the retrieval system has nothing to quote. The self-audit checklist for AI visibility covers the manual version of this.

Rendering belongs here too. Client-side rendering is the quietest way to lose, because the HTML arrives, the bot gets a shell, and your own logs record a successful 200. None of the crawler documents cited below commits to executing JavaScript before the content is used, and we cannot see from outside which assistants do, so text that appears only after hydration is better treated as unverified than as safe. Fetch the raw HTML with curl and search it for your answer sentence. If the sentence is not in the bytes, you cannot show it reached the engine.

Identity, and whether the engine can resolve you

In our runs, answers about a company get vaguer when the company on its pricing page, the company in its JSON-LD, the company in a directory listing and the company in a trade article do not obviously resolve to one entity. We cannot see inside the retriever, so we cannot show the contradiction caused the hedge, only that we keep finding the two together. A hedge reads to the buyer as absence either way.

The audit should record how your organisation is described in each of those places and flag the contradictions. Google's Article guidance is unusually relaxed about required properties, stating there are none and telling you to add what applies, which means markup is rarely the binding constraint. Consistency is the harder constraint. Two different founding years and three different company suffixes are the kind of contradiction we flag ahead of a missing dateModified.

This is also where the audit should say what it cannot see. Third-party corroboration moves on its own schedule and no report can date it.

The fix list, ranked and costed

The last section is the only one anyone acts on. Each finding gets the evidence that produced it, an effort estimate, an owner, and an honest expected lag. Access repairs tend to show up inside a crawl cycle. Passage rewrites take longer and depend on a re-crawl you do not control. Corroboration is slowest and least predictable, so a date attached to it is a guess wearing a suit.

Sequencing matters as much as the list. Fixing prose on a page that a firewall blocks is money set on fire. The first 90 days of AEO work sets out the order we use, and the naming argument in AEO vs GEO vs SEO matters far less than whether one person owns this list. The sample audit report walks through the shape section by section.

What an audit cannot tell you

It cannot tell you revenue. Referral data from these surfaces is patchy, some visits arrive with a referrer that names the assistant and many do not, and we have not measured that split in a form we would publish. So a mention in an answer and a deal in your pipeline stay unjoined by anything stronger than timing.

It cannot promise durability either. Engines change their retrieval behaviour without notice, and a finding that held in March can stop holding in September. Our own index gets re-run monthly for exactly that reason, and the numbers move.

Our audit produces one deliverable: the report. Scope varies by client, so the section list above is the shape of the thinking, not a fixed table of contents.

Honest limits

The access section rests on documents OpenAI and Google publish about their own crawlers, and we have checked nothing comparable for the other engines in our run, so read those parts as being about ChatGPT and Google surfaces rather than about assistants in general. The address counts above are one pull on 12 October 2026 and will be wrong on some later date. The passage argument is a working model drawn from Google's passage ranking documentation and the public retrieval literature, rather than from a published description of how the engines in our run split documents, which we have not found for them. Our claims about run-to-run variance and about hedging come from our own prompt runs rather than from a controlled study, and we do not quote a rate for either because we have not measured one an outsider could reproduce.

FAQ

How long should an AEO audit take? Ours runs seven working days. Most of that is the prompt run and the repetition, because a single pass across five engines is quick and tells you very little.

Can I run this myself? The access and rendering checks, yes, with curl, your robots.txt and the published bot IP files. The repeated multi-engine prompt run is where the manual version gets expensive, because the cost is in doing it the same way every time.

Does an audit include the fixes? No. The audit's deliverable is the report. Shipping the fixes is separate work, and keeping them shipped is separate again.

Should I block AI crawlers while I audit? By OpenAI's documentation, opting out of OAI-SearchBot keeps you out of ChatGPT's search answers, which makes the audit measure a situation you created. Each engine has its own tokens and its own consequences, so check them one at a time. Test first, decide after.

Sources

Related reading

The first 90 days of AEO workWhat an AEO budget actually buysA self-audit checklist for AI visibilityAEO vs GEO vs SEO: what the three terms actually meanGetting engineering to ship your AEO fixesAn AEO roadmap you can defend in a planning meeting