Blog · Diagnosis

How to check what engines say about you, properly

Published 22 September 2026 · Updated 8 October 2026 · 9 min read

How to check what engines say about you, properly
Photograph: MJ Duford / unsplash

The short answer: Checking properly means three passes, not one. Ask buyer questions rather than your own name, several times per question, on more than one engine. Run each question twice, once where the engine can browse and once where it cannot, so you learn whether a mention came from retrieval or from what the model already believed. Then read your server logs to confirm which bot actually fetched you.

Typing your company name into ChatGPT and reading the paragraph that comes back is not a check. It is a mood. The engine has been handed the answer in the question, so of course your name appears, and you have learned nothing about the prompts a buyer would really use.

Ask the question a buyer asks, not the one that names you

Buyers rarely type brand names at the start. They describe a constraint. Something like "payroll software that handles contractors in India and the UK" or "who does SOC 2 audits for a twelve person startup". Your name is the output they are hoping for, not the input.

So build the prompt list from the shape of a purchase, not from a keyword tool. Mix four kinds: the category question, the comparison question, the objection question, and the fit question. Only a handful should carry your brand name, and those exist to test accuracy rather than presence.

There is a mechanical reason the unbranded prompts matter more than people expect. Google documents a query fan-out in AI Overviews and AI Mode: the system issues multiple related searches across subtopics and data sources, which it says lets it surface a wider and more diverse set of supporting pages. On those two Google surfaces, then, you are not competing for one query but for a family of queries nobody typed, generated from the one that was, and testing only the literal phrasing never shows you the family. Other engines do not publish an equivalent description, so treat fan-out as a documented Google behaviour, documented for those two surfaces, and not as a property we can claim elsewhere. The deeper version of that argument sits in Why AI never mentions your brand.

Run it twice, once with browsing and once without

This is the single change that turns a transcript into evidence.

Ask the same question in two conditions. First with the engine allowed to search, then in a mode or a model where it cannot, so it answers from parameters alone. Four results are possible and each one means something different.

Named in both: the model has absorbed you and the live web agrees. Comfortable position.

Named only with browsing on: your presence is rented. It depends on pages being fetchable today, and a blocked crawler or a slow origin can delete you tomorrow.

Named only with browsing off: the model remembers something, often an old funding round or an old positioning, and the current web is not reinforcing it. Check the claim carefully. Stale memories are where wrong pricing and dead product names come from.

Named in neither: nothing to debate. Access, identity and corroboration all need work.

Record which condition produced which answer, every time. A visibility number that mixes the two conditions together cannot tell you what to fix on Monday morning, because the repair for a retrieval problem and the repair for a knowledge problem are different jobs.

Repetition is not optional, and three runs is the floor

Answers are generated per request rather than read from a stored result. Across the runs we do in audit work, two people asking the same thing a minute apart can get different sources, and so can one person asking twice. None of the engines publish how this works, so read it as a pattern in the transcripts we collect rather than as a documented property.

Treat a single run as an anecdote. Run each prompt at least three times, five if the answer matters to a board slide, and write down the mention rate rather than a yes or no. A brand named in two runs out of five is a real and reportable state. Calling it "we appear in ChatGPT" is not.

Timestamp every run. When a mention rate moves, the first question is whether your site changed or the engine did, and only the timestamps can separate those.

Read your logs, because the transcript is only half the evidence

The part most checks skip entirely. An answer tells you what the engine said. Your access logs tell you what it fetched, and the two disagree more often than you would like.

Filter the logs for the documented agents and keep them apart, because they mean different things. OpenAI publishes four: GPTBot for model training, OAI-SearchBot for the search surface inside ChatGPT, ChatGPT-User for pages fetched because a person asked right now, and OAI-AdsBot for ad safety checks. A ChatGPT-User hit on your pricing page is a live buyer in progress. A GPTBot hit is a deposit into a model that ships months later. Those are not the same event and a dashboard that merges them is lying to you politely.

Perplexity draws the same line and documents it bluntly: PerplexityBot crawls for the search index and obeys robots.txt, while Perplexity-User, which fires when a person asks a question, generally ignores robots.txt because a human initiated the request. The same page tells you to whitelist both agents in a web application firewall, which is the giveaway: a WAF can still refuse them. Read that twice if you run one. It means a rule in your edge configuration can block the fetch that a real prospect triggered while your robots.txt looks perfectly welcoming.

Google splits its tokens too. Google-Extended is a robots.txt control token, with no user agent string of its own, that governs use of your content in Gemini Apps and in Grounding with Google Search on Vertex AI. Google states plainly that it does not affect inclusion in Google Search and is not a ranking signal there. That is Google's token only, and says nothing about any other engine's. Blocking it is a licensing decision, not a technical one.

Verify the agent string before you believe it. User agents are trivially spoofed, so match the source address against the published ranges: OpenAI lists them at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json. In our own log work we routinely find traffic claiming to be an AI crawler that fails that check, usually scrapers borrowing the name. We have no method written down that would let you reproduce a rate, so we are not publishing one: take it as a reason to verify every agent string against the published ranges, not as a number to plan against.

Three checks worth running this week

Confirm the fetch path. Request one of your key pages with curl using a documented AI agent string, then request it from a cloud IP. If the second one returns a challenge page or a 403 while a browser gets a 200, your edge rules are the bottleneck and no amount of content work will move the answer.

Diff what the engine quotes against what you published. When an answer paraphrases you, find the sentence it came from. If that sentence does not exist on your site, the engine is stitching it from a directory listing or a review page, which tells you where your identity actually lives.

Check the entity, not the page. Ask each engine "what is <company>, who runs it, where is it based, what does it cost" and compare the four answers against your own schema. Mismatch usually traces back to markup, which is the subject of Organization schema done properly.

What this method cannot tell you

Attribution. In the client logs and analytics we have looked at, the AI answer surfaces either send no referrer or send one too coarse to tie to a single answer, so a mention and a signup in your CRM cannot be joined with any honesty. We cannot inspect what we are not sent, so that is a statement about our own visibility, not proof that no signal exists.

Coverage. You are sampling a space of possible questions that is effectively infinite. A good prompt set is representative, never complete.

Causation. Engines ship changes without telling anyone, and a mention rate can move while your site sits untouched. Keeping a dated baseline is the only defence, and it is also why a replatform needs its own protocol, covered in Migrating a site without losing citations.

Scale is the honest limit on doing this by hand. Fifteen prompts, five engines, two retrieval conditions and three runs is 450 transcripts, and that is a slow week for one person. Our audit is the compressed version: about 45 prompts across five engines with retrieval on and off, evidence attached to each finding, delivered as a report. If you want the first pass without spending anything, the self-check lists ten things you can test this afternoon. The vocabulary, if any of the terms above were unfamiliar, is in an AI search glossary that defines the terms properly.

Honest limits

This post documents four things with sources: OpenAI's four agent tokens and their IP range files, Perplexity's split between its crawler and its user fetcher, Google's query fan-out on AI Overviews and AI Mode, and Google-Extended's scope. Everything else here is method, and method is argued rather than proved. The run counts we suggest, three as a floor and five when the answer matters, come from watching transcripts vary in our own audit work and are not derived from a published study; a statistician would want a stated variance and a power calculation before accepting them, and we have neither. We have deliberately removed the one proportion this post used to carry about spoofed crawler traffic, because we could not state the method, sample and date behind it. Where an engine publishes nothing, as most do about regeneration and about referrer behaviour, we can tell you what our logs and transcripts show and no more.

FAQ

How many prompts do I actually need? Fifteen well chosen questions beat a hundred generic ones. Stop adding prompts when new ones stop surfacing new sources or new wrong claims.

Does blocking an AI crawler show up in these checks? Yes, and it shows up as absence rather than as an error, which is why log evidence matters. Check the agent tokens separately, since the one that trains models and the one that powers the search surface are different robots with different consequences.

Why do two people get different answers to the same question? Session context, location, account history and plain regeneration variance. Run the prompt several times from a clean session before you conclude anything about a difference.

Can I rely on Search Console for AI traffic? Partly, and further than it used to go. Google folds impressions and clicks from AI Overviews and AI Mode into the overall Web search type in the Performance report, and since June 2026 also breaks impressions out in a dedicated Generative AI performance report, available to all sites worldwide since 31 August 2026. That report gives impressions and not clicks, so you can see the AI surface on its own for impressions but not isolate its clicks.

Is a low mention rate always a content problem? No. Start with access, because a page that cannot be fetched or rendered inside the engine's time budget never reaches the content stage at all.

Sources

Related reading

Why AI never mentions your brandMigrating a site without losing citationsAn AI search glossary that defines the terms properlyOrganization schema done properlyContradictions across your own pagesDiagnosing a drop in AI citations