Blog · Engines

How each AI engine picks the sources it cites

Published 8 October 2026 · Updated 12 October 2026 · 9 min read

How each AI engine picks the sources it cites
Photograph: Paul Hanaoka / unsplash

The short answer: The engines whose documentation we can read describe the same four stages. The model rewrites your question into several queries, retrieves candidate documents through a search index or a live fetch, cuts them into passages and keeps a handful, then attaches links to the sentences it wrote. What differs between engines is which robot does the fetching, what makes a page eligible in the first place, and how many sources survive to the citation list.

That framing matters because in the audits we run, the visibility problems we find are usually eligibility problems at one identifiable stage rather than ranking problems. That is what we see in our own client work, and we have not measured how often it holds elsewhere.

The four stages, and where each engine diverges

Stage one is reformulation. Google documents this openly as query fan-out: the model issues a set of concurrent related searches rather than the single question you typed, so an answer about fixing a weedy lawn pulls pages about herbicides, chemical-free removal and prevention. Your page is competing against sub-questions you never targeted.

Stage two is retrieval. Google's guidance says its AI features rely on its core Search ranking systems to retrieve pages from the Search index. OpenAI's and Perplexity's crawler documentation each describe a separate search crawler that surfaces sites inside their own product, so sitting in Google's index does not put you in either of theirs.

Stage three is passage selection. The engine keeps the few chunks that look like they answer the sub-question. Stage four is attribution. No lab we can find publishes how links are matched to sentences, so we treat a citation as evidence that a document was in the candidate set rather than proof that the claim came from it. That is our reading, not a documented mechanism, and it is checkable one URL at a time: open the cited page and look for the cited claim.

The fetcher is not one robot, and it is not one rulebook

This is the part we most often find misconfigured in audits. The three labs whose documentation is linked below each run several agents with different jobs and different robots.txt handling, and blocking the wrong one costs you the thing you wanted.

OpenAI publishes four: OAI-SearchBot for ChatGPT's search features, GPTBot for model training, OAI-AdsBot for ad safety checks, and ChatGPT-User for actions a person started. The documentation is direct about the last one: ChatGPT-User does not crawl automatically, and because a human initiated the request, robots.txt rules may not apply. Perplexity runs PerplexityBot for search indexing and Perplexity-User for live fetches, and its documentation says that because a user requested the fetch, Perplexity-User generally ignores robots.txt rules.

Anthropic splits its robots three ways too, across ClaudeBot, Claude-SearchBot and Claude-User, but its answer on the second question is the opposite one: the article states that its robots honour industry standard directives in robots.txt, and it gives no user-initiated exception. So two labs document a user-triggered door that your robots.txt rule may not govern, and a third says the directive is honoured. Read the rule per engine rather than once.

A number worth checking yourself

OpenAI and Perplexity publish the address ranges their agents use, one JSON file per agent, so an operator can verify a request rather than trust a user-agent string. Our method was one curl of each of the five files on 12 October 2026, counting the entries in the prefixes array and reading the creationTime field. That is a single snapshot on one day, not a time series, and the URLs are in the sources below so you can repeat it.

On that reading, OAI-SearchBot listed 39 CIDR prefixes, stamped 2 January 2026. GPTBot listed 18, stamped 22 September 2026. ChatGPT-User listed 234, stamped 7 October 2026. PerplexityBot listed 8, stamped 7 February 2025, and Perplexity-User listed 4, stamped 17 October 2025.

Read that spread again. On the day we looked, the OpenAI agent that fetches a page because a real person asked about you published an address pool about six times the size of its search crawler's. We cannot tell you how often any of these files change, because we have one reading of each rather than a history, so we are not claiming a rotation rate. What one reading does show is more than a year and a half between the oldest stamp and the newest, which means a WAF allowlist, bot-management rule or CDN policy copied from these files months ago may no longer match what they hold today. Re-fetching them and diffing against your rules takes about a minute.

Google publishes an eligibility gate in writing

Google's documentation states the condition plainly: to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. That single sentence does more work than most optimisation checklists. It is a condition for Google surfaces. We have not found an equivalent published gate in OpenAI's, Anthropic's or Perplexity's documentation, which is not the same as saying that none of them applies one.

It means noindex removes you. It means nosnippet removes you. It means data-nosnippet around the paragraph that answers the question removes exactly the passage you most wanted lifted, while the rest of the page stays eligible. We keep finding those attributes sitting in client templates long after whatever prompted them had passed. That is what turns up in the audits we run, and we have no figure for how widespread it is.

Google is equally clear about what does not help. Structured data is not required for generative features, there is no special schema.org markup for them, and the guidance says you do not need to create new machine-readable files or AI text files to appear. The same page names llms.txt among the tactics a publisher can ignore for Google Search. Note what it does not say. It does not say the file is ignored by Google's crawlers, and it says nothing at all about what other engines do with it. What changes for publishers under that gate is covered in Google AI Mode: what changes for a publisher.

ChatGPT's version of the gate is softer. OpenAI says a site that opts OAI-SearchBot out will not be shown in ChatGPT search answers, though it can still appear as a navigational link. The mechanics of what makes the cut there are in How ChatGPT decides what to cite.

Why two engines cite completely different pages

Teams often assume one of the engines is broken when the citation lists disagree. They are measuring different corpora with different retrieval, so disagreement is the expected result, not an anomaly.

There is a second cause, and it is statistical. A 2026 preprint on measuring generative search, *Quantifying Uncertainty in AI Visibility*, sampled citations on three generative search platforms, Perplexity Search, OpenAI SearchGPT and Google Gemini, across three consumer product topics, daily over nine days and again at ten-minute intervals. It reports citation distributions following a power law, bootstrap confidence intervals in which many apparent differences between domains fall inside the noise floor, and rank instability across samples throughout the frequently cited set, not only at the top. Three platforms and three topics is a narrow base, so read it as evidence that single-run numbers can mislead rather than as a measured instability rate for your own category. Its argument is that a visibility number is a sample estimate and should carry an uncertainty range.

A third cause is that much of what gets cited is not yours at all. Directories, forums, review sites and trade press carry the claim, and the engine cites them. That pattern is the subject of The third-party sources AI engines keep citing.

Three checks, about ten minutes

Check the doors separately. Grep your access logs for OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot and Claude-SearchBot over the last thirty days. A zero for any search agent on a site that has been live for a year is a finding, not a quirk. OpenAI notes that for search results it can take around 24 hours from a robots.txt update for its systems to adjust, and Perplexity's documentation gives the same figure for its own, so test and wait rather than test and panic.

Check the snippet gate. Search your templates for nosnippet, data-nosnippet and max-snippet. Then view the rendered source of your three highest-intent pages, not the template, because tag managers inject these too.

Check what your documentation URLs actually return. OpenAI's own bot page is a live example: platform.openai.com/docs/bots returned a 301 to developers.openai.com/api/docs/bots when we checked it on 12 October 2026. Redirects are normal and fine. Chains of them, or a redirect to a page that no longer holds the content, quietly strand citations, which is covered in Redirect chains and lost sources. The self-check is ten of these, written out, with no tooling needed. The sample audit report shows how we attach that evidence to a finding so it can be argued with later.

Honest limits

No lab publishes its ranking function for citations, so the selection described here is assembled from crawler documentation plus observed outputs, and the passage and attribution stages in particular are our inference rather than documented mechanism. The prefix counts above are one curl of five JSON files on 12 October 2026; they say nothing about how often those files change, and a reading taken on another day could differ. The claim that visibility failures are usually eligibility failures comes from the audits we run, which is a client set rather than a sample of the web, and we have no figure to attach to it. The preprint we cite measured three platforms and three consumer product topics, which supports the direction of its instability argument and not a rate you can apply to your category. We re-run our prompt sets monthly because the answers move, and we have watched a fix hold for six weeks and then stop mattering when an engine changed its retrieval, so treat every finding here as true on the date attached to it.

FAQ

Does allowing the AI crawlers improve my chances? Allowing the search and user agents makes you eligible on the engines that run them, which is a precondition rather than a result. OpenAI's crawler documentation says its robots.txt settings are independent of each other, so the training crawler is a separate decision from search.

If ChatGPT cites my page, did it read my page? Not necessarily. Attribution is attached after generation, so a citation tells you the document was in the candidate set. Verify by checking whether the cited claim is actually present on the cited URL.

Why did my competitor appear and then disappear within a week? Repeated sampling of the same prompt returns different sources. A 2026 preprint sampling three generative search platforms found citation rankings unstable across repeated samples, so one appearance and one absence are two data points rather than a trend.

Does structured data get me cited? Google says its generative features need no special schema.org markup and that structured data is not required for them. Google still recommends it for rich results, and it helps make your entity unambiguous, which is a different argument from citation. Other engines do not publish a comparable statement either way.

Sources

Every URL below was opened on 12 October 2026 and carried the statement attributed to it.

Related reading

How ChatGPT decides what to citeGoogle AI Mode: what changes for a publisherThe third-party sources AI engines keep citingRedirect chains and lost sourcesWhat a model update does to your visibilityHow Perplexity picks sources, and how to track the URLs