Blog · Access and retrieval

The AI crawlers and user agents worth knowing

Published 8 October 2026 · Updated 15 October 2026 · 11 min read

The AI crawlers and user agents worth knowing
Photograph: Abi abdullah miyad / unsplash

The short answer: Roughly a dozen tokens decide whether you appear in AI answers, and they split three ways. Training crawlers, such as GPTBot, ClaudeBot and Google-Extended. Search crawlers that build the index an answer is written from, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot. User-triggered fetchers, such as ChatGPT-User and Perplexity-User, which fire when a person pastes your URL. OpenAI and Perplexity both document what blocking the second group does: you stop being surfaced in their answers. Blocking the first group is a training opt-out, and at those two vendors it is documented as a separate setting.

That three-way split is the whole lesson. Everything below is detail.

The list, in one table

Every row below was read off the operator's own documentation on 15 October 2026. Match your rules on the token, never on the full header: the version moves, and the rest of the string is shared with ordinary browser user agents.

| Token | Operator | Job | robots.txt | |---|---|---|---| | GPTBot | OpenAI | Training crawl | Honoured | | OAI-SearchBot | OpenAI | ChatGPT search index | Honoured | | ChatGPT-User | OpenAI | Page fetched because a person asked | May not apply | | OAI-AdsBot | OpenAI | Checks pages submitted as ads, visits nothing else | Not stated separately | | ClaudeBot | Anthropic | Training crawl | Honoured | | Claude-SearchBot | Anthropic | Search index | Honoured | | Claude-User | Anthropic | Page fetched because a person asked | Honoured | | PerplexityBot | Perplexity | Search index, links shown in results | Honoured | | Perplexity-User | Perplexity | Page fetched because a person asked | Generally ignored | | Googlebot | Google | The Search index that AI Overviews and AI Mode draw from | Honoured | | Google-Extended | Google | Control token, never crawls, governs Gemini training and grounding | Honoured as a control | | Applebot | Apple | Crawl | Honoured | | Applebot-Extended | Apple | Control token, never fetches anything | Honoured as a control |

Versions documented on the same day: GPTBot/1.4, OAI-SearchBot/1.4, ChatGPT-User/1.0, OAI-AdsBot/1.0, PerplexityBot/1.0 and Perplexity-User/1.0. Anthropic publishes its three tokens without version numbers. Two rows in that table will never appear in a log file, which is the trap the Google and Apple section below is about.

The three jobs, and why the split matters

One company runs several bots, and each bot has a different job. OpenAI documents four: GPTBot/1.4 for model training, OAI-SearchBot/1.4 to surface sites inside ChatGPT search, ChatGPT-User/1.0 for actions a person starts, and OAI-AdsBot/1.0 to vet pages submitted as ads. Anthropic runs ClaudeBot for training, Claude-SearchBot for its search index and Claude-User for a page a user asks Claude to read. Perplexity runs PerplexityBot for its index and Perplexity-User for a live fetch during a conversation.

Here is the part that costs people visibility. OpenAI states that each of its robots.txt settings is independent of the others, so a rule aimed at one token does nothing to the rest. A block list written by pattern, every token with the letters AI in it, therefore disallows the search crawler alongside the training crawler. The training opt-out is usually the intention. The search block is collateral, and at the vendors that document a consequence, it is the one that removes the brand from answers. Cloudflare's 2025 year in review reports that AI crawlers were the most frequently fully disallowed user agents in the robots.txt files it observed, so blanket blocks are not a rare mistake.

OpenAI's documentation states the consequence directly: a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though OpenAI adds that it can still appear as a navigational link. Perplexity says the same thing in reverse, asking sites to allow PerplexityBot if they want to be surfaced. Those two are the documented cases. Anthropic's crawler page sets out the same three-way split and says disabling Claude-SearchBot may reduce visibility in search results, which is softer wording, so do not read the OpenAI sentence as a rule the whole category has signed up to. Read your robots.txt with that distinction in hand before you read any visibility report.

Google is a special case, and the special case catches people

Google does not run a separate crawler for AI Overviews or AI Mode. Those features are built on the Search index, which means Googlebot is the crawler, and robots.txt rules for Googlebot are the control. Google-Extended is a different lever. It governs whether what Google already crawled may be used for training Gemini models and for grounding in Gemini Apps and on Vertex AI. It is not an AI Overviews switch, and Google states it does not affect inclusion in Search. It also has no HTTP user agent of its own: Google documents that the crawling is done with existing Google user agent strings and the token works only as a control.

So the common instinct, disallow Google-Extended and stay out of AI answers, does not work. What does change AI Overviews behaviour is the preview family: nosnippet, max-snippet and data-nosnippet, which limit what Google may show. They also limit ordinary snippets. You are trading one surface for the other, knowingly.

Apple works the same way at a different scale. Applebot crawls. Applebot-Extended never fetches anything at all, by Apple's own description; it is a usage signal applied to data Applebot has already collected. A rule written for a bot that does not crawl behaves nothing like a rule written for one that does, and no log line will ever tell you it fired.

The second list: fetchers that may ignore your robots.txt

User-triggered fetchers are the category that surprises engineers. When a person pastes a URL into an assistant, the assistant fetches it on that person's behalf, and the vendors do not agree on what robots.txt means in that moment. OpenAI warns that the file's rules might not govern ChatGPT-User, since a person asked for the page. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic takes the other position and says its user-facing fetcher honours the file.

Treat that disagreement as a design fact rather than a grievance. If a page must not be read by an assistant, robots.txt is the wrong instrument, because at least two major vendors have said in writing that they may walk past it. Authentication, a paywall or a server rule keyed to verified IP ranges is the instrument. OpenAI, Perplexity, Anthropic and Apple each publish their IP ranges as JSON, and Apple also documents reverse DNS under applebot.apple.com. That is how you verify a bot is genuine, because the user agent header is self-reported and anyone can set it.

How big is this traffic, really

Two published Cloudflare datasets put the scale in proportion. Both measure traffic to Cloudflare's own customer base, not the web, so read them as a large sample taken at one network.

Cloudflare's 2025 year in review, published December 2025, found that AI bots other than Googlebot averaged 4.2 percent of HTML requests across the year, dropping to 2.4 percent in early April and reaching 6.4 percent in late June, while Googlebot alone accounted for 4.5 percent. Googlebot is broken out there because it is dual purpose, crawling for search indexing and for AI training. On that measure the rest of the AI crawler category was about the size of one long-standing search crawler.

The mix changed fast after that. Cloudflare's 2026 analysis of agent traffic reports that crawler requests declaring a training purpose were 22 percent in spring 2025 and 52 percent by June 2026, and that daily requests from AI agents on its network grew by more than 1,700 percent year on year. Those shares come from the purpose each crawler states, which is not something the measurement verifies independently.

The year in review carries the other figure. It reports a crawl-to-refer ratio, pages fetched per visitor sent back, a metric Cloudflare launched on Radar on 1 July 2025 and reports across the whole of 2025. Google started 2025 just over 3:1 and reached 30:1 in April before falling back. The highest ratio recorded for any single AI platform reached as much as 500,000:1, which Cloudflare attributes partly to sparse referral traffic rather than to crawl volume alone, and it calls the metric volatile day to day.

That spread is the honest case for reading your logs rather than a dashboard. A bot can be heavy on your origin and send nothing back, or light and send real traffic. Computing the ratio per bot separates the two cases, and the numbers are already sitting in your logs.

A check you can run this afternoon

Pull a week of access logs and group by user agent, then sort into the three buckets above. Four questions usually settle it.

Which named tokens appeared at all, and which never did. Treat a missing token as something to chase rather than a conclusion: it may mean the crawler never came, or that a CDN answered from cache without passing the request to origin, or that your retention window is shorter than the crawl interval. Rule those out before you call it a finding.

Which tokens appear in robots.txt, matched against the current names. The community list at ai-robots-txt/ai.robots.txt held 181 distinct user agent entries when we counted it on 8 October 2026, counted as the User-agent: lines in the repository's generated robots.txt, which matched the number of keys in its robots.json on the same day. That is one community list on one date, not a census of the crawlers that exist, and the count on its own says nothing about how many of those names are new. The narrow use is the one that matters: a hand-written block list cannot contain a name added after it was written, so diff yours against the current file rather than judging it by its age.

Whether the requesting IP is in the vendor's published JSON. The user agent string is self-reported, so a spoofed GPTBot will make you believe you are being crawled when you are not. We have seen forged tokens in client logs and have never counted them, so we cannot tell you how common they are.

What the crawler received. A 200 with an empty shell is worse than a 403, because nothing in your monitoring will flag it. Can AI crawlers actually read your site covers that failure mode, which comes up often in the audits we run.

Honest limits

We have no visibility into the retrieval stack of the engines we test, so the mapping from crawler to citation is inferred from logs and from repeated prompt runs, not observed directly. Vendor documentation changes without announcement, and tokens get added quietly; the OpenAI ads bot appeared that way. The traffic figures above are Cloudflare's measurements of Cloudflare's own customer base, large but still one network, and the training and search shares in them are the purposes crawlers declare rather than purposes anyone verified. The documented consequence of blocking a search crawler comes from OpenAI and Perplexity; we have not found equivalent wording from the other engines, and we have not supplied it on their behalf. Our own numbers come from a few dozen sites we have audited, a sample and not a census, which is enough to spot a pattern and not enough to publish a rate.

One more limit worth stating plainly. Allowing every search crawler does not get you cited. It gets you eligible. If nothing on the site answers a question a buyer asks, access work buys you nothing, which is part of when answer engine optimisation is not worth doing yet.

FAQ

What does the GPTBot user agent actually look like? A compatible token with a version inside an otherwise ordinary header, GPTBot/1.4 on 15 October 2026, pointing at OpenAI's GPTBot page. PerplexityBot/1.0 and Perplexity-User/1.0 follow the same shape, and Anthropic's three tokens carry no version at all. Write your robots.txt rules and your log greps against the bare token, because the version moves and the rest of the header is indistinguishable from a browser.

Should I block GPTBot? It depends on what you sell. OpenAI documents its robots.txt settings as independent, so blocking GPTBot keeps your pages out of its training crawl and leaves ChatGPT search access untouched, as long as OAI-SearchBot stays allowed. For a product site the usual answer is to allow both. The case for the training crawler is that a model's unprompted knowledge of your brand has to come from somewhere, but that is an argument, not a measurement, and we do not have a number for it.

Does a separate file help the crawlers find my content? Not yet, on the evidence we have. See does llms.txt do anything yet for what the adoption data actually shows.

How do I tell a real crawler from a forged one? Check the source IP against the vendor's published JSON ranges. OpenAI, Perplexity, Anthropic and Apple all publish them for exactly this purpose, Apple also documents reverse DNS, and the user agent string on its own proves nothing.

Do crawler logs tell me what the answer said about me? No. Access is one layer and the wording of the answer is another, including whether the sentence next to your name is flattering. Sentiment in AI answers deals with that half.

Where does this fit in an audit? Crawler access is the first section, because it gates everything after it. The sample audit report shows how the evidence is presented.

Sources

The OpenAI, Anthropic, Perplexity, Google and Apple crawler pages were opened again and re-checked token by token on 15 October 2026, which is the date on the table above.

Related reading

Can AI crawlers actually read your siteDoes llms.txt do anything yetWhen answer engine optimisation is not worth doing yetSentiment in AI answers: worth tracking or notGeography, CDNs and who gets served your pageYour WAF is blocking the bots you want