Blog · Access and retrieval

Should you block AI crawlers

Published 28 September 2026 · Updated 12 October 2026 · 9 min read

Should you block AI crawlers
Photograph: Flower Lover / unsplash

The short answer: Usually no, not as a blanket rule. Blocking is three decisions, not one: whether a model may train on you, whether a search index may store you, and whether an assistant may fetch a page a reader just asked about. Refusing training is defensible and costs you little. Refusing retrieval removes you from the answers while everyone you compete with stays in them.

Most companies hold this as a single yes or no, usually settled by whoever owns the CDN dashboard. That framing is the problem. One robots.txt file controls several unrelated outcomes, and the directive that protects your archive is not the one that erases you from ChatGPT.

Three jobs, three bots, three decisions

The three operators whose crawler documentation we read on 12 October 2026, OpenAI, Anthropic and Google, each publish more than one user agent with a declared purpose. We have not surveyed every operator, so read this as three worked examples rather than an industry rule. OpenAI's documentation separates GPTBot, which gathers content for training foundation models, from OAI-SearchBot, which builds the index behind ChatGPT's search answers, from ChatGPT-User, which fetches a page because somebody in a chat window asked for it. That page is blunt about the consequence: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although the docs note such a site can still turn up as a plain navigational link, and that ChatGPT-User is not what decides whether content appears in search. It carries one caveat worth holding on to, that because ChatGPT-User requests are initiated by a person, robots.txt rules may not apply to them.

Anthropic's support article, dated 7 April 2026, publishes the same shape: ClaudeBot for training, Claude-SearchBot for search quality, Claude-User for a fetch on behalf of someone mid-conversation. Google splits it differently, and that split is the one that catches people out.

So the honest version of the question is which of the three jobs you object to, and whether your robots.txt currently says that or something cruder.

What a retrieval block actually costs

Block the training bot and you lose some unquantified future influence on a model's internal memory. Block the retrieval bot and you lose something you can watch happen this week.

Here is the mechanism as we read it. An engine that cannot fetch you can still answer about you, because it has other material to work from: an old review, a comparison page a competitor wrote, a forum thread, a directory entry with your old pricing. In audit runs we have watched answers get written that way, and what we see is that the block removes the correction rather than the reputation. How reliably that holds across engines we cannot say, because we sample prompts rather than survey them.

There is a second cost, and it sits in the highest-intent slice of all. When a reader pastes your URL into a chat and asks what this product does, a live fetch agent is doing the work. On OpenAI's and Anthropic's own descriptions of their user agents, refusing that agent means the assistant answers from memory or not at all. OpenAI also notes that robots.txt may not govern a user-initiated fetch, so the outcome turns on the operator rather than on one line you wrote. Either way the reader was already on your page, and losing them there costs more than a training opt-out does.

Google does not give you the same switch

Google's crawler documentation states that Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal. It governs whether your content trains and grounds Gemini models. Useful, and narrower than it sounds.

What it does not do is take you out of AI Overviews or AI Mode. Google's AI features page says AI is built into Search, that robots.txt directives for Googlebot are the control for how a site is crawled for Search, and that the way to limit what is shown from your pages is nosnippet, data-nosnippet, max-snippet or noindex. Each of those also shrinks or removes your ordinary Search result. On Google's documented behaviour as of 12 October 2026 there is no setting that drops you from the generative answer and leaves the blue link untouched. Decide with that trade on the table, not after.

The block nobody decided to make

Many of the blocks we find in audits were never argued about. They were inherited: a CDN default, a managed rule set, a bot-fight toggle somebody enabled during a scraping incident. What share of blocked sites got there that way we cannot tell you, because the sites we see are a self-selected sample of companies that already suspected a problem.

The platform numbers moved fast, and they are one platform's view of its own network rather than a measurement of the web. In a post dated 1 July 2025 Cloudflare reported that more than a million customers had enabled its one-click AI bot blocking, and that GPTBot was disallowed in 7.8 percent of the robots.txt files it found for the top domains. In a post dated 15 September 2026 it reported 17 percent of sites on its network enabling some mechanism to block AI training against less than 1 percent choosing to block search bots, and shipped a setting that lets a mixed-use crawler keep indexing a site while being refused for training. The same post says that from 15 September 2026 its Block and Block on pages with ads settings also apply to mixed-use crawlers including Applebot, Bingbot and Googlebot, so either setting now affects search as well as training. Platform defaults move under you. Your robots.txt does not send a changelog.

A block can also be invisible in robots.txt entirely. A WAF rule that returns 403 or a managed challenge to a declared bot IP has the same effect as a disallow, with none of the paper trail. In audits we have run we have seen individual sites allow every agent in robots.txt and still serve a challenge page at the edge. That is a handful of cases we can point at, not a measured rate. Can AI crawlers actually read your site goes through that class of failure in detail, including the empty-shell problem on client-rendered pages, which looks identical from the outside and has a completely different fix.

When blocking is the right call

Some sites should block, and the reasoning is not hard.

If your archive is the product, refuse training. A paid research library, a licensed image collection, a subscriber archive: those have a revenue model that a trained model erodes and a citation does not repair. Refuse training, keep retrieval, let the engine describe the paywall and send the reader to it.

If you are negotiating a licence, a block is a bargaining position rather than a policy. If crawl volume is a real cost, rate limit rather than ban, because a 429 is a negotiation and a 403 is an exit. If the pages hold user data that was never meant to be public, the fix is authentication, not a politely worded line in a file that nobody is obliged to read. Enforcement is the word that matters there. The robots exclusion protocol is voluntary by design.

Run the check before you argue about it

Do not debate this from memory. Thirty minutes of evidence settles it.

Fetch your own page as each declared agent and compare what comes back. Something like curl -sI -A "OAI-SearchBot/1.4" yoursite.com/pricing tells you the status code that agent string gets, which in the audits we run is not always 200. Then grep a week of server logs for the agent strings and count requests by status per agent. A column of 403s against one operator is usually a block somebody forgot about. Zero requests from an operator that should be crawling you is a different problem, usually upstream.

Verify before you trust the log line, because user agents are trivially spoofed. OpenAI, Anthropic and Google each publish crawler IP ranges in the documentation listed below, so match the source address against the published list rather than the string. Operators outside those three may not, and we have not checked them all. Read your robots.txt as served, not as it sits in your repository. The self-check has the rest of the list.

One thing not to spend the afternoon on: adding a permissive llms.txt and calling it an access fix. It is a different file solving a different problem, and the evidence on whether engines read it is thinner than the enthusiasm suggests. Does llms.txt do anything yet has what we can and cannot show.

Honest limits

This post does not establish what a training block costs you in model memory, because nobody outside the labs can see inside the weights, and a model that already learned about you in 2024 does not unlearn you when you add a line to robots.txt in 2026. It does not establish how common any of these failures are: our own evidence is prompt-set runs and crawler checks on client sites that arrived already suspecting a problem, which is a biased sample rather than a census, and we have deliberately not attached a percentage to it. It does not establish revenue impact, because most of these surfaces send no useful referrer, so we can see an appearance and not a sale. The third-party figures here are one platform's view of its own network, and the IETF AI preferences drafts that would give train-ai and search separate signals are drafts rather than deployed behaviour. Treat crawler access as a technical finding and not a growth number: what an AEO audit should contain places it in the order of work, and what answer engine optimisation is covers why access comes before anything else.

FAQ

Does blocking GPTBot remove me from ChatGPT? No. GPTBot is the training crawler. Appearing in ChatGPT's search answers is governed by OAI-SearchBot, which is a separate directive. Blocking the first and allowing the second is a supported configuration: OpenAI's documentation states that each setting is independent of the others.

Will blocking AI crawlers hurt my Google rankings? Google's documentation says Google-Extended is not a ranking signal and does not affect inclusion in Search. Blocking Googlebot is a different matter, and it removes you from Search and from AI Overviews together, because they draw on the same index.

If I block everything, will AI stop mentioning my brand? Probably not. Engines can describe a brand from third-party sources, and in our audit runs they do. You remove your own page from the evidence pool and leave everyone else's in it, which in the cases we have seen makes the answer less accurate rather than quieter.

How do I know whether I am blocking anything right now? Request your own pages with each published user agent and check the status code, then confirm the result against the operator's published IP ranges. Check the robots.txt your server returns, not the one in your repository, and check your CDN rules separately.

Sources

All sources below were opened and checked against the sentences that cite them on 12 October 2026.

Related reading

Can AI crawlers actually read your siteDoes llms.txt do anything yetWhat an AEO audit should containWhat is answer engine optimisationSitemaps, feeds and freshness signals that matter to AISlow pages get dropped before they get read