Blog · Access and retrieval

Your WAF is blocking the bots you want

Published 8 October 2026 · Updated 12 October 2026 · 8 min read

Your WAF is blocking the bots you want
Photograph: Michael Chacon / unsplash

The short answer: A site can be absent from AI answers without anyone having decided to block it. A bot preset, a managed challenge or a rate limit hands a 403 to the crawler that feeds ChatGPT, Perplexity or Google's AI surfaces. No human sees that error, because no human made the request. None of those providers document how long they keep retrying after a block, and your analytics look normal throughout. Check your edge logs by IP range, not by user agent string.

A blocked page does not look broken. It looks fine to you, fine to your monitoring, fine to every colleague who loads it in a browser. The only party that knows is a crawler with no way to complain.

The failure is silent by design

Think about what a firewall is for. It decides, in a few milliseconds, whether a request looks like a person or like something you would rather not serve. Requests from a datacentre IP, with no cookie, no referrer, no prior session and a user agent nobody on your team recognises, score badly on those axes. That describes an attack. It also describes the crawlers OpenAI and Perplexity document for themselves: datacentre ranges, no session, a declared agent string.

So the rule fires, and the crawler gets one of three things: a 403 block page, a challenge it cannot solve because it does not run JavaScript, or a 429 after the second or third request on a burst of pages. What the engine does next we cannot see, because none of these providers publish their retry behaviour after a block. What you can see at your own edge is that the request never reached the page.

Usually nothing reaches your error budget. Synthetic monitoring passes if your monitor runs from an allowlisted IP with a browser user agent, and Core Web Vitals look healthy because the field data comes from real Chrome users who were never challenged. This is why crawl access is the first thing an audit checks and almost the last thing a marketing team suspects, and why can AI crawlers actually read your site is a question worth answering with a request log rather than a hunch.

The category you blocked is not the category you meant

At least one edge provider has stopped treating AI traffic as one thing. Cloudflare sorts crawlers by behaviour into three buckets: Search, which builds an index to answer questions later; Training, which collects text for model training; and Agent, which fetches a page in real time because a person just asked something. Its documentation describes those three categories explicitly. This section is about that one provider's controls. We have not audited how other CDNs group AI traffic, so do not assume the same three buckets behind a different edge.

The interesting part is what happens to crawlers that do more than one job. On 15 September 2026, Cloudflare extended its block options to mixed-use crawlers, and named Googlebot, Bingbot and Applebot among them. A site owner who switched on "block AI training crawlers" at some point in 2025, reasonably, to keep their writing out of a training set, could find that the same switch now reaches a crawler they very much want. Cloudflare's own answer to that problem is a separate "disallow AI training" signal, so a mixed-use crawler can keep indexing while honouring the training opt-out. The point stands: the toggle you flipped and the traffic you lost are two different things, and the gap between them is only visible in logs.

Scale is worth knowing here. In that same 15 September 2026 post, Cloudflare reports that fewer than 1% of sites on its network choose to block Search bots, while 17% enable some mechanism to block training. Those are figures for one network on one date, not for the web, and they count settings rather than outcomes: they say nothing about how many site owners understand what their setting now reaches. If the decision itself is live at your company, should you block AI crawlers sets out what each choice costs.

Why your IP allowlist went stale without telling you

Here is the mechanism that catches careful teams. Someone sensible decides that user agent strings are spoofable, which they are, and writes a WAF rule that permits a published list of crawler IP ranges instead. Correct instinct. The trouble is that those lists move.

OpenAI publishes one JSON file per bot. We fetched all three on 8 October 2026, counted the IPv4 prefixes in each and read the creationTime field the file carries. searchbot.json carried 39 prefixes, stamped 2 January 2026. gptbot.json carried 18, stamped 22 September 2026. chatgpt-user.json carried 234, stamped the previous evening, 7 October 2026. That is one fetch of three files on one day, not a time series: it shows the spread between the files at that moment, not how often any of them changes.

ChatGPT-User is the one that fires when somebody in ChatGPT asks a question whose answer needs your live page, and on the day we looked it was by far the widest range. An allowlist copied by hand goes stale the moment the file behind it changes, and it fails closed. A per-IP rate limit tuned against 39 prefixes is also worth re-reading against 234. We have not measured how requests distribute across that range, so treat it as a reason to check your threshold rather than as a prediction about traffic shape.

Perplexity publishes its own ranges for PerplexityBot and Perplexity-User. Both OpenAI and Perplexity note that their user-triggered fetchers behave differently from their indexing crawlers, and that robots.txt rules may not govern a fetch a person explicitly asked for. A WAF rule, unlike robots.txt, governs all of them, which is exactly why a firewall misconfiguration is more damaging than a robots.txt mistake.

A check you can run before lunch

Four steps, no tooling, roughly ten minutes.

Request your own page the way a crawler does, from outside your network, with no cookies and the real user agent string. OpenAI publishes it in full on its bots page: a browser prefix, then compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot with no brackets around that part. Put it in a shell variable and run curl -sI -A "$UA" yoursite.com/your-best-page, then read the status line. A 200 is good news. A 403, a 503, or a 200 carrying a challenge interstitial instead of your content is the answer you came for.

Repeat with the ChatGPT-User, PerplexityBot and ClaudeBot strings. Different rules catch different agents, and a site can pass one and fail another.

Then open your edge logs and filter on IP, matching against the published range files rather than on the user agent text. Anyone can send a crawler's user agent string, so a hit on the string proves nothing; for the operators that publish ranges, OpenAI and Perplexity among them, verified traffic sits inside those CIDRs. Google documents reverse DNS plus its own published ranges as the way to verify its fetchers, which is the same discipline applied to a provider that publishes both.

Last, look at what the blocked requests were asking for. A firewall that blocks your pricing page but serves your blog is a specific, fixable rule, not a philosophy.

The self-check walks through ten of these, including the rendering questions that come after access. Access first, though. Everything else in answer engine optimisation assumes the document arrived.

What this does not fix

Getting the bots in is necessary and nowhere near sufficient. A crawler that reaches an empty client-rendered shell has been served, politely, nothing. A page that answers the question in paragraph nine still loses to one that answers it in paragraph one.

We also cannot tell you how long recovery takes. OpenAI's documentation notes that it can take about 24 hours from a robots.txt update for its search systems to adjust, and firewall changes take effect immediately at your edge, but when an engine chooses to re-crawl a site it previously found hostile is not something these providers document. On the few sites where we have watched a block come off, crawl volume returned before citations did. We did not log those cases to a method anyone could reproduce, so there is no number attached to that ordering and you should not plan a launch around it.

And none of this is a naming debate. Whether you call the work AEO, GEO or SEO, and the three terms overlap more than the category admits, a 403 is a 403.

Honest limits

This post establishes how a firewall can block a crawler and how to test whether yours does. It does not establish how common that is: the only population figures here are Cloudflare's, for sites on Cloudflare, and they count settings rather than blocked crawls. The three OpenAI prefix counts come from a single fetch on 8 October 2026 and describe those files on that day only, so they are evidence that published ranges move, not a measure of how fast. We cannot see what any engine does after it receives a 403, because none of these providers publish their retry or re-crawl behaviour, which is why the recovery paragraph above carries no timeline. The curl check tells you what your edge returns to a given user agent from outside your network, which is useful and is not the same as knowing whether a real crawler reached the page.

FAQ

Does robots.txt matter if my WAF blocks the request anyway? No. robots.txt is a request for cooperation, read after the connection succeeds. A firewall rule runs first and ends the conversation before the file is ever fetched. Fix the edge, then fix the file.

Will allowing AI crawlers put my content into a training set? Not automatically. OpenAI, Google, Microsoft and Apple each document a separate crawler or a separate opt-out signal for training, and Cloudflare's bot categories reflect that split, so you can permit the search and agent fetchers and refuse the training ones. That holds for the providers who document it. A crawler that does not publish a split is a different question.

My logs show plenty of GPTBot hits, so I am fine? Check the source IPs before you relax. Anyone can set a user agent string, and crawler names are popular cover for scrapers. For the operators that publish ranges, verified traffic sits inside them; the rest is unverified and may be inflating the rate limit that blocks the real thing.

How often should I re-run the check? After every change to firewall rules, bot management settings or CDN provider, and otherwise quarterly. Published IP ranges change without announcement, and one of the three OpenAI files we read had been updated the previous day.

Sources

Related reading

Can AI crawlers actually read your siteShould you block AI crawlersWhat is answer engine optimisationAEO vs GEO vs SEO: what the three terms actually meanJavaScript rendering is where AI visibility quietly diesDoes llms.txt do anything yet