Blog · Access and retrieval

Geography, CDNs and who gets served your page

Published 8 October 2026 · Updated 12 October 2026 · 8 min read

Geography, CDNs and who gets served your page
Photograph: CHUTTERSNAP / unsplash

The short answer: The published crawler address ranges we have been able to place all sit in American cloud regions, so the version of your page your CDN serves to a United States cloud address is the version those fetchers read and can quote. Country redirects, regional price pages and edge rules that filter by geography all act on that single visitor. Your Indian page, your German page and your blocked-country rule stay unseen by those fetchers unless each one has its own URL.

Most teams treat the CDN as plumbing. It is closer to an editor, because it decides which of your several pages a machine is allowed to see.

Where the fetch actually comes from

The engines publish their source addresses, and the files are short enough to read in a browser. We pulled each one and re-checked it on 12 October 2026. OpenAI lists 18 prefixes for GPTBot in a file dated 22 September 2026, 39 for its search crawler in a file dated 2 January 2026, and 234 for ChatGPT-User, the fetcher that runs while somebody waits for an answer, in a file dated 7 October 2026. Google publishes four range files covering Googlebot, special-case crawlers and its user-triggered fetchers, 272 to 1,068 prefixes each, regenerated at least daily and stamped 12 October 2026 when we fetched them. Perplexity publishes twelve prefixes in total across its two bots.

Those twelve are the only ones we could pin to a place. We matched each Perplexity prefix against Amazon's own ip-ranges.json, taken the same morning with a sync date of 12 October 2026, and all twelve fall inside ranges Amazon labels us-east-1, the Northern Virginia region. None of the twelve falls in a range Amazon labels Europe, India or Singapore. That is one published file checked against another, on one day, and it covers Perplexity alone: OpenAI and Google do not label their ranges by region, so we cannot say from their files where a fetch of yours originated. Within that limit, the rule your edge applies to a request arriving from Virginia is the rule that decides whether Perplexity reads you.

Run the same check yourself. Download the vendor file, download Amazon's or Microsoft's range file, and match the prefixes. It takes ten minutes and it turns an argument about geography into a table.

The redirect that eats your regional pages

Here is the common failure, and it is almost always invisible in analytics.

A site detects visitor country at the edge and redirects. A request from India lands on /in/pricing, a request from Germany on /de/pricing, and everyone else on /pricing. A fetch from one of those Virginia addresses takes the American branch and leaves one document indexed. Your rupee pricing, your GST line, your local support hours: none of it reaches that fetcher, because the redirect never hands it the Indian URL.

Google's multi-regional documentation argues against IP-based redirection for exactly this reason, and recommends separate crawlable URLs per locale with the regional versions linked to each other. That is Google's guidance, and we have not found an equivalent published statement from the other engines, so treat it as the one documented position rather than an industry rule. It predates assistants and it matters more now, because an answer engine cuts your page into passages and keeps a few. A passage that was never fetched cannot be kept. The mechanics of that selection are covered in what an agent-readable page looks like.

The fix is dull. Give each locale a stable URL, link them, and offer a banner that suggests switching rather than a 302 that forces it.

Geo-blocking, cloud ASN rules and the silent 403

The second failure is a block nobody remembers writing. A fraud team restricts traffic to a list of countries. A platform team blocks hosting and cloud ASNs to cut scraping. A legal review drops all non-EU requests after a misread of a privacy rule. Each of those is a reasonable decision in its own frame, and each one can land on the addresses the assistants publish: the twelve Perplexity prefixes we checked sit inside Amazon's address space, so a rule written against cloud hosting catches them rather than a consumer ISP.

The resulting 403 is easy to miss. Search Console reports Google's own crawlers and nobody else's, so a block that hits only Perplexity or ChatGPT-User leaves no mark there, it does not move a conversion number, and none of the engine documentation we read describes a notification to the site owner when a fetch is refused. We cannot detect these blocks from outside a site at all. What works is grepping your own edge logs for the published prefixes, or replaying a request from a cloud box with the crawler's user agent and comparing status codes.

Whether to allow those fetches at all is a business question with real arguments on both sides, and we have written the case either way in should you block AI crawlers. What is not defensible is blocking by accident and calling the result a content problem. If you want the broader version of the access audit, start with can AI crawlers actually read your site.

Cache keys, Vary, and serving the wrong country to everyone

A subtler one. Many sites vary their HTML by country at the edge while leaving the cache key unchanged, usually because the geography logic was added long after the caching rules and nobody revisited them.

Two things then go wrong at once. A crawler fetch can be answered from a cache entry that a human in another region filled, so the engine stores a page that was built for somebody else. And the crawler's own response can be written into that cache and handed to the next several thousand human visitors at the same edge node. The second case is worse and harder to spot, because it shows up as scattered complaints rather than a single broken URL.

The check is one request. Ask for a page that you believe is country-aware and read the response headers:

  • Vary should name whatever header carries country, often CloudFront-Viewer-Country,

    • CF-IPCountry or your own.

  • Cache-Control should not mark a geography-dependent page as publicly cacheable with a

    • long age unless the key includes the country.

  • The age and hit headers from your CDN should tell you whether you were served a copy

    • built for a different region.

Our self-check has the request-level version of this, along with nine other checks that need nothing more than curl.

Why the gap gets filled by somebody else

An engine answering a question in Swedish or Hindi still has to retrieve something. If your local page is unreachable, unlinked or merged into an American original, the retrieval step does not return empty. It returns the next document it can reach, which is often a forum thread, a marketplace listing or a news piece written by somebody with no commercial interest in being accurate about you.

We have no figure for how often that happens, and we are not going to borrow one. The citation-share counts circulating in this category are vendor estimates we cannot open or reproduce, so they are not cited here. The mechanism is the part we can stand behind: retrieval returns what it can fetch, and community platforms are reachable at crawlable URLs in markets where many brands publish nothing. The practical response is in why Reddit gets cited so much, and what to do about it.

The lesson is narrower than it sounds. You do not need a translated site to win a local answer. You need the local facts to exist at an address a Virginia-based fetcher can reach.

Honest limits

This post establishes one thing: on 12 October 2026, Perplexity's twelve published prefixes mapped to Amazon's us-east-1 ranges. It does not establish where OpenAI's or Google's fetches originate, because those vendors do not label their ranges by region, and it does not establish that a given engine fetched any particular page of yours. Published prefix files are a floor, not a ceiling: they say where a vendor says its crawlers run, not where every fetch comes from, and the maintenance varies, with ChatGPT-User's file regenerated on 7 October 2026 while the PerplexityBot file still carries a February 2025 date and OpenAI's search crawler file a January 2026 one. An allowlist pinned to today's prefixes decays quietly.

We also cannot see inside a CDN from outside it, and we have not measured how often the redirect, geo-block and cache-key failures described above actually occur across sites, so treat them as mechanisms to check for rather than a prevalence claim. Edge behaviour differs by point of presence, by plan tier and by rules written years ago, so the evidence that settles it is your own logs joined against the published ranges.

FAQ

Do AI crawlers ever fetch from outside the United States? We cannot tell from the published files. Google's four range files are large and carry no region label, so they neither confirm nor rule out non-US fetches, and vendors change infrastructure without announcement. What we verified is narrower: Perplexity's twelve prefixes were all in Amazon's Northern Virginia ranges on 12 October 2026. Planning for an American fetch is the safe default; proving the general case needs your own logs.

Should I stop redirecting visitors by country? Redirecting humans is a product decision. The requirement is that every regional version also lives at its own URL, reachable without a redirect, and linked from the others, so a crawler can fetch all of them.

Will a VPN test tell me what the crawler sees? Only partly. A consumer VPN exit is a different ASN from a cloud region, and many edge rules treat those differently. Replaying the request from a cloud instance in the right region, with the documented user agent, is closer to the real thing.

Does serving the same page everywhere solve this? It removes the whole class of problem and costs you local relevance. A better trade is one URL per locale, no forced redirects, and consistent facts across all of them.

Sources

All range files below were fetched and counted on 12 October 2026; the dates given are the creationTime each file carried that day.

Related reading

Can AI crawlers actually read your siteShould you block AI crawlersWhy Reddit gets cited so much, and what to do about itWhat an agent-readable page looks likeYour WAF is blocking the bots you wantJavaScript rendering is where AI visibility quietly dies