Blog · Diagnosis

Redirect chains, and the sources they lose

Published 5 October 2026 · Updated 12 October 2026 · 9 min read

Redirect chains, and the sources they lose
Photograph: Luemen Rutkowski / unsplash

The short answer: Google documents following up to ten redirect hops, so chains rarely stop a search crawler. An answer engine fetching live has less room. Every hop is another round trip inside a fetch that a user is waiting on, robots.txt is defined per authority so a cross-host hop lands under a different file (RFC 9309, section 2.3), and a chain that ends on a category page or a homepage returns a document that does not answer the question. The page is not demoted. It is simply absent from the set the model wrote from.

Search gave redirects a long grace period. Four hops, a mixed bag of 302s, an old URL still resolving after two migrations: none of it was fatal, because an indexer has tomorrow. Retrieval does not.

Where the hops actually cost you

Google publishes a number that is worth holding in your head. Its crawlers follow up to ten redirect hops by default, and content returned by the redirecting URLs along the way is discarded in favour of the final target. That is the documented ceiling for a system that is patient by design, built to come back later and finish the job.

We looked for an equivalent number on the OpenAI and Perplexity crawler pages listed in Sources, read on 12 October 2026, and found neither a hop ceiling nor a fetch timeout for their user-triggered fetchers. That is two vendor pages, not a survey of the field, and silence in a doc is not proof that no limit exists. We cannot see these limits, so we do not assert them.

What is not in dispute is the cost of a hop. Each one is a DNS lookup, a TLS handshake if the host changed, and a full round trip before a single byte of your content moves. We have no verified figure for how much latency that adds inside a live assistant fetch, and we are not going to invent one, so treat the direction as the claim and not a threshold.

The relative part is what you can act on. You are not racing a published timeout. You are racing other URLs that returned their content on the first request. Some of what gets blamed on content quality in why AI never mentions your brand may be this instead: the page was fine, and it arrived late or not at all.

The failure that leaves no trace on your server: a cross-host hop

Start with the part that is specified rather than inferred. RFC 9309 puts the rules at scheme:[//authority]/robots.txt, so a different host is a different robots.txt. When a redirect crosses hosts, a crawler that honours robots.txt evaluates the destination host's file, not the one it started with.

That makes the following possible. Your main site allows the documented AI crawlers. A legacy URL on that site redirects to a help centre, a docs subdomain, a CDN hostname or a hosted knowledge base, and that host ships a default robots.txt that disallows those agents. Search traffic is untouched, because Googlebot is allowed on both. The crawler-driven fetch is refused at the second hop with nothing logged on your origin, because the request that was refused never reached your server at all. We have found this pattern in client audits. We have not counted how often it occurs across any defined population, so read it as a failure mode worth checking, not as a prevalence claim.

One correction to a story that is often told too broadly. This blocks the crawlers, not necessarily the user-triggered fetchers. OpenAI's bot documentation says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply, and Perplexity's crawler documentation says Perplexity-User generally ignores robots.txt rules for the same reason. So a disallow on the destination host is a reliable block for GPTBot, OAI-SearchBot and PerplexityBot, and is not a reliable block for the live, user-triggered path on those two engines. We have no documentation either way for the other engines named on this blog.

Both vendors publish a propagation window. OpenAI states that for search results it can take around 24 hours from a robots.txt update for its systems to adjust, and Perplexity states changes may take up to 24 hours to be reflected. We read the inverse as likely rather than documented: a fix on the destination host plausibly takes about a day to show up. Test the day after, not the same afternoon.

A worked example you can repeat. The OpenAI bot documentation moved, and the old address issues a permanent redirect to a different hostname. This is the output we got on 12 October 2026:

`` curl -sIL -o /dev/null -w '%{num_redirects} hops, final %{url_effective}\n' \ platform.openai.com/docs/bots # 1 hops, final developers.openai.com/api/docs/bots ``

One clean hop, cross-host. Harmless here, because the destination welcomes crawlers. Run the same command against your own retired URLs and count both the hops and the hostname changes. Any row where the hostname changes is a row where you now owe a robots.txt check at the other end.

Chains built by accident, not by migration

Nobody plans a four hop chain. They accumulate, and the usual ingredients are boring.

HTTP to HTTPS is one hop. Non-www to www, or the reverse, is another. A missing or extra trailing slash is commonly a third, issued as its own redirect by many server configurations. Locale detection can add a fourth, sometimes based on the requesting IP, which is how a crawler fetching from a different region lands somewhere a human never would. Stack those and a URL printed in an old press release takes four requests to resolve, before any of your own migration redirects are counted.

Two further ingredients turn a slow chain into a dead one. A JavaScript redirect is invisible to a fetcher that does not execute scripts, and Google's own guidance says to use JavaScript redirects only where server-side and meta refresh redirects are impossible, warning that rendering may fail and that Google might then never see the redirect. That is Google's documented behaviour for its own rendering pipeline. For an assistant fetching raw HTML we are reasoning from the mechanism rather than from a vendor statement: a client-side redirect leaves a near-empty document, and a pipeline that does not render will keep nothing from it.

The other is the redirect to nowhere useful. Mapping a hundred retired articles to the blog index is tidy for a human and loses the passage. The fetcher retrieves a list of links, and there is nothing on that page that answers the prompt. Our working model, set out in how each AI engine picks the sources it cites, is that citation is decided at the passage level rather than the domain level. On that model, a redirect that preserves the domain and loses the answer preserves nothing that counts.

Why old URLs keep getting fetched

A reasonable objection: surely the engines have the new URL by now. Sometimes they do not, and the reason is structural.

An engine reaches your pages partly through third-party pages that link to them. An old review round-up, a forum answer, a partner's integration page, a news article: each links to the URL that existed when it was written, and most will never be updated. Those are the corroborating sources we discuss in how Perplexity picks and ranks its sources. We cannot see what share of a given engine's retrieval arrives this way. We found no vendor documentation reporting it, and it does not appear in our logs as a distinguishable category. What we can say is the mechanism: every one of those links enters your redirect chain at hop zero and inherits every hop after it.

That is the real argument for keeping redirects alive for years rather than through one migration cycle. Not link equity. The fact that the inbound graph pointing at your old URLs is frozen, and the engine walks it fresh on every answer.

The checks, in the order we run them

Pull your redirect map, or generate one from the last two years of 301s in your server config, and resolve every entry with a tool that reports hops rather than the final status alone. Four columns are enough: source, hop count, hostname change, final status code.

Sort by hop count and look at anything above one. Then, separately, filter for hostname changes and fetch each destination host's robots.txt, checking it against the documented user agent strings for GPTBot, OAI-SearchBot and PerplexityBot. Those are the agents both vendors document as honouring robots.txt, which is why they are the ones a disallow actually controls. A disallow you did not know about is a high-value find here, and it takes minutes.

Last, check where the chains land. Any destination that is a homepage, a category index or a generic "this content has moved" page is a lost source whether or not the status codes are clean. Collapse the chains to one hop, point each old URL at the page that answers the same question, and re-run the prompts a week later. The self-check covers the ten things worth testing before any of this, and a self-audit checklist for AI visibility puts the redirect pass in sequence with the rest.

Honest limits

The ten hop figure is Google's, documented for Google's crawlers, and we did not find a hop limit or a fetch timeout for user-triggered fetches on the OpenAI or Perplexity pages we read on 12 October 2026. We checked two vendors, so this post establishes nothing about limits on the other engines, and a number absent from a doc may still exist in the system. The cross-host robots.txt failure is a mechanism we have seen in client audits and have not measured across any defined population, so nothing here supports a claim about how common it is. We also cannot separate a slow chain from a blocked destination using answer output alone, because both look like absence, and only logs on the destination host tell them apart. The latency argument in this post carries no measured figure at all and should be read as a direction, not a threshold. The sample audit report shows how that evidence gets attached to a finding.

FAQ

Does a 301 pass the same signals to an AI engine as it does to search? Where the engine draws on a search index, the consolidation that index performs carries over. Google documents treating a 301 as a strong signal that the target should be processed. For a live fetch we have no vendor documentation describing a consolidation step, so we treat the redirect as a cost to pay rather than a transfer to count on.

How many hops is too many? One is the target. Google documents its crawlers following up to ten before giving up, and that is the patient case. No comparable figure is published for the live fetchers we checked, so for live fetches treat every hop past the first as latency you chose to spend.

Are 302s a problem for AI visibility? Google documents treating a 302 as a weak signal for the target and a 301 as a strong one, so a permanent move left on a temporary code can keep two candidate URLs alive. Our reading, not Google's statement, is that two URLs for one answer splits the corroboration an engine sees.

Should old redirects ever be removed? Only when nothing external still points at the old URL, which is rarer than people assume. Third-party pages that cite you are not maintained, and those citations are what the engines walk. Cheap to keep, expensive to lose.

Sources

All five URLs below were opened and checked on 12 October 2026.

Related reading

Why AI never mentions your brandA self-audit checklist for AI visibilityHow each AI engine picks the sources it citesHow Perplexity picks and ranks its sourcesMigrating a site without losing citationsThe small-site authority problem