Blog · Access and retrieval

Slow pages get dropped before they get read

Published 17 September 2026 · Updated 12 October 2026 · 9 min read

Slow pages get dropped before they get read
Photograph: Stanislav / unsplash

The short answer: When an assistant fetches your page to write an answer, it is holding a user who is watching a cursor blink. OpenAI and Perplexity both document a separate fetcher for exactly that case, and Perplexity says plainly that the visit happens because a user just asked something. A request made while somebody waits cannot wait long, so the plausible failure mode is not a penalty applied later but a page that simply was not in the pile when the model started writing.

That sentence hides the part worth arguing about. Most teams treat page speed as a ranking input, a small multiplier applied to an otherwise fine page. In retrieval it is closer to a door that is either open or shut at the moment somebody tries it.

There are two fetchers, and they have different amounts of patience

At least two engines document two distinct kinds of request against your site, and the distinction is the one that matters here. We have read the crawler documentation for OpenAI, Perplexity, Google, Microsoft Bing and Anthropic; OpenAI, Perplexity and Anthropic are the three that publish a user-triggered fetcher as its own named agent.

The first is a background crawler. OpenAI documents GPTBot, OAI-SearchBot and OAI-AdsBot; Perplexity documents PerplexityBot; Anthropic documents ClaudeBot and Claude-SearchBot. These build an index. They can retry, they can come back tomorrow, and a slow response costs you crawl throughput rather than an immediate miss.

The second is a user-triggered fetcher, ChatGPT-User, Perplexity-User and Claude-User in those same docs. Perplexity is blunt about what it is for: a user asked something, so the system may visit a page to answer accurately. That request is happening while a human waits. It has no tomorrow. If your origin is thinking, the assistant moves on with what it already has, which is usually a cached copy from a search index or three other sites that answered quickly.

The practical consequence is uncomfortable. You can be perfectly crawlable, fully indexed, and still be absent from live answers, because the only request that mattered was the one with a stopwatch on it. If you have not confirmed the first half of that, can AI crawlers actually read your site is the place to start, since a 403 and a timeout look identical from the outside.

You are not racing a timeout, you are racing the other candidates

Here is the mechanism people usually miss. A retrieval step does not have to fetch one URL. In the access logs we review during audits we see several fetches from the same agent land within the same second, which is consistent with an answer assembled from whichever candidates returned in time. None of the documentation we read describes this, so treat it as our reading of our clients' logs rather than a published design.

If that reading is right, the threshold is relative. Where four candidate pages return their first byte in 200 milliseconds and yours takes 2 seconds, you do not have to exceed any published limit to lose, because the model may have enough material before your response arrives. Whether the request is then cut off or quietly ignored, the outcome on the page is the same: your name is not in the paragraph.

This also explains a pattern that confuses people when they test. A page that gets cited on Monday and skipped on Thursday has often not changed at all. The competition changed, or your cache was cold, or the fetch landed during a deploy. Each answer is composed fresh, which is why a single run is weak evidence either way.

Most of the web is already over the only published bar

We have not found a retrieval timeout published anywhere in the crawler and fetcher documentation we checked in October 2026, which was OpenAI, Perplexity, Google, Microsoft Bing and Anthropic. That is an absence in what we read, not proof that no such number exists. The one threshold we could open belongs to Google's own performance guidance: Time to First Byte is good at 0.8 seconds or less, needs improvement between 0.8 and 1.8 seconds, and is poor above 1.8 seconds, measured at the 75th percentile of real page loads. That document was last updated on 18 November 2025.

Set against that bar, the web is not in good shape. The 2025 Web Almanac performance chapter, drawing on Chrome UX Report data from July 2025, found 55 percent of desktop websites and 44 percent of mobile websites in the good TTFB band. The mobile figure moved by two points in a year. More than half of mobile origins sit above the only published threshold we could find, and that threshold is written around what a human waiting on a browser will tolerate, which is a more generous standard than a fetch made while an answer is being composed.

Treat 0.8 seconds as a ceiling you measure against, not a target you aim for. The gap between 200 milliseconds and 800 is where citations are actually won.

Slow is not one missed fetch, it is a shrinking budget

The background crawler side has its own penalty, and Google states it plainly in the crawl budget documentation: if the site slows down, if latency increases, or if it answers with 5xx errors or rate-limiting signals such as HTTP 429, the crawl capacity limit goes down and Google crawls less. The same document says that if Google can load and render pages faster, it may be able to read more content from the site.

Two things compound from that. Fewer pages get fetched, so your coverage thins. And the pages that do get fetched are older, so the version an engine holds drifts away from the version you shipped. Teams usually notice the second symptom first and blame the content.

There is a hard cutoff to go with it. Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF, with the limit applied to uncompressed data. Once that is reached the fetch stops and only the downloaded part goes forward for indexing. A bloated HTML document with its answer near the bottom can be truncated before the useful passage is ever seen, which is the same failure as being slow, just expressed in bytes. It is also why chunking: how your page is cut up before it is read and raw payload size are the same conversation.

Where the milliseconds actually go

Across the few dozen sites we have audited, measured with the curl check at the end of this post against each site's answer-shaped URLs, the recoverable loss is rarely the application. It sits in the four places below, in the order we meet them most often in that set. A few dozen client sites is a sample we chose, not a census of the web, and the ordering is what we have seen rather than a rate we can publish.

Redirect hops. Each hop is a full round trip before any content exists. A canonical chain of three, which we see regularly after a domain migration, means three round trips completed before a single useful byte is sent. Resolve the chain so the first request lands on the final URL.

Bot challenges. A firewall rule that issues a JavaScript challenge or a managed challenge to an unrecognised agent adds latency, and in our own testing with curl a non-browser client usually never gets past it at all. This is the quiet version of blocking, and the decision behind it deserves to be made on purpose rather than inherited from a default rule set. We walk through the trade in should you block AI crawlers.

Cold cache on the long tail. Your homepage is usually warm at the edge locations that serve your traffic. The comparison page a buyer's question actually maps to may be fetched once a week and miss cache every time. Since comparison pages that get cited are the pages most likely to be pulled into a live answer, their cold TTFB is the number that matters, not your site average.

Client-side rendering. The first byte arrives fast and contains nothing. For a fetcher that reads HTML and does not run your bundle, a quick empty document is worse than a slow full one, because it looks like a success.

The check, in two commands

Pick your twenty most answer-shaped URLs. Not the homepage. Run each cold, from a region away from your origin, with a documented agent string:

`` curl -s -o /dev/null -A "ChatGPT-User" \ -w "%{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{num_redirects} %{size_download}\n" \ example.com/your-page ``

Read time_starttransfer as your TTFB, and compare it against time_total to see whether the delay is the server thinking or the payload being large. Any num_redirects above zero is free latency you can delete. Then grep your access logs for requests from those agents that ended without a completed response. On nginx that is status 499, a client closed request, which means the other side gave up while your server was still working. A cluster of 499s against named AI agents is the most direct evidence we know how to collect that you are being dropped rather than never asked for in the first place. A quiet log is ambiguous: it is equally consistent with nobody fetching you and with a fetch that never reached your origin.

Run the same pages again with the agent string set to a normal browser. If the two sets differ, you have a bot rule, not a performance problem. The self-check has the rest of the ten things worth testing before anyone spends money.

Honest limits

This post establishes two sourced things and infers the rest. The sourced parts are Google's published TTFB thresholds and the Web Almanac's July 2025 CrUX figures, and Google's own statements about crawl capacity and its 2MB fetch limit, all of which are about Google and the open web rather than about an assistant's retrieval step. The latency budget itself is an inference: we have not found a published timeout, we cannot observe one from outside, and a budget that exists almost certainly varies by surface, by load and by whether a cached copy is already held. Our own evidence is a few dozen audited client sites, a self-selected sample rather than a census, which is enough to see which four causes recur and not enough to publish a rate or a threshold. We also cannot prove the counterfactual: we can show 499s stopping and citations rising afterwards, not that speed alone caused it.

FAQ

Is TTFB a ranking factor for AI answers? Framing it as a factor is probably the wrong shape. Our reading is that it governs whether your content is present when the answer is composed rather than how well it scores once it is. A page whose bytes arrive after the paragraph is written contributes nothing to that paragraph.

My Core Web Vitals pass. Am I fine? Not necessarily. Those scores are field data from real browsers, warmed by real traffic, mostly on popular URLs. A retrieval fetch is cold, headless and often aimed at a page few humans visit. Measure the pages buyers ask about, individually.

Does a CDN solve this? It helps most where it is least needed and least where it matters. Edge caching is strong for popular URLs and weak for the long tail that answers actually cite. Check your cache hit ratio on those specific pages before assuming coverage.

Should I serve a lighter page to AI fetchers? Serving different content by user agent is risky and easy to get wrong. A better route is making the real page faster and putting the answer high in the document, which helps every reader. If you do vary anything, vary compression and payload, never the substance.

Sources

Every link below was opened and checked against the sentence that cites it on 12 October 2026.

Related reading

Can AI crawlers actually read your siteShould you block AI crawlersChunking: how your page is cut up before it is readComparison pages that get citedThe AI crawlers and user agents worth knowingGeography, CDNs and who gets served your page