Blog · Access and retrieval

Can AI crawlers actually read your site

Published 8 October 2026 · Updated 12 October 2026 · 8 min read

Can AI crawlers actually read your site
Photograph: Henning Kesselhut / unsplash

The short answer: Usually less of it than you think. In the one network-wide test we can open, Vercel's December 2024 study, the crawlers run by OpenAI, Anthropic, Meta, ByteDance and Perplexity fetched the HTML and stopped there without running a line of JavaScript. Three other things eat the rest: dead URLs, bot rules that return a 200 with a challenge page inside, and Google's snippet directives, which allow the crawl and then forbid the quote.

That is the whole answer. The useful part is knowing which of the four is costing you, and that takes about twenty minutes with a terminal.

The fetch is one request, and it is not a browser

A search crawler and an AI crawler look similar in your access log and behave very differently once the bytes arrive. Googlebot queues your page for rendering, runs the scripts, and indexes what the DOM looks like afterwards. Google's own documentation says AI Overviews and AI Mode draw on the same crawling and indexing systems as Search, which is the documented reason Gemini and AI Overviews can reach client-side content that a non-rendering crawler cannot.

The other crawlers, on the only public evidence of this kind we can open, do not render. Vercel's network-wide study, published 17 December 2024, counted 569 million GPTBot fetches and 370 million from Anthropic's crawler in a single month, and reported that the five of the operators it tested, OpenAI, Anthropic, Meta, ByteDance and Perplexity, executed no JavaScript. The same study found that AppleBot does render and that Gemini inherits Googlebot's rendering, so this is a finding about those five, on that network, in that month, and not a property of AI crawlers in general. The detail worth keeping is stranger than the headline: GPTBot downloaded JavaScript files on 11.50 per cent of its requests and Anthropic's on 23.84 per cent. They pull the bundle without running it.

So the test is not "does my page work". It is "what is in the response body before anything executes". One command answers it:

`` curl -sA "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \ example.com/pricing | wc -w ``

If that number is four hundred words and your rendered page is two thousand, the missing sixteen hundred are not in the running. Repeat it without the custom agent to see whether anything is being varied by user agent, which is a separate problem.

Dead URLs are a bigger tax than anyone budgets for

The same study measured where the fetches landed on Vercel's network. Roughly 34.8 per cent of ChatGPT's fetches hit a 404, with another 14.4 per cent on redirects. Anthropic's crawler sat at about 34.2 per cent on 404s. Googlebot, over the same window and the same network, was on 8.22 per cent.

Why those URL lists are so stale is not visible from outside, and we are not going to guess at how the lists get built. What the data shows is narrower and still useful: on that network, in that month, about one AI crawler fetch in three landed on a page that was gone, while Googlebot's rate was four times lower. A sloppy relaunch can therefore burn fetches you were already getting, which is the argument for migrating a site without losing citations rather than redirecting everything to the homepage and calling it done.

Pull your own numbers. Filter last month's access log to the agent strings in the AI crawlers and user agents worth knowing, group by status code, and sort the 404s by hit count. In the access logs we have gone through during audits, the top twenty are usually old blog URLs, an old docs tree, and one product page somebody renamed in a hurry. That is a pattern we keep meeting, not a measured distribution, and your own list is the only one that counts.

A 200 is not proof that content was served

This is the failure mode that survives audits, because the dashboard shows green. Bot management and firewall rules can answer an unfamiliar crawler with an interstitial, a JavaScript challenge or a holding page that asks the visitor to wait, and the status code attached to all of it is 200. Your log says the request succeeded. What arrived was a holding page with none of your content in it. We have hit this on client sites during audits. We have no measurement of how common it is across the web, and the log on its own will not show it to you.

Check the body, not the code. curl -s ... | grep -ci "<h1" and a word count will separate a real page from a challenge in one line. Also check whether anyone on your team has an allow rule written against the user agent string, because that string is free to type. The defensible version of an allow rule matches published address ranges. OpenAI lists them at openai.com/searchbot.json. We fetched that file once on 8 October 2026 and again on 12 October 2026, and on both days it carried 39 IPv4 prefixes, no IPv6 prefixes, and a creation timestamp of 2 January 2026. That is two readings of one file, not a history of it, and the other engines publish their ranges on their own pages and their own schedules, which we have not sampled here. Ranges change. A rule pinned to a file you re-read beats a rule pinned to a string anyone can type.

One more thing from the same checks: OpenAI's crawler documentation at platform.openai.com/docs/bots answered 301 on both days and pointed at developers.openai.com/api/docs/bots. Vendors move their own canonical pages. So do you.

Crawlable and quotable are two different permissions

A page can be fetched perfectly and still never be used. nosnippet and max-snippet restrict how much of the page Google may show, and Google's guidance on AI features says those same preview controls govern what can appear in AI Overviews and AI Mode. A data-nosnippet attribute wrapped around a summary block, added years ago by someone protecting a paywall, removes the exact passage you now want quoted from Google's surfaces. Whether other engines read these directives at all is not something we can see from outside: their documentation does not say, and a crawler that ignores them leaves nothing behind in your logs.

There is a second version of this. Content that sits inside the HTML but is collapsed behind an accordion is fine, because the text is in the source. Content that an accordion *fetches* on click is gone for a crawler that does not render, which on the December 2024 evidence above covers the five operators tested. The difference is invisible in a browser and obvious in view-source.

What the engine keeps is a passage, not a page

Clearing all four gates gets your text into the candidate pool. What happens next is selection, not reading: the document is split and scored, and a few fragments survive. A thousand-word page where the answer sits in paragraph nine can be fully accessible and still contribute nothing, because the fragment that got picked was the preamble. Chunking, and how your page is cut up before it is read covers the mechanics.

This is also the honest context for the files people hope will fix access. A manifest at /llms.txt does not change whether a crawler ran your JavaScript or whether your firewall answered it. Whether it earns its place at all is a separate question, and does llms.txt do anything yet takes it on directly.

Why the stakes are uneven

Not every fetch is a chance at a citation. Cloudflare's analysis of crawler traffic by declared purpose, with its charts covering the first 28 days of July 2025, put close to 80 per cent of AI bot crawling in the training bucket, and the user-action and undeclared buckets together at under 5 per cent. Those are Cloudflare's network figures under Cloudflare's own purpose labels, not yours, and only your logs can tell you how your mix differs. Training crawls do not cite you today. Search crawls can.

The practical consequence: if you have to choose what to fix first, fix what the search-purpose agents touch. Your commercial pages, your documentation, your comparison pages. Those get fetched live, at question time, by an agent working to a short timeout and, on the evidence above, without a rendering engine.

Honest limits

We can tell you what a crawler received. We cannot tell you what it did with it afterwards, because these systems do not publish a parse log, and a successful fetch is not a promise of a citation. Almost everything above rests on two public datasets, Vercel's December 2024 study of its own network and Cloudflare's July 2025 traffic analysis: neither is a census of the web, neither is ours, and crawler behaviour drifts between such studies without announcement. Our own contribution here is two fetches of one OpenAI file, on 8 and 12 October 2026, which shows what that file said on those two days and nothing more. The snippet directives are Google's, documented for Google's surfaces, and we cannot see whether the other engines honour them. Where this post reports an absence, such as a firewall serving nothing, it means we did not find it in a response body, not that the content was never served to anybody.

The self-check walks through ten of these tests in an afternoon, including the curl checks above. Our audits do the same work against about 45 prompts and five engines, with the evidence attached, and the deliverable is the report.

FAQ

Do any AI crawlers run JavaScript? Some do. Gemini and Google AI Overviews rely on Google's existing crawling and indexing systems, which render, and Vercel's December 2024 study found AppleBot rendering as well. In the same measurements, the crawlers run by OpenAI, Anthropic, Meta, ByteDance and Perplexity executed no JavaScript. That is one network over one month, not a standing commitment from any of them.

Is prerendering for bots risky? Serving pre-rendered HTML is fine while the content matches what a person sees. Google's spam policies define cloaking as presenting different content to users and search engines in order to manipulate rankings, so it is the divergence that creates the problem, by Google's published rule at least. We know of no equivalent published policy from the other engines. Server rendering removes the question.

My logs show 200s for GPTBot, so I am fine? Not necessarily. Check the response body size and whether an h1 is present. Challenge pages and empty app shells both return 200.

How often should I re-run these checks? After every deploy that touches rendering, routing or the firewall, and once a quarter otherwise. Crawler address ranges and agent strings change without announcement.

Sources

Related reading

The AI crawlers and user agents worth knowingDoes llms.txt do anything yetChunking: how your page is cut up before it is readMigrating a site without losing citationsGeography, CDNs and who gets served your pageYour WAF is blocking the bots you want