Blog · Access and retrieval

Reading your server logs for AI bot traffic

Published 8 October 2026 · Updated 15 October 2026 · 10 min read

Reading your server logs for AI bot traffic
Photograph: Jordan Harrison / unsplash

The short answer: Your origin access log is the best record you have of what AI systems fetched from you, because these clients are documented as crawlers and fetchers rather than browser sessions, so they leave no row in your analytics. Filter the log by user agent, verify each claimed bot against the operator's published IP list where one exists, then split the requests into three buckets: training crawls, search indexing, and live fetches made because a person asked a question. Read the status codes per bucket. That is where the loss is.

Most teams discover their AI visibility problem in a dashboard and then argue about it for a month. The log would have settled it in an afternoon.

Why analytics cannot see this and the log can

A tag manager fires when a browser runs your JavaScript. The clients in this post are described by their own operators as crawlers and fetchers, not browser sessions, and in the logs we have examined none of them produced a matching analytics hit. We cannot speak for every client that will ever fetch you. So a site can be fetched thousands of times a week by systems that write answers about it and show a flat line in the analytics panel.

The access log has a smaller blind spot. Every request that reached your origin leaves a row, and a standard combined log format records timestamp, path, method, status, bytes, user agent and source IP. A client cannot opt out of your logging, though your log format can omit fields and your retention can discard the row. If your CDN terminates most traffic, pull the edge log instead, because requests that the edge blocked or served from cache never reach the origin at all, and a block is exactly the thing you are hunting for. That is the same trap covered in Can AI crawlers actually read your site.

Step one: pull the rows, and do not trust the string

Start crude. One grep over a week of logs gets you most of the way:

`` grep -Ei 'GPTBot|OAI-SearchBot|OAI-AdsBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Googlebot|Bingbot|Applebot|meta-externalagent|Amazonbot|CCBot' access.log \ | awk '{print $9}' | sort | uniq -c | sort -rn ``

Two tokens are deliberately missing from that list, and both are tokens people add by instinct. Google-Extended and Applebot-Extended are control tokens that never crawl. Google documents that Google-Extended has no HTTP user agent string of its own and that the crawling is done with existing Google user agents; Apple says plainly that Applebot-Extended does not crawl webpages and only governs how Applebot's data is used. Grep for either and you get nothing, and that nothing is not a finding. The Google rows worth reading are Googlebot, because AI Overviews and AI Mode draw on the Search index rather than on a crawler of their own. We checked both of those pages on 15 October

  1. The full token list, operator by operator, is in

    2. the AI crawlers and user agents worth knowing.

A user agent is a string the client chose to send. Anyone can type GPTBot into curl. Google's documentation on verifying its crawlers treats user agent spoofing as the reason to verify by IP range or reverse DNS rather than by string, and the same reasoning applies to the rest. We have not measured how common spoofing is across our clients' sites, so treat it as a risk to close rather than a quantity we can give you. OpenAI, Anthropic and Google each publish their ranges as machine readable files you can fetch on a cron. Not every operator publishes one.

We fetched four of those files once, on 8 October 2026, and counted the entries in each file's prefixes array. This is a single snapshot of four files on one day, not a trend and not a census of the operators, and the counts will have moved since. The date after each count is the creationTime stamp inside the file itself, not our fetch date:

  • openai.com/gptbot.json carried 18 prefixes, stamped 22 September 2026.
  • openai.com/searchbot.json carried 39 prefixes, stamped 2 January 2026.
  • openai.com/chatgpt-user.json carried 234 prefixes, stamped 7 October 2026.
  • claude.com/crawling/bots.json carried 38 prefixes, stamped 7 October 2026.

We added OpenAI's fourth file to the same check a week later, on 15 October 2026: openai.com/adsbot.json carried 2 prefixes, stamped 12 May 2026. Two addresses against the hundreds above is a reminder that the size of a list tells you about the fleet behind it and nothing about how much it matters to you.

Look at the third line again. On that day the fleet behind user-initiated fetches was roughly an order of magnitude larger than the training fleet, and its file carried the previous day's stamp while the training file's stamp was more than two weeks old. One snapshot cannot establish how often either file is rewritten, only that on that day the two were not on the same clock. So an IP allowlist you cached a month ago can quietly start rejecting real requests from the one bucket that corresponds to a human being asking about you right now. Refresh these files on a schedule, not once during a project.

The lists are also not all the same shape, which matters more than it sounds. Anthropic publishes one combined file for its bots with no label on each address, so a verified Anthropic row tells you the request came from Anthropic, not which of its three robots sent it. Anthropic's crawler article also says that blocking IP addresses may not work correctly or persistently as an opt-out, and points site owners at robots.txt instead. Google, separately, documents reverse DNS verification alongside its JSON lists. Where an operator publishes neither convention, you cannot verify at all, and saying so is more honest than guessing. Treat verification as three-state. Verified, unverified but plausible, and spoofed. Do not collapse the middle one into either end.

Step two: split the traffic by purpose, because the buckets mean different things

Training crawls, search indexing and live fetches are three different businesses wearing similar clothes, and on the one network that publishes numbers the mix is lopsided. Cloudflare, measuring traffic across its own network, reported in August 2025 that training accounted for nearly 80 percent of crawling from AI bots, with user action and undeclared purpose together under 5 percent over that period. Its 1 July 2026 report put AI training at 52 percent of crawler requests as of June 2026, up from 22 percent in spring 2025, with mixed-use crawlers that blend search, agent use and training at more than 36 percent of activity. Those two figures are not one trend line: the first counts crawling from AI bots, the second counts crawler requests of every kind. Both describe Cloudflare's network on the dates given, not the web and not your site.

Two readings follow from that, and only one of them is comfortable. If your mix resembles Cloudflare's, the volume you see is mostly not the volume that can cite you. A wall of ClaudeBot or GPTBot hits in your log tells you your content is being ingested, not that any answer will name you. The requests that can produce a citation with a link are the search indexers and the live fetchers, and in Cloudflare's measurement they were the minority of the rows. Check your own split before you act on theirs, then read Should you block AI crawlers before deciding that volume alone justifies a rule.

The live-fetch bucket deserves separate treatment because it is not really bot traffic in the old sense. One row there is one person, mid-question, waiting. That distinction, and what it does to your rate limits, is the whole of Agent traffic is not bot traffic.

Step three: read status codes per bucket, not in aggregate

This is the part that pays, and an aggregate hides it. A site-wide 2 percent error rate looks fine. The same site can be returning 403 to every verified OAI-SearchBot request while serving humans perfectly, because a managed bot rule at the edge fires on a header signature that browsers do not produce.

Group by bucket and status:

`` awk '$9 ~ /^[0-9]+$/ {print $9}' ai-rows.log | sort | uniq -c ``

Then read it against these patterns:

403 or 429 concentrated in one bucket. A firewall rule or rate limit is making the decision for you. Check whether the rule is yours or a managed ruleset default you never reviewed.

200 with a suspiciously small byte count. The fetch succeeded and returned a shell. This is the client-side rendering failure, and a byte-size histogram often catches it before a rendering test does, because it uses rows you already have. Compare the bytes served to a bot against the bytes served to a browser on the same path. If the bot's number is a fraction of the human's, your content is arriving in JavaScript the client never ran.

Repeated fetches of the same URL inside minutes by a live fetcher. Often the agent did not find what it came for and went back. A log records requests, not intent, so treat this as a prompt to go and read the page rather than a finding in itself. When we have chased it, the answer was usually buried below navigation, or split across pages.

Heavy crawling of paths you do not monitor. If bots keep requesting a comparison page or a pricing FAQ that is nowhere in your tracked prompt set, your prompt set is probably too narrow. The log tells you which pages the systems went looking for, which is a hint about the questions being asked, not a record of them.

Requests to /.well-known/, your OpenAPI document or an MCP endpoint. Rare in the logs we have read so far, and we have no basis for calling the trend. Those rows mean something is probing you as a programmatic surface rather than a page, which is the subject of Your API and MCP server as a visibility surface.

What the log cannot tell you

Three limits, stated plainly, because log analysis gets oversold.

A fetch is not a citation. Nothing in your log records whether the answer that followed named you, said something true, or linked anywhere. The log proves access. Prompt testing proves the outcome, and you need both.

Retention will bite you. A default stack often discards raw rows after a week or two, and the first real finding turns out to be that last month's evidence no longer exists. Fix retention before you start the analysis.

Caching hides reality. If your CDN serves a bot from cache, the origin log never sees it. Edge logs or nothing.

Honest limits

The prefix counts in this post are one fetch of four files on a single day, 8 October 2026, counted by reading each file's prefixes array. Nothing here establishes how often any operator rewrites its list, because we did not sample the files over time. The traffic mix figures are Cloudflare's measurements of Cloudflare's own network on the dates cited, and two of them use different denominators, so they should not be read as one trend or as a description of your site. Where a client publishes neither a user agent convention nor an IP range, our method cannot see it at all, and the absence of such rows in a log is not evidence that nothing fetched you. The status code patterns above are things worth checking, not diagnoses: a log shows what was requested and what was returned, never why.

The sample audit report shows how we present this when it is part of an audit: verified bot rows, the status split per bucket, and the rule or render that caused each failure, with the evidence attached so a finding can be argued with.

FAQ

Which user agent tokens should I grep for? GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User for OpenAI; ClaudeBot, Claude-SearchBot and Claude-User for Anthropic; PerplexityBot and Perplexity-User for Perplexity; Googlebot, Bingbot and Applebot for the search engines whose indexes feed AI answers. Leave Google-Extended and Applebot-Extended out: both are control tokens that never fetch anything, so they never appear in a log.

Which log do I actually need, origin or CDN? CDN or edge, if you have one. Blocks and cache hits never reach the origin, and those are the two things most likely to be causing the problem.

How do I verify a bot that publishes no IP list? You cannot, fully. Check reverse DNS where the operator supports it, watch for behaviour that does not match a real agent, such as ignoring robots.txt or hammering one path, and label the row unverified rather than guessing.

How much log do I need before the numbers mean anything? As a working rule we ask for thirty days, and seven days is usually enough to find an outright block. That is a rule of thumb from practice, not a measured threshold. Access failures show up immediately. Volume trends need longer.

Does high crawl volume mean I am doing well? Not on its own. On Cloudflare's network most AI crawling is for training, which produces no citation and no link. A smaller number of verified search and live-fetch requests returning 200 with full content is usually worth more than a large training crawl, but only your own log tells you which you have.

Sources

Related reading

Can AI crawlers actually read your siteShould you block AI crawlersAgent traffic is not bot trafficYour API and MCP server as a visibility surfaceSitemaps, feeds and freshness signals that matter to AISlow pages get dropped before they get read