Blog · Agents and commerce

Agent traffic is not bot traffic

Published 8 October 2026 · Updated 12 October 2026 · 8 min read

Agent traffic is not bot traffic
Photograph: MealPro / unsplash

The short answer: Agent traffic is a request made because a person asked for something right now, carried out by a program on their behalf. Bot traffic is a crawler collecting pages on its own schedule. Where an operator publishes address lists, the two arrive over different ranges, and the operators that document the difference say different robots.txt rules apply. Treating them as one bucket means you either block a customer or count one as abuse.

Most bot defences were built when the only question was whether a request was human. That question has stopped being useful.

Three kinds of non-human request, not one

Sort the traffic by what caused it, not by whether it has a browser window.

The first kind is the training crawler. It collects pages to build a model, on its own schedule, with no user waiting. The second is the retrieval crawler, which keeps a search index fresh for an assistant to query later. The third is the agent: a fetch issued the moment somebody typed a question or clicked buy, with a human watching the spinner.

Two engines document this split rather than leaving it to inference. OpenAI lists its crawlers separately from ChatGPT-User, which it describes as user-initiated and not used for automatic crawling. Its documentation says a robots.txt directive may not govern such a fetch, because the person asked for it. Google's page on user-triggered fetchers says those fetchers generally ignore robots.txt for the same reason, in the context of its own products such as Site Verifier and Read Aloud. Both pages were read on 8 October 2026. We have not found equivalent published wording from the other engines, so read this as the documented position of those two operators rather than as a rule of the web.

The rules genuinely differ, and that is deliberate

A crawler that ignored your robots.txt would be misbehaving. A fetcher acting on a user's instruction that obeyed it would be refusing a request its user is entitled to make, in the same way a browser is entitled to fetch the page you typed into it.

That has a practical consequence people miss. Disallowing a training crawler does not remove you from the surface where buyers actually are. It removes you from the corpus and leaves the live fetch untouched, or it does nothing at all to the live fetch while your rate limiter quietly refuses it anyway. The decision about training and the decision about retrieval are separate decisions, and the robots file only cleanly expresses the first.

What the shapes look like in a log

Run the split yourself before you argue about it. Pull a week of access logs and group by user agent, then look at the shape of each group rather than the volume.

The shapes below are what we have seen in the access logs clients attach to audits, read by hand rather than counted. We have not quantified them across a sample we could publish, so treat them as patterns to look for in your own logs, not as thresholds.

A training crawler tends to read broadly and slowly. It walks the sitemap, takes archive pages nobody has linked to in years, and spaces its requests out. A retrieval crawler is narrower and more repetitive, revisiting the pages that already earn citations. An agent session has looked like neither in the logs we have read. It is a short burst over a few URLs, arriving at the specific page that answers one question, and it often skips images and stylesheets because nothing needs rendering for a reader who is not looking.

Two tells are worth grepping for. In those logs, agent sessions frequently fetched a product or pricing page directly with no preceding category page, which fits a route chosen by a model that already had a URL. They also produced a higher share of non-GET methods than the crawler groups in the same files, since the point of the visit may be to submit something. The method column is the cheapest signal you own. Reading your server logs for AI bot traffic covers the parsing side; this post is about what the groups mean once you have them.

Verify the claim, do not trust the string

A user agent header is free text. Google says so in its own documentation, and the warning applies to every name in your logs. Anyone can send ChatGPT-User from a laptop, and a scraper that does so inherits the politeness you extend to real agents. We have seen requests carrying that string from addresses outside OpenAI's published prefixes, which is the cheap way to catch it.

Two checks make the claim falsifiable.

The first is the published address list. OpenAI maintains a JSON file of IP prefixes per agent. We fetched the three files once on 8 October 2026 and counted the ipv4Prefix and ipv6Prefix entries in each: the ChatGPT-User list carried 234 prefixes and a creationTime of 7 October 2026, the day before, while the GPTBot list carried 18 and the OAI-SearchBot list 39. That is a single snapshot of three files, not a series, so the counts will have moved by the time you read this. Fetch them yourself and count. The gap is informative on its own. User-driven fetching runs on far more infrastructure, and it churns, which is why an address list copied into a firewall rule six months ago is now wrong.

The second check is cryptographic, and it is still being standardised. Web Bot Auth has agents sign their requests using RFC 9421 HTTP Message Signatures, carrying a Signature-Agent header that points at a key directory plus Signature-Input and Signature headers for the signature itself. The origin validates against the published key rather than against a name or an address, so this is the one check here that a spoofer cannot pass by typing a different string. Scope it carefully before you rely on it. The Signature-Agent format comes from an IETF draft rather than a finished RFC, the documentation we checked on 8 October 2026 is one CDN's implementation of it, and that implementation lists RFC 9421 components it does not support. Whether your own edge offers it is a question for your edge, not something we can assert for the category.

The failure mode that costs money

Here is a sequence we have seen repeatedly in audits, without having counted how often. A WAF rule is written against AI bots because somebody read a scraping story. It matches on a substring of the user agent. The rule now also matches the fetch that happens when a buyer asks an assistant whether your product ships to Chennai. The assistant gets a 403, says it could not verify the information, and names a competitor whose page answered.

Your client-side analytics will not show this. The blocked request did not reach your tag, the session did not exist, and the lost sale looks like a sale that was never attempted. The only place it is visible is the edge log, in the status code column, which is why the sample audit report keeps blocked agent requests as a finding in their own right rather than folding them into crawl health.

The inverse failure is quieter and just as expensive. The agent gets through, reads a page built for a human skimmer, and cannot extract a price, a stock state or a shipping rule because all three live in a tab that renders after a click. It leaves and reports that the information was not available. What an agent-readable page looks like is the fix for that, and Agentic commerce: what an agent needs from your catalogue is the fix for the catalogue behind it. Where the agent needs to act rather than read, Can an agent book, buy or fill your form is the one to read next.

Honest limits

We cannot tell you what share of your traffic is agent traffic, because it varies by category by an order of magnitude and because the classification is a judgement call at the edges. A deep research run issues dozens of fetches over several minutes for one person, so it looks like a crawler and is not one.

Attribution is worse. The requests we have looked at carried no referrer worth the name, so you can observe the visit and not the outcome. We treat the counts as directional and the status codes as actionable, which is a narrower claim than most dashboards make.

The log shapes in this post come from client logs read by hand during audits, not from a sample we have sized, held constant or published, and a reader is right to challenge them on that. The prefix counts are one fetch of three files on one day. The robots.txt position is documented by OpenAI and Google and we do not claim it for operators that have published nothing, and Web Bot Auth rests on a draft plus one CDN's implementation of it, so it may change under us.

FAQ

Does robots.txt stop an agent? For the two operators that document it, not reliably, and by design. OpenAI's documentation says a robots.txt directive may not govern a fetch the user asked for, and Google says its user-triggered fetchers generally ignore robots.txt. Other operators may behave differently and we have not found comparable wording from them. Use robots.txt to express a position on training and crawling, not as an access control layer.

Should I block AI user agents in my WAF? Block the ones you have evidence are abusive, by behaviour rather than by name, and leave user-driven fetchers alone unless you have decided you do not want that demand. A substring rule on the word bot will catch buyers.

How do I tell a real agent from a spoof? Where the operator publishes a prefix list, check the source address against it, refetching the list rather than trusting a copy. Where your CDN supports Web Bot Auth, prefer signed requests, since a signature validates against a published key rather than a header anyone can type. For operators that publish neither, you are judging by behaviour.

Will agent traffic show up in my analytics? Not in a client-side tool when the fetch does not run your JavaScript, which is the case for plain fetches that skip rendering. We cannot see what we cannot see, which is the point: server or edge logs are the record we trust, and the status code matters more than the hit count.

Sources

All seven links were opened and checked on 8 October 2026 and rechecked on 12 October 2026.

Related reading

Agentic commerce: what an agent needs from your catalogueCan an agent book, buy or fill your formReading your server logs for AI bot trafficWhat an agent-readable page looks likeShopping inside ChatGPT, and what it reads