Blog · Access and retrieval

How to write robots.txt for AI crawlers

Published 21 May 2026 · Updated 15 October 2026 · 10 min read

How to write robots.txt for AI crawlers
Photograph: elaine alex / unsplash

The short answer: Name each AI crawler token in its own group, decide per token whether you are allowing retrieval or allowing training, and keep the file under 500 kibibytes. The trap is group matching. A crawler obeys exactly one group, so the moment you write User-agent: GPTBot, that bot stops reading your User-agent: * rules completely, and every path you thought was blocked is open to it.

That last sentence is the whole article in miniature. Getting the bot names right is the easy half. The hard half is what happens to the rest of the file once a named group appears.

The GPTBot directive, if that is all you came for

Two lines in /robots.txt at the root of the host. To keep your pages out of OpenAI's model training crawl:

`` User-agent: GPTBot Disallow: / ``

To let it crawl everything:

`` User-agent: GPTBot Allow: / ``

The token takes no version number. RFC 9309 matches product tokens case-insensitively and ignores the version, so GPTBot matches the GPTBot/1.4 string the crawler sent when we checked OpenAI's bot documentation on 15 October 2026. The file must come back as a 200 with a text/plain content type, and OpenAI documents a robots.txt change taking about 24 hours to reach its search systems, so do not judge it the same afternoon.

Two things to know before you paste either block. GPTBot is the training crawler and nothing else: ChatGPT's search behaviour runs on a separate token, OAI-SearchBot, and OpenAI documents each robots.txt setting as independent of the others, so a rule aimed at one does nothing to the other. And the moment a named group appears, that crawler stops reading your User-agent: * rules completely. That second point is the next section, and it is why most of these files do not do what their author thinks.

The group-matching trap

Both the published standard and Google's own parser documentation say the same thing in different words. RFC 9309 says the most specific match wins and is measured in octets. Google's robots.txt reference is blunter: "Only one group is valid for a particular crawler," and "User agent specific groups and global groups (*) are not combined."

Picture a normal file. The * group disallows /search, /cart, /*?sort= and a few faceted paths, because those generate near-infinite URLs. Somebody reads a post about AI visibility and appends a friendly block:

`` User-agent: GPTBot Allow: / ``

GPTBot now has its own group. It no longer reads the * group at all. Your faceted search, your cart, your session-parameter URLs are all fair game for it, and the crawler spends its budget on pages you would never want quoted. Your logs will not flag it either: they show you the requests, not the rules that quietly stopped applying. The fix is dull and reliable: whatever you disallow globally, repeat inside every named group. Treat each group as a standalone file.

Which token controls what

The second common error is treating all AI bots as one category. They are not. A single company usually runs separate tokens for separate jobs, and blocking the wrong one costs you visibility without buying you anything.

OpenAI documents GPTBot for crawling content that may train foundation models, and OAI-SearchBot for surfacing sites inside ChatGPT's search features. Four tokens in all, on the documentation we read on 15 October 2026: the other two are ChatGPT-User, for a page fetched because a person asked, and OAI-AdsBot, which checks pages submitted as ads and visits nothing else. Neither of those two is a general crawl, so neither belongs in a blanket rule. Block GPTBot and you have opted out of training while staying eligible for citation. Block OAI-SearchBot and you have removed yourself from the answers. Anthropic splits the same way, with ClaudeBot on the training side and Claude-SearchBot on the retrieval side, plus Claude-User for fetches a person triggered.

Google is a different shape again, and this one catches experienced people. Google-Extended is not a crawler. It has no user agent string of its own, it never makes a request, and it exists only as a control token governing whether already-crawled content feeds Gemini training and grounding. AI Overviews and AI Mode are not covered by it. Those run on ordinary Googlebot access, which means the only robots.txt lever for them is the one that also removes you from Search. If you want to stay in the index but limit what gets lifted, the controls are nosnippet, max-snippet and data-nosnippet, not a Disallow line. That distinction is worth internalising before you argue about it in a meeting, and it is part of why AEO, GEO and SEO keep collapsing into each other in practice.

A file that says what you mean

Here is the shape we use as a starting point for a product site that wants to be cited and does not want to be training data. Adapt the disallow list; do not copy it blind.

``` User-agent: * Disallow: /search Disallow: /cart Disallow: /*?sort= Sitemap: example.com/sitemap.xml

User-agent: OAI-SearchBot Disallow: /search Disallow: /cart Disallow: /*?sort=

User-agent: Claude-SearchBot Disallow: /search Disallow: /cart Disallow: /*?sort=

User-agent: GPTBot Disallow: /

User-agent: ClaudeBot Disallow: /

User-agent: Google-Extended Disallow: / ```

Three things to notice. The retrieval bots inherit nothing, so their junk paths are spelled out again. The training tokens get a clean Disallow: / with no extra lines, because partial training opt-outs are hard to reason about and hard to verify from outside. And Sitemap is a non-group directive, so it applies file-wide regardless of which group a crawler lands in.

Whether that split is the right one for you is a commercial call, not a technical default. A paywalled archive and a SaaS pricing page want opposite things.

Three silent failures

The 500 kibibyte cliff. RFC 9309 sets the parse limit at "at least 500 kibibytes", and Google's robots.txt reference states that Google enforces a 500 kibibyte limit and ignores everything after it. A CMS that writes one Disallow line per retired URL can cross that line without anyone noticing. No AI crawler operator we could find publishes its own parser limit, so the Google number is the one to design against. If your AI crawler groups sit at the bottom of the file, they are the first thing to fall off the edge. Put them near the top.

The 24 hour cache. RFC 9309 says a crawler SHOULD NOT use a cached robots.txt for more than 24 hours. Google's documentation says it generally caches for up to a day and may hold a copy longer when a refetch fails, and OpenAI's crawler page says a robots.txt change can take about 24 hours to reach its search systems. Treat any change you make today as a tomorrow measurement, and do not edit the file and check the logs an hour later.

The 5xx that means a blackout. This is the expensive one. RFC 9309 says that if /robots.txt is unreachable because of a server or network error, the file is undefined and the crawler MUST assume complete disallow. A WAF rule that challenges unfamiliar user agents will happily return 503 on the robots file, and a conforming crawler then treats your entire domain as forbidden while you see an allow-everything file in your browser. What each engine actually does next is only documented in one place we could find: Google says it stops crawling for the first 12 hours, then falls back to the last cached robots.txt for up to 30 days. We could not find published error handling for any other AI crawler, so for those tokens plan for the strict reading of the standard rather than assume a similar grace period. Note that 403 is a different failure, not the same one: RFC 9309 and Google both read a 4xx as no robots.txt at all, which means no restrictions, so an edge that 403s the file deletes your rules instead of enforcing them. The same edge layer overrides the file in the other direction too: a CDN that blocks AI crawlers at the network level makes your Allow: / meaningless. One large provider announced on 1 July 2025 that it was changing the default for sites on its network to block AI crawlers unless they pay for content, a crawl-status change that appears in nobody's robots.txt. We have not measured how many sites it moved. Our longer piece on whether AI crawlers can actually read your site walks through the rendering half of the same problem.

Checks you can run today

Fetch the file as the bot, not as yourself. A plain browser request proves nothing, because the edge decisions are made on user agent and ASN:

`` curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \ example.com/robots.txt ``

You want a 200 and a text/plain content type. Repeat it for GPTBot, ClaudeBot, Claude-SearchBot and PerplexityBot, then repeat the whole set against a real product URL rather than the root. A site can pass on / and fail on /pricing because a bot rule was scoped to a path, so the root is not a safe proxy for the pages you want quoted.

Then count bytes with curl -s example.com/robots.txt | wc -c and confirm you are nowhere near 512000. Finally, grep a week of access logs for each token. A file that permits a crawler which has never once appeared in your logs is telling you the block lives somewhere above the origin. The self-check lists ten of these, robots access among them, if you want the full pass rather than this one slice.

What robots.txt will not do for you

It is a crawling instruction, not an access control and not a privacy mechanism. Two limits deserve saying plainly.

OpenAI's current crawler documentation states that because ChatGPT-User actions are initiated by a person, robots.txt rules may not apply to them. Read that as written: a user-triggered fetch can reach a path your file disallows. If a page must not be read by a model, authentication is the control, not a text file.

The second limit is that permission is not retrieval. Allowing every AI crawler does not make an engine choose you, any more than an open door fills a shop. Access is the floor of answer engine optimisation, and the floor is where most recoverable loss sits, but it is still the floor. We also do not know how much weight any engine gives the newer conventions, which is why llms.txt is worth watching and not worth prioritising over a robots file that returns 200 to every bot you care about.

Honest limits

This post rests on published documentation and on a check you can run yourself, not on a measurement of how engines behave in the wild. The group-matching rule, the 500 kibibyte limit, the cache guidance and the 5xx handling come from RFC 9309 and Google's robots.txt reference, both read on 12 October 2026; the other crawler operators do not publish their parser limits or their error handling, so for those tokens we are assuming conformance rather than observing it. The curl check above spoofs a user agent string from your own machine, which tells you what your edge does with that string and nothing about what the real crawler sees from its own IP ranges. We have no figure for how many sites edge-level AI blocking affects, and we cannot see inside any engine to tell you how fast a robots.txt change is acted on once the cache expires. Where this post says a crawler must do something, that is the standard speaking, not our log data.

FAQ

What is the robots.txt entry for GPTBot? A group of its own: User-agent: GPTBot followed by Disallow: / to opt out of OpenAI's model training crawl, or Allow: / to permit it. The token carries no version number, and it does not govern OAI-SearchBot, which is the separate token for ChatGPT search. Whatever you disallow in your global group, repeat it inside this one.

Does blocking GPTBot remove me from ChatGPT? No. GPTBot is documented as the training crawler. ChatGPT's search behaviour uses OAI-SearchBot, a separate token, and blocking one does not block the other.

Does Google-Extended control AI Overviews? No. Google-Extended governs Gemini training and grounding and has no user agent of its own. AI features in Search run on Googlebot access, so the available levers there are the snippet controls rather than a Disallow line.

Should I use a wildcard group for all AI bots? You cannot. There is no pattern matching on user agent names beyond the single * group, so each bot needs its name spelled out, and each named group must repeat the rules you want it to keep.

How long until a robots.txt change takes effect? Expect up to a day for the cached copy to expire, then another crawl cycle before behaviour visibly shifts. Measuring the same afternoon will mislead you.

Is a missing robots.txt better than a bad one? RFC 9309 and Google both read a 404, and any other 4xx except 429, as no robots.txt at all, which means no crawl restrictions. A 5xx is the dangerous one: the standard says the crawler must then assume complete disallow. Serving no file beats serving one your edge turns into a server error.

Sources

Every URL below was opened and checked on 12 October 2026. The OpenAI, Google and Anthropic crawler pages were opened again and re-checked token by token on 15 October 2026.

Related reading

Can AI crawlers actually read your siteDoes llms.txt do anything yetWhat is answer engine optimisationAEO vs GEO vs SEO: what the three terms actually meanReading your server logs for AI bot trafficShould you block AI crawlers