Blog · Access and retrieval

Paywalls, logins and what retrieval can reach

Published 22 September 2026 · Updated 8 October 2026 · 8 min read

Paywalls, logins and what retrieval can reach
Photograph: Elimende Inagella / unsplash

The short answer: Retrieval reaches whatever an anonymous request returns. No search crawler that publishes documentation, OpenAI's included, describes itself as carrying a reader's credentials, so a page behind a login reaches the index as a sign-in form. A paywall is different: a fetcher reads as much as the page serves to a signed-out visitor, which is sometimes the whole article and sometimes sixty words. The question worth answering is not whether you are gated. It is what the gate returns.

Most teams discuss this as a policy question. It is a response-body question first, and the answer is sitting in your own server response right now.

Three walls that behave nothing alike

A login wall is the blunt case. The fetcher arrives with no cookie and no account, so the server hands it a sign-in form. That form is what gets parsed, and when such a page is cited the reader lands on "email address" and "forgot password" rather than on your argument. The content behind the wall never entered the retrieval pipeline at all.

A client-side paywall is the opposite failure. The server returns the full article to every requester and the browser hides the tail of it with CSS or a JavaScript overlay. A human sees three paragraphs. A retrieval fetcher, which parses markup and rarely cares about your overlay, sees the lot. Google's paywall guidance warns about this directly: if you do not want the text accessible to the browser at serving time, it says, choose an implementation that does not supply it, because a client-side JavaScript solution does. The configuration looks correct in a browser and is wrong only in the raw bytes.

A server-side paywall sits between those two. The origin truncates the body before it leaves the building, so the fetcher receives whatever free sample you configured. That sample is the only part a fetcher can turn into a passage. If it is a headline plus a standfirst, your entire argument for being the authority on the topic is competing against other people's full pages with sixty words.

The metered wall is a fourth case and it usually works in your favour. Metering counts visits with a cookie. A fetcher that does not carry one between requests looks like a first-time reader every time and gets the free view. No search crawler we have read documents retaining cookies, but we cannot see inside their fetchers, so treat this as the likely behaviour and not as a promise.

Why the index is not logged in as you

It helps to be precise about who is knocking. OpenAI documents separate agents for separate jobs: GPTBot for model training, OAI-SearchBot, which indexes pages so ChatGPT's search answers have something to cite, and ChatGPT-User for fetches a person has just triggered in a conversation. The documentation is explicit that because ChatGPT-User requests begin with a user action, robots.txt rules may not apply to them, and it notes that a change to your robots.txt takes roughly 24 hours to register for the search crawler.

OpenAI's documentation describes none of those as carrying a reader's credentials, and the shape of the problem generalises: a credential belongs to one person, while an index is shared. We can point at what OpenAI publishes. For engines that document less we are reasoning from that same architecture and from what reaches the logs we are given, which is weaker ground. So the thing people mean when they say "our docs are gated but AI still knows about them" is usually not retrieval. It is either training exposure from an older open version of the page, or a third party who republished your material in the clear. Separating those two cases is the whole subject of retrieval vs training data: which one is answering about your brand, and it changes what you should do next.

An agentic browser is the exception that proves the rule. When an assistant drives a browser inside the user's own session, it can read a logged-in page, because the user is logged in. That read serves one conversation. It does not deposit anything in an index, it is not repeatable by anyone else, and it is not a visibility strategy.

Tell the engine which half is free

For paywalled articles there is a documented way to declare the arrangement rather than leave it to be guessed. Google's structured data for subscription and paywalled content uses isAccessibleForFree set to false on the CreativeWork or NewsArticle node, plus a hasPart entry whose cssSelector names the class wrapping the gated section. The documentation, last updated 8 September 2026, says in plain terms that this markup is how Google separates a legitimate paywall from cloaking, which is a spam policy violation.

Two rules in that page are the ones people get wrong. The selector must be a class, not an ID, and gated sections must not be nested inside one another. Get either wrong and the markup stops describing your page, which leaves you serving different content to crawlers than to humans with no declaration attached. That is the position nobody wants to be in.

There is now a third kind of answer a wall can give. Cloudflare's pay per crawl, announced on 1 July 2025, revives HTTP 402 for crawler traffic: an unpaid request gets a 402 with a crawler-price header, and a crawler willing to pay sends crawler-exact-price or crawler-max-price and receives a 200 with crawler-charged. Worth understanding before you switch it on: in Cloudflare's own description the unpaid request gets the 402, so for a crawler that has not agreed the price the immediate effect on retrieval is the same as a deny rule. It was announced as a private beta, and an operator can only pay after setting payment up with Cloudflare, so how many fetchers actually pay is not visible from a publisher's side. Whether that trade is right for you is the argument in should you block AI crawlers.

The check, in four commands

Do not reason about this from your CMS settings. Fetch the page the way the engines do and count words.

`` curl -sL -A "OAI-SearchBot/1.0" example.com/article | wc -w curl -sL -A "Mozilla/5.0" example.com/article | wc -w curl -sL -A "OAI-SearchBot/1.0" example.com/article | grep -c "paragraph from deep in the piece" curl -sL -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot/1.0" example.com/article ``

Read it like this. If the third command returns 1, your paywall is cosmetic and the full text is being served to anyone who asks. If the first count is a small fraction of the second, your origin is varying by user agent, which is the cloaking shape and needs the markup above to be legitimate. If the status code is 403 or 401, you are blocked at the edge rather than gated at the application, which is a firewall conversation, not a content one. Running the same four against a competitor's article takes five minutes and tells you what their origin serves, which is the part of their setup you can verify yourself.

Pair this with a render check, because a page that returns full text to curl can still render empty in a headless browser. Can AI crawlers actually read your site covers that half. The self-check has the rest of the list, and none of it needs tooling.

What the gated half is worth

Gating is often correct. Across the twenty countries the Reuters Institute tracks as its pay basket, 17 percent of people paid for online news in its Digital News Report 2026, published 16 June 2026, against 18 percent in 2025, a change the report itself calls roughly unchanged. The same report puts use of AI chatbots for news at 10 percent of people, up from 7 percent a year earlier, and at 16 percent among the under-35s, while noting that several large markets including the USA, the UK, France and Germany recorded no increase at all. Those are survey answers about readers, not measurements of what any engine retrieves, so read them as context for the decision rather than as evidence about your own pages.

The practical move is to stop treating the decision as one switch. Split the corpus. Keep the proprietary analysis, the data tables and the numbers behind the wall, where they are worth paying for. Put the definitional and comparative material, the part that makes you the obvious source for a question, in front of it, written so that a single passage answers something. Then measure whether the change moved anything, by running the same prompt set before and after with testing with retrieval on and off. A sample audit report shows what that evidence looks like when it is written down.

Honest limits

We can show you what a fetcher receives from your origin, because a response body is observable, and that is the part of this post we stand behind without qualification. We cannot tell you how any engine weights a short free sample against a long open page, because none of them publish it, and what we see in client audits is a sample rather than a study, so treat anyone offering a precise number there as guessing. The crawler behaviour described here comes from OpenAI's own documentation; for engines that publish less we are inferring from the same architecture and from the server logs clients give us, which is weaker evidence and is the claim a careful reader should push on first. No credentialed fetcher appears in the logs we have seen, which is not the same as none existing anywhere. The Reuters Institute figures are survey data about news audiences, and Cloudflare's pay per crawl was a private beta at announcement, so neither tells you what is reaching your pages today.

FAQ

Can ChatGPT or Perplexity read pages behind a login? Not through their indexes, so far as their documentation describes them. OpenAI's documentation describes none of its fetchers as carrying a reader's credentials, and an anonymous request receives the signed-out response, usually the login form. An assistant driving a browser inside your own logged-in session is a separate thing, and what it reads stays in that one conversation.

Does a paywall stop my content being used for training? Only if the server actually withholds the text. A paywall implemented in CSS or JavaScript ships the full article to every requester, so the gate exists in the browser and nowhere else. Check the raw response before assuming.

Is serving a shorter page to crawlers cloaking? It is when it is undeclared. Google's paywalled content markup, isAccessibleForFree plus a hasPart with a cssSelector, is the documented way to declare the difference so it is read as a paywall rather than a spam signal.

Should I open an archive to get cited? Sometimes, and rarely all of it. Opening the pages that answer definitional and comparative questions costs little subscription revenue and does most of the retrieval work. Keep the material people actually pay for gated.

Sources

Related reading

Can AI crawlers actually read your siteShould you block AI crawlersRetrieval vs training data: which one is answering about your brandTesting with retrieval on and offHow to write robots.txt for AI crawlersReading your server logs for AI bot traffic