Blog · Engines

How Claude cites sources on the web, and when it does not

Published 10 July 2026 · Updated 15 October 2026 · 9 min read

How Claude cites sources on the web, and when it does not
Photograph: Brina Blum / unsplash

The short answer: Claude cites the web through two separate tools. A search tool returns ranked results with a title, a URL and a freshness stamp, and citations on it are on by default. A fetch tool pulls the full text of one page, and citations there are optional and off unless the application turns them on. Each search citation carries a quoted snippet that Anthropic's documentation caps at 150 characters, so a short sentence fits inside the excerpt and a long one cannot.

Does Claude cite sources? Yes, when it searches. Citations are on by default for the web search tool, and each one carries the URL, the page title and a quoted excerpt of up to 150 characters. When Claude answers from what it already holds, it runs no search, shows no citation, and the interface does not announce which of the two just happened. So the honest answer is that Claude cites sources for the questions it decides need the live web, and not for the rest.

That cap applies to the search tool, which is where most brand answers come from, and it is the part most people miss.

Two tools, not one retrieval system

Anthropic's documentation describes web search and web fetch as separate server tools with separate behaviour. Search runs inside the request, returns a list of results, and hands Claude a url, a title, a page_age value for when the page was last updated, and an encrypted copy of the content. Fetch takes a single URL and returns the whole document, text, HTML or PDF only.

The split matters because your page can win one and lose the other. Search puts you in the candidate pool. Fetch is what happens when Claude decides your page specifically is worth reading in full. A site can be findable and still unreadable, and the error codes name the reasons: url_not_accessible for an HTTP failure, unsupported_content_type for anything outside text, HTML and PDF, url_too_long for a URL over 250 characters.

One documented limit is worth pinning to the wall. The fetch tool does not render JavaScript. A page whose body text only appears after the client-side bundle executes arrives at that tool as an empty shell, and writing quality does not change what the fetch tool received. Anthropic documents this for Claude's fetch tool only. Other engines render JavaScript to varying and mostly undocumented degrees, so do not read it as a rule that holds beyond Claude. The failure is invisible in a browser and visible in curl.

There is a second constraint on fetch that reads like a security note and behaves like a distribution rule. Claude can only fetch a URL that already appeared in the conversation: in a user message, in a client tool result, or in an earlier search or fetch result. A URL that Claude produced from memory alone is refused. In practice, search is the doorway to fetch, which is why the ordering in how each AI engine picks the sources it cites is not a detail.

The citation object, and the 150 character rule

Each citation Claude returns from a search carries four fields: the URL, the page title, an encrypted index used to keep the citation valid across turns, and cited_text, which the documentation describes as up to 150 characters of the cited content. Anthropic also notes that cited_text, title and url do not count toward input or output token usage. What the documentation does not say is which 150 characters get selected, so the reasoning that follows is about what can fit, not about what Claude prefers.

Sit with the number. One hundred and fifty characters is roughly twenty to twenty five English words. Spread the claim across a forty word sentence with a subordinate clause in the middle, and no excerpt of that length can hold the whole of it. State it in one short declarative sentence, and the whole claim fits inside the budget.

So the practical instruction is unglamorous and specific. Write the quotable version of each fact as one sentence under about 140 characters, with the subject and the number in it. "Our audit turnaround is seven working days" is 42 characters, so the whole claim fits with room to spare. "Turnaround times vary by scope, but in most engagements clients receive their findings within roughly one working week of kickoff" is 129 characters and also fits, but what it spends the budget on is the hedge, because the number was never in it.

The same discipline is the whole of answer-first writing: state the fact, then explain it, rather than the other way round.

Three crawlers, three separate robots.txt decisions

Anthropic's crawler documentation, updated in April 2026, names three agents and gives each a different job. ClaudeBot collects content for model training. Claude-User fetches a page when a person asks Claude a question. Claude-SearchBot crawls to improve search quality inside Claude.

Blocking one does not block the others, and this is where sites lose answers by accident. A robots.txt rule written during the 2024 scraping panic usually targets ClaudeBot, which is the training crawler. Leaving it in place is a defensible choice. A rule that also catches Claude-User stops retrieval at the moment a real person is asking a real question, and Anthropic's own wording says blocking it may reduce the site's visibility. Blocking Claude-SearchBot keeps you out of the index that search draws candidates from.

Anthropic publishes the IP ranges its crawlers use at https://claude.com/crawling/bots.json, and says IP blocking is the wrong instrument because it also stops the crawler reading your robots.txt. Check that list before anyone adds a firewall rule from a log line.

When Claude does not search at all

Retrieval is conditional. Anthropic's documentation says Claude searches when the request depends on information that is current, changing or outside training data, and answers directly for established facts, creative work, or analysis of what is already in the conversation. A developer can cap it with max_uses, and searches are metered at $10 per 1,000 on the API, so the restraint is also commercial.

Here is the consequence for a brand. "What is the best project tool for a small agency" is the kind of question that triggers a search. "What does [your company] do" may not, if the model already holds an answer, and the answer it holds was fixed at training time and carries no citation you can inspect or correct. Two prompts about you, one retrieved and one remembered, can disagree, and only one of them has a source you can influence this quarter.

That is a different pattern from how Google AI Overviews chooses its sources. Google Search Central says there are no additional requirements to appear in AI Overviews or AI Mode beyond ordinary Search, and that AI Overviews were designed to show on queries where they add benefit. Neither company publishes the rate at which a query triggers retrieval, so treat the contrast as structural rather than quantified. Treat the two as different channels with different mechanics, which is also the reason the naming argument in AEO vs GEO vs SEO is mostly a distraction from engine-specific work.

Newer versions of the search tool add a filtering step before results reach the model. From web_search_20260209 onward, Claude can write and run code that trims the result set so only relevant content enters the context window, which Anthropic frames as a token saving rather than as a ranking change. The filtering code is written per request and is not published, so what it keeps is not documented. Our reading, and it is a reading rather than a finding, is that a page stating its answer in one identifiable block is easier for any relevance filter to retain than one where the answer is spread across 2,000 words.

Four checks worth running this week

First, fetch your own key pages the way the tool does. curl -s yoursite/page | grep -c "your key claim" with no browser involved. A zero means the claim is not in the HTML, which means it is not in the answer.

Second, read robots.txt for all three agent names separately. Decide each one on its merits and write the decision down, because the next person to touch the file will not know why it is there.

Third, measure your own quotable sentences. Take the ten facts you want an engine to repeat and count the characters. Anything over roughly 140 gets rewritten until it is one clause.

Fourth, run the same brand prompt five times across a week and record whether a search happened at all. Five runs is a spot check, not a measurement, but a run with no sources is telling you something different from a run with the wrong sources, and the fixes are not the same. The self-check is ten tests in this shape if you want a fuller list.

Honest limits

Everything documented above is documented for Claude, and for the API versions of the tools named in the Sources. It is not a description of how other engines behave, and the consumer apps may differ from the API in ways neither we nor Anthropic have published.

The ranking logic inside Claude's search step is not published, so nobody outside Anthropic can say why one of two similar pages was cited. The same goes for which 150 characters become cited_text and for what the dynamic filtering step keeps: the writing advice here follows from the documented size budget, not from any disclosed selection rule.

This post reports no measurement of our own. Our audits run a prompt set of about 45 prompts across five engines with retrieval on and off, which is a sample of one studio's prompts on the days it ran, never a census of how Claude behaves. Public figures for how often Claude searches come mostly from vendor samples rather than from Anthropic, so we quote none of them here.

FAQ

Does Claude always show a source link? No. Citations are on by default for the search tool, but Claude only searches when the question needs current or changing information. An answer drawn from training data has no citation attached, and nothing in the interface flags that difference clearly.

How long is the quoted snippet next to my link? For the web search tool, up to 150 characters of cited_text, per Anthropic's web search documentation. Fetch citations are a separate feature with their own behaviour. Which 150 characters are chosen is not documented.

If I block ClaudeBot, am I out of Claude's answers? Not by itself. ClaudeBot is the training crawler. Retrieval for a live question runs through Claude-User, and search indexing through Claude-SearchBot, each with its own robots.txt entry.

Does Claude cite sources? Yes, when it searches. Citations are on by default for the web search tool, and each one carries a URL, a title and a quoted excerpt of up to 150 characters. When Claude answers from training data it runs no search and shows no citation, and nothing in the interface marks the difference, so a brand answer with no sources is a different problem from one with the wrong sources.

How do brand citations in Claude work? The citation attaches to a passage, not to your page. The search step puts you in the candidate pool, and the citation appears when a sentence in the answer traces back to text that fits inside a 150 character excerpt. That is why a page can be found by Claude, be entirely accurate, and still never be quoted: the claim was never stated in one liftable sentence.

Why does my page get found but never quoted? Three causes we can point at in Anthropic's documentation: the content needs JavaScript to appear, which the fetch tool does not render, the content type is outside text, HTML and PDF, or the claim is never stated in one compact sentence a 150 character excerpt can hold. Selection beyond those documented limits is not something anyone outside Anthropic can observe.

Sources

Related reading

How each AI engine picks the sources it citesHow Google AI Overviews chooses its sourcesWhat is answer engine optimisationAEO vs GEO vs SEO: what the three terms actually meanMicrosoft Copilot and the Bing index: what it means for visibilityWhy five engines give five different answers about you