Blog · Machine readability

Duplicates, canonicals and confused AI answers

Published 25 August 2026 · Updated 8 October 2026 · 8 min read

Duplicates, canonicals and confused AI answers
Photograph: Jonas Jacobsson / unsplash

The short answer: Google and Microsoft both document the same mechanism: near-identical URLs get grouped into one cluster, and one member of that cluster is handed the speaking part. Nobody is penalising you for the copies. The cost is that your brand gets represented by whichever variant the system picked, which may be an old print view, a parameterised copy or a stale regional page. The answer then quotes a price, a claim or a product name you stopped using months ago, and nothing in your analytics tells you why.

Most teams treat duplicate URLs as a tidiness problem for the sitemap. In answer engines it is a casting problem. One page gets the speaking part.

What clustering actually does

Google's own documentation is unusually direct about the mechanism. When it finds several pages whose primary content looks the same, it clusters them, then selects the page it judges most complete and useful for users and marks that one as the canonical. The signals it weighs include HTTP versus HTTPS, redirects, whether the URL sits in a sitemap, and the rel="canonical" annotation. The companion page on consolidating duplicate URLs grades them: a redirect and a rel="canonical" annotation are each described as a strong signal, and sitemap inclusion as a weak one. That grading is Google's, written for Google Search; we have not found a published equivalent from the other engines, so do not carry the ordering across. Google also now states plainly that the canonical page should carry a self-referential canonical pointing at itself.

Read that list again from the point of view of a brand page. Nothing in it says "the page the marketing team considers official". The selection is made from signals, and a cluster with contradictory signals gets resolved by the system's own judgement rather than yours.

Microsoft described the consequence for AI answers in a Bing webmaster post dated 19 December 2025: when several URLs carry the same content, signals including clicks, links, impressions and engagement get spread across them, and language models group near-duplicate URLs into one cluster before choosing one page to represent the set. If the pages barely differ, the model may pick a version that is out of date or simply not the one you wanted shown. That is the whole failure in two sentences.

Why the retrieval layer makes it worse

A classic search result at least showed you the URL that won. An answer often does not. The passage arrives stripped of its address, blended with two or three other sources, and the citation chip may point at a cluster representative you have never audited.

Retrieval pipelines also tend to deduplicate the passages they have fetched before the model writes. How much that step removes depends on the corpus. A preprint submitted 10 May 2026 measured byte-exact chunk deduplication on public benchmarks and reported three very different regimes: 0.16 percent of bytes removed across 22.2 million BeIR passages, 24.03 percent on constructed enterprise patterns, and 80.34 percent in multi-turn conversational traffic. Those are that paper's corpora, not any commercial engine's. We have not found an engine that publishes whether it deduplicates or by how much, so read the figures as a range the technique can produce rather than as a measurement of what happens to your pages. The useful part for a marketer is that duplicate load is not a constant: a tidy corpus barely notices duplicates and a messy one is dominated by them.

When a near-duplicate survives, the model has two slightly different versions of your claim in front of it. We cannot watch it reconcile them. What we can read is the output, and in the audits we run the output is typically a hedge, a claim averaged between the two, or the wording of one variant used whole. In those cases it was not the sentence on the page the client treats as official.

Where the duplicates come from

The list is boring, and in the audits we run it is nearly always the same.

URL parameters from campaign tags, filters, sorting and sessions. Trailing slash and non-slash variants of one path. HTTP alongside HTTPS, www alongside the bare host. Print views and legacy mobile paths. Regional pages that share most of their body text with the global one. Syndicated copies on partner domains with no attribution back. Staging hosts nobody blocked. PDFs of the same datasheet, renamed each quarter.

Then the newer one. If your canonical tag is injected by JavaScript, canonicalisation can happen before rendering as well as after, so the tag in the raw HTML and the tag in the rendered DOM have to agree. Two different canonicals for one page leave an engine picking which of your own instructions to believe, and the annotation is only one signal in Google's list, so a contradictory pair weakens the one input you control.

A check you can run this afternoon

This takes about twenty minutes for a small site and tells you more than a crawl report.

  1. Pick your five commercial pages. For each, write down every variant you can construct:

    2. http, https, www, non-www, trailing slash, one with ?utm_source=test, the print view if one exists, and the regional versions.

  2. Fetch each variant from the command line, not the browser, and read the response headers

    3. and the raw HTML. curl -sIL <url> gives you the redirect chain and the final status. curl -s <url> | grep -i 'rel="canonical"' gives you the declared canonical.

  3. Compare. Every variant should either redirect with a 301 or 308 to one address, or declare

    4. that same absolute canonical. The winner should declare itself.

  4. Render the page with JavaScript on and check the canonical again. If it changed, you have

    5. found a real defect.

  5. Ask three engines a brand question and record which URL each one cites. If the cited URL

    6. is not the one you just declared canonical, the cluster is being resolved against you.

Step five is the one people skip, and it is the only step that measures the thing you actually care about. A canonical tag is an input. The cited URL is the output.

The identity layer underneath

Duplicate pages produce duplicate structured data, and that is a second, quieter failure. Four copies of the same Organization node with four different @id values, or the same Product marked up on a parameterised variant with a different offer, gives an engine more than one description of a single thing. Structured data for AI search: what to mark up first covers the markup order; the point here is narrower. One entity, one stable identifier, declared on the canonical page and referenced from the others.

Identity links do the same job from outside. When your profiles, directory listings and knowledge-graph entries all point back at one address rather than three, the cluster has an obvious winner. sameAs and the identity links engines check goes through which of those links carry weight. On the assistant side, how Gemini answers brand questions shows what a loosely resolved entity looks like from the user's seat: a hedge, a competitor named alongside you, or a confident sentence about the wrong company with a similar name.

What to expect after you fix it

Slowly. A duplicate cluster is a stored decision, and the system has to notice your change, recrawl the variants, and re-resolve the group. Google grades a redirect as a strong signal, and a redirect also takes the variant out of circulation rather than leaving it live, so it is the firmer of the two moves. We have not measured how much sooner a redirected variant leaves an answer than an annotated one. Canonical annotations are a request, honoured often but not unconditionally.

Expect the search index and the answer surfaces to disagree for a while. We have seen answers keep quoting a retired page after the index had moved on, with the redirect already in place. Whether that is a cached retrieval or older training material we cannot tell from outside, and neither can you. How long AEO takes to show up in answers sets out the realistic windows. Keep the old URL redirecting for a long time rather than deleting it, because third-party links and assistant caches both take their time.

If you want the shape of this written up properly, with the cited URL recorded next to each finding, the sample audit report shows how we lay it out.

Honest limits

We cannot see the clusters. None of the five engines we test publishes which URLs it grouped or why it picked the representative it picked, and we have found no such disclosure in their documentation, so every diagnosis here is inferred from which URL gets cited rather than observed directly. Our audits run a set of about 45 prompts across five engines, with retrieval on and with it off, recording the cited URL for each answer and repeating prompts to average out some of the run-to-run variation; that is a sample of one brand's answer surface on the days we ran it, not a census of anything. The grading of canonicalisation signals quoted above is Google's own, written for Google Search, and we do not assume the other engines weigh them the same way. The deduplication figures come from one preprint on public benchmarks rather than from an engine's own measurements, so they show what the technique removes on those corpora and not what happens inside a commercial answer engine. We also have no figure for how much answer error traces back to duplicate resolution rather than thin content or weak corroboration: it turns up repeatedly in our casework and rarely alone, but we have not counted it.

FAQ

Do duplicate pages get a site penalised? No. Google's documentation describes clustering and canonical selection, not a punishment. The cost is dilution and misrepresentation: your signals spread across variants, and one variant you did not choose speaks for the group.

Is a canonical tag an instruction or a hint? A hint, and Google says so: its documentation calls the annotation a strong signal rather than a directive, and says the canonical it picks may differ from the one you declare. If a URL must not represent you, redirect it rather than annotating it.

Should the main page carry a canonical pointing at itself? Yes. Google's duplicate-URL documentation, updated 10 July 2026, recommends a self-referential canonical on the canonical page itself. It costs nothing and removes an ambiguity.

My AI answer quotes an old price. Where do I look first? Search for every URL that still carries that price, including parameterised copies, PDFs and syndicated versions on partner sites. The engine is probably reading one of them, and removing or redirecting it is faster than trying to outrank it.

Sources

Related reading

Structured data for AI search: what to mark up firstsameAs and the identity links engines checkHow long AEO takes to show up in answersHow Gemini answers brand questionsBrand entity resolution: making AI engines sure who you areFAQPage and HowTo after the rich-result changes