Blog · Running the work

When the model makes something up about you

Published 30 September 2026 · Updated 12 October 2026 · 8 min read

When the model makes something up about you
Photograph: Kaŕeem Saleh / unsplash

The short answer: A model inventing a fact about your company is almost never random noise. It is a gap being filled, and the gap is usually yours: a claim nobody states plainly on a page the engine can fetch, a stale version of you still sitting in training data, or another company whose name collides with yours. Work out which of the three it is before you try to fix anything, because the three have different fixes and very different timelines.

The reflex when a buyer forwards you a screenshot of ChatGPT getting your pricing wrong is to treat it as a bug in the model. It is more useful to treat it as a reading of your public footprint, taken by something that had to produce a sentence whether or not the evidence was there.

Why it guesses rather than declining to answer

The behaviour has a mechanism, and the mechanism is about scoring. A paper submitted on 4 September 2025 by researchers at OpenAI and Georgia Tech argued that hallucination survives because the benchmarks used to rank models mark an answer right or wrong and give nothing for saying "I do not know". Under that scoring, a model that guesses your founding year outscores a model that abstains, every time. The paper calls the result an epidemic of penalising uncertain responses, and proposes changing how the main leaderboards are graded rather than adding more hallucination tests.

That matters for you in one specific way. The model is not trying to deceive anyone about your company. It is completing a pattern under uncertainty, and it has been trained in an environment where completing beats hedging. Remove the uncertainty and the behaviour changes. Leave the uncertainty in place and no amount of complaining does anything.

It is worth knowing how good the best case looks. A public factual-consistency leaderboard, last updated on 22 September 2026, hands a model the full source document and asks only for a summary of it, across more than 7,700 curated articles at temperature zero. Even in that setup, where every fact needed is sitting in the context window, the three best rates on that run were 1.8 percent, 3.1 percent and 3.3 percent invented content. Read those figures narrowly. They are scored by the leaderboard's own evaluation model rather than by human raters, they rank named models on a summarisation task, and they say nothing directly about what a consumer engine answers when somebody asks about your brand. That case is harder, because nothing has been handed over. The engine has to go and find the document first, and often it does not.

Four things that look identical and are not

Every wrong sentence about a brand that we have had to diagnose has fallen into one of four buckets. That is a reading of the audits we have run, not a taxonomy anyone has validated at scale, so treat the four as a working sort rather than a closed set.

Gap fill. Nobody states the fact anywhere fetchable, so the model supplies a plausible one. Pricing that lives only inside a sales deck is the classic case. So is a list of integrations that exists as logos on a page and as nothing else.

Stale memory. The fact was true, you changed it, and the old version is in the training data. Models carry a knowledge cutoff, and anything after it is invisible unless the engine retrieves live. A rename, a pivot or a price change can sit in parametric memory for a long time after your site has moved on.

Entity collision. There is another company with a near-identical name, and the engine has merged the two. This produces the strangest errors, because the sentence is true about somebody. In our work it shows up most often for firms whose name is a common word in a crowded category.

Corroborated error. Somewhere there is a real page, often a directory listing, an old press release or a review site, that states the wrong thing, and the engine is citing it correctly. This one is not a model failure at all.

The check that separates them: run it twice

This is the most useful diagnostic we have, and it takes about ten minutes. Ask the same question twice, once with browsing or search switched on, once with it off. Then read the pair.

Wrong with retrieval off, right with retrieval on: the fact exists on a page the engine can reach, and the stale version is in the weights. Lower urgency on the surfaces that retrieve by default, which covers most of the consumer ones we test, though we have not checked them all and a default can change without notice.

Wrong in both, same wrong answer: a source is feeding it. Ask the engine for its citations and go read them. This is usually a corroborated error, and the fix is at the other end, not on your site.

Wrong in both, differently wrong each time: a true gap. Nothing authoritative exists, and the model is improvising afresh each run. This is the one you can fix fastest.

Right with retrieval off, wrong with retrieval on: rarer, and it means a competing or confusing source is outranking your own page at retrieval time. Check for a near-name collision before anything else.

Run each question five or six times, not once, because two identical prompts minutes apart can take different paths. Our own audit prompt set exists for this reason: about 45 prompts across five engines, retrieval on and retrieval off, with the evidence kept so a finding can be argued with rather than taken on faith. The sample audit report shows how the pairs get written up.

What each one costs to close

A gap is the cheap one. Put the claim in plain prose, in a sentence that stands on its own without the paragraph around it, on a page that returns it in the HTML rather than assembling it in the browser. Then mark it up. Everything in what makes a page citable by an AI engine applies here, with one addition: write the fact in the form the question takes. If people ask what it costs, the page should contain a sentence with the number and the currency in it, not a link to a pricing calculator.

An entity collision is identity work. This part is Google's documented behaviour, and we cite it as Google's alone. Google's Organization structured data guidance says the url property helps Google uniquely identify your organisation, and that some properties, naming iso6523 and naics, are used behind the scenes to tell your organisation apart from others. It separately recommends name and alternateName for what you are called, logo and url for online presence, and sameAs for your profile pages elsewhere. Other engines may read the same markup, but we have not found equivalent published guidance from them, so treat the rest as reasonable inference. If you share a name with another firm, the disambiguation properties are not decoration.

Stale memory is the one you cannot push. We know of no published retraction endpoint and no documented form that edits a model's weights, and we have looked. That is what the public documentation shows, which is not the same as proof that no route exists. What you can do is make the current version so well corroborated that retrieval always finds it, and then wait for the next training run. That wait is measured in model releases, which is one more reason to track what a model update does to your visibility rather than only your own site changes.

A corroborated error is a correspondence job. Find the page, ask for the correction, and if the source will not move, publish a page that states the right version more clearly and hope retrieval prefers it. Nobody enjoys this part.

Who should own it, and when

Finding these is cheap. Fixing them crosses teams, which is where it stalls. The schema change sits with engineering, the pricing sentence sits with marketing, and the directory correction sits with whoever will pick up the phone. Sequencing that is most of the first 90 days of AEO work. If the fixes keep getting queued behind a product roadmap, that is a structural answer about in-house or agency for AEO, not a motivation problem.

Honest limits

The four-bucket sort is a reading of the audits we have run, not a validated taxonomy, and we have not counted how often each bucket occurs, so no proportion is offered here. We cannot tell you how long a corrected fact takes to displace a stale one in parametric memory, because that depends on training schedules nobody publishes. We have no reliable way to count how often a wrong sentence about a client is shown, only how often it appears in prompts we chose to run: a prompt set is a sample, not a census, and anyone quoting you a hallucination rate for your brand is extrapolating from one. The retrieve-twice check and the markup advice are drawn from Google's published guidance and our own runs, so the parts that go beyond Google are inference rather than documented engine behaviour. Where we say no correction route exists, we mean none is published, which is weaker than none existing.

FAQ

Can I get a model to retract something false about my company? Not as far as we can tell. We have found no published correction form that edits a model's training data. You can change what retrieval finds, and you can correct the third-party page the engine is citing, which covers most of the cases we see.

Is a wrong answer about my brand the model's fault or mine? Often it is neither, strictly. The most common cause in our audits is that the fact was never stated plainly anywhere the engine could fetch it, so the model filled the gap. That is usually fixable on your side, and the fix lands whenever the engine next crawls the page.

How many times should I run a prompt before I believe the result? Five or six runs, on more than one engine, and record the citations each time. A single run tells you a path was taken, not that it is the usual path.

Does structured data stop hallucinations? It helps with identity and collision, which account for a large share of the brand errors we have diagnosed. It does nothing for a fact you have not written down in readable prose, because the markup describes the page rather than replacing it.

Sources

All three checked on 12 October 2026, and each still states what the sentence citing it says.

Related reading

The first 90 days of AEO workIn-house or agency for AEOWhat makes a page citable by an AI engineWhat a model update does to your visibilityQA for changes meant to move AI visibilityTool sprawl in AI search, and the cost of it