Sentiment in AI answers: worth tracking or not

The short answer: Worth tracking, but not as a score. The useful object is the sentence an engine writes about you and the source it got that sentence from. A single positive or negative number on top of that is a second model grading a first model, and the research on model-as-judge documents biases in that grading layer. Track recurring claims. Treat the index number as a rough thermometer, not as evidence.

Many AI visibility products now ship a sentiment metric. The pitch is reasonable. An engine that mentions you while calling you hard to learn is not the same outcome as an engine that mentions you while calling you the default choice, and a presence count cannot tell those apart. The pitch is right about the problem. The number it hands you is weaker than it looks, and knowing exactly where it is weak is what lets you use it.

What the number actually is

None of the engines we checked publishes a sentiment field. In October 2026 we read the public developer and publisher documentation for ChatGPT, Perplexity, Google AI features, Gemini and Copilot and found no documented field that returns "this answer was 0.62 favourable towards your brand". We can only speak to what those pages document, not to what exists behind an enterprise contract. On that evidence, a sentiment metric is constructed rather than read, in two steps. Step one samples an answer: a prompt goes to ChatGPT or Perplexity or Gemini, retrieval runs, text comes back. Step two feeds that text to a classifier, usually another language model with a rubric, which returns a label or a figure. The dashboard aggregates step two.

Both steps are stochastic, and they fail in different ways. Step one varies because retrieval varies, which is the subject of its own argument about measuring AI visibility without fooling yourself. Step two varies because the judge is a model. That second source of noise is the one almost nobody budgets for.

The judge has known biases, and they are measured

The research on using a language model as an evaluator is unusually honest about its limits. The NeurIPS 2023 paper that popularised the method, *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*, reports GPT-4 as a judge reaching over 80 percent agreement with human preference, and names three failure modes in the same breath: position bias, verbosity bias and self-enhancement bias.

Read that 80 percent figure for what it measured. It was measured in 2023, on one judge model, choosing between two chat assistant responses on MT-Bench and Chatbot Arena. It is not a measurement of brand sentiment labelling, and nobody has published the equivalent number for that task, so treat it as the best available analogy rather than your error rate. The authors note it is roughly the level at which two humans agree with each other, which makes it a good result for a benchmark and a thin foundation for a weekly trend line. If your dashboard moves four points this week, the arithmetic has no way to tell you whether four points is signal.

Position bias is sharper still. *Large Language Models are not Fair Evaluators* (Wang et al., 2023) showed that the ranking a judge produces can be flipped by changing nothing but the order in which the candidates appear in the prompt. That was shown for pairwise comparison of model responses, not for sentiment labels, so carrying it across is an inference and not a result. It is an inference worth testing, because your brand often sits in a list with rivals, and the passage order is decided by the engine. The shuffle test below is how you find out whether your own score inherits it.

Two tests that tell you whether your score is real

Neither of these needs a tool. Both take an afternoon.

The shuffle test. Take thirty answers you have already collected. Classify each one twice: once with the brands in the order the engine wrote them, once with the paragraph order reversed or the rival names swapped in. Count how many labels change. There is no published threshold for how much flipping is too much, and we have not measured one, so pick a line before you look and record it. The flip rate itself is the useful output. Publish it next to the score, internally, forever. It is the error bar.

The blind-name test. Replace every occurrence of your brand with a neutral token before classifying, then do the same for a rival. If the label moves when only the name moved, the classifier is scoring its own prior about the company rather than the passage in front of it. This test exists to catch a failure we cannot rule out from the outside: a judge model carrying a prior about a well-known brand into a label that is supposed to be about the passage. We have not measured how large that effect is, and we know of no published figure for it, which is exactly why the test is worth running on your own answers.

Run both before you present a sentiment chart to anyone who will make a budget decision from it. The same discipline applies to the presence numbers people quote more often, and the limits are the same ones described in share of voice in AI answers.

Sentiment is downstream of retrieval, so fix the source

Here is the mechanism that makes sentiment tractable rather than mysterious. An engine grounded in retrieved documents is working largely from those documents. Google's guidance on AI features, last updated 10 December 2025, says there are no special optimisations and no special structured data for appearing in AI Overviews or AI Mode, only normal indexing and snippet eligibility, and describes a query fan-out that issues multiple related searches. Google documents the retrieval, not the tone. That the tone of an answer mostly follows the tone of what got retrieved is our inference from how grounding works, and it is the assumption the rest of this section rests on.

So when a score drops, the first question is never "how do we improve sentiment". It is "which documents changed". A review roundup that re-ranked. A two year old comparison page that resurfaced. A support thread that got indexed. In the client audits we run, each one a prompt set of about 45 prompts across five engines with retrieval on and off, the sentences an engine repeats about a brand usually trace back to a small number of pages, and some of those pages are not owned by the brand. That is a pattern we notice across engagements, not a counted rate: we have not run it as a defined sample with a tally, so there is no figure here to quote back at us.

That also explains the most common false alarm. Hedged phrasing, the kind that puts you in a list of alternatives while a rival gets the definite article, often reflects thin or contradictory evidence about who you are rather than a negative opinion of you. When an engine cannot resolve your identity it tends to hedge, and a hedge classifies as lukewarm. We infer that from the answers we read rather than from anything an engine publishes. The fix is identity work, not reputation management. Access is the layer underneath that, and whether the relevant fetchers can reach you at all is worth confirming against the AI crawlers and user agents worth knowing.

What to track instead of a score

Keep a claim ledger. For each answer, record the brand-level claims the engine made in plain words, the engine, the date, and the sources it cited. Then count. Four columns, one row per claim instance.

After a few hundred rows you will have something a score cannot give you: a ranked list of the sentences the engines keep saying about you, with the pages that feed each one. Some of those sentences will be wrong or stale. Those are tickets. At least one is likely to be a real weakness your sales team already knows about, and no amount of content will move it. Say so and move on.

Three properties make the ledger better than a number. It is auditable, because every row has a quote and a URL. It is actionable, because a wrong claim maps to a page you can change. And it degrades honestly, because a sparse ledger looks sparse, whereas a score computed from eleven mentions looks exactly as confident as a score computed from eleven hundred. Our own sample audit report shows the shape of that evidence trail, claim by claim.

Keep the aggregate number if your leadership wants a trend line. Just label it with the flip rate from the shuffle test and the number of mentions behind it, and refuse to explain small movements.

When it is not worth tracking at all

When your tracked prompts return only a handful of mentions a month, aggregating them into a sentiment figure adds a decimal point and no information. We have no sourced cut-off to offer, and the honest version of the advice is to read the answers themselves while they are still few enough to read. Sentiment also earns little attention when your real problem is absence rather than tone, which is the common case early on, and the argument for waiting entirely is set out in when answer engine optimisation is not worth doing yet.

Spend the effort on sentiment when you are already named in most of the answers you care about and the question has shifted from whether you appear to what gets said. That is a real stage. It is just later than most dashboards imply.

Honest limits

This post establishes less than it may read like. The two judge-bias papers it cites measured model-response comparison in 2023, not brand sentiment labelling in 2026, so the read-across to your dashboard is argued, not demonstrated. Our statement that none of the engines exposes a sentiment field covers the public documentation we read in October 2026 for five engines, and absence from a doc page is not proof of absence. The claim-tracing pattern comes from client audits we have not counted, so it carries no rate and should not be quoted as one. We cannot separate judge drift from reality drift over long periods: when the classifier is a hosted model that gets updated, a series can move because the grader changed, and we have no way to see that from outside. And we cannot show that improving a claim in an engine's answer moves revenue, because these surfaces pass almost no usable referrer.

FAQ

Do answer engines expose a sentiment score? Not in any documentation we could find. We checked the public developer and publisher docs for five engines in October 2026 and found none. Every sentiment metric we have examined is computed by a third party, by sampling answers and running a classifier over the text.

How many runs do I need before a sentiment change is meaningful? Enough to see the spread. Run the same prompt at least five times on the same day, measure how much the label moves with nothing changed, and treat that spread as your floor. Changes smaller than the floor are not changes.

Is negative sentiment in AI answers a reputation problem or a content problem? Usually a content problem wearing reputation's clothes. Trace the claim to the pages that support it first. Genuine negative coverage needs a different response from a stale comparison table, and you cannot tell which you have without the trace.

Can I just ask the engine to be more positive about my brand? Not in the engine documentation we checked, and Google's AI features documentation says there are no special optimisations for appearing in AI Overviews or AI Mode. What moves the sentence is changing the evidence the engine retrieves, on your pages and on the third-party pages it trusts.

Sources

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track: arxiv.org/abs/2306.05685
• Wang et al., Large Language Models are not Fair Evaluators, 2023: arxiv.org/abs/2305.17926
• Google Search Central, AI features and your website, last updated 10 December 2025: developers.google.com/search/docs/appearance/ai-features