Blog · Data and trends

What changed in AI search this year

Published 2 October 2026 · Updated 12 October 2026 · 8 min read

What changed in AI search this year
Photograph: Paico Oficial / unsplash

The short answer: Three things changed in 2026, and none of them is a new ranking factor. At OpenAI and Google, access rules became documented and machine checkable, so for those two you can verify who is allowed to fetch you instead of guessing. Advertising moved inside the answer surface, with its own separate crawler. And the opt-out question left one company's robots.txt convention for a standards body. The work moved from speculation to checking.

Most year-in-review pieces on this topic list adoption numbers and stop. Adoption is real, but it is not a change you can act on this week. The changes below are.

Access stopped being folklore

A year ago, deciding which bots to allow meant reading forum threads. Two of the engines we work with most now publish both the agent strings and the address ranges: OpenAI and Google. Those two are the extent of it, and we do not claim more.

OpenAI's crawler documentation names four agents with four different jobs. OAI-SearchBot surfaces pages in ChatGPT's search answers. GPTBot collects content for model training. ChatGPT-User handles a fetch that a person triggered in a conversation. OAI-AdsBot checks pages submitted as advertisements. Each one has a published JSON file of its IP ranges. We fetched the search crawler's file on 12 October 2026 and counted 39 IPv4 prefixes in it, against a creationTime of 2 January 2026 inside the file. That is one fetch of one file on one day. OpenAI edits it, so count it yourself rather than quoting ours.

Two consequences fall out of that, and both of them are things teams get wrong.

The first, for OpenAI specifically, is that blocking training does not remove you from search answers, and blocking search answers does not stop training. OpenAI's documentation says the settings are independent, because these are different agents reading different robots.txt lines. Other engines may or may not separate the two, and you should read each one's own documentation rather than carrying this across. We have seen sites disallow GPTBot in a panic, then complain months later that they are invisible in ChatGPT, when the rule they needed to look at was a firewall pattern catching the search crawler instead.

The second is timing. OpenAI's documentation says that, for search results, it can take about 24 hours from a robots.txt update for its systems to adjust. That figure is OpenAI's and covers OpenAI. If somebody unblocks a crawler at eleven in the morning and declares the fix a failure by three in the afternoon, they have measured nothing at all.

The check worth running: pull a week of access logs, grep for the agent tokens, then resolve the requesting addresses against the published prefix files. User agent strings are trivial to forge. Address ranges are harder, so for the engines that publish them this is the strongest check available to you. For an engine that publishes no ranges, you cannot verify the fetch at all, and the honest position is to say so.

Advertising arrived inside the answer

The quiet item in that crawler list is the ads agent. OpenAI's documentation says it visits only pages submitted as ads, and that what it collects is not used to train foundation models. So on that one surface, paid placement now has its own access path, separate from the one that earns an organic citation. We are describing ChatGPT here, not the category.

That matters operationally before it matters strategically. In the audits we run, a recurring finding is an edge configuration that blocks anything whose user agent contains the substring "bot", usually a rule nobody has reopened in years. We cannot tell you how common that is across the web, only that we keep finding it. A page can pass every organic check and still fail an ad review for the same reason a page fails a crawl, and the two failures show up in completely different dashboards.

It also sharpens an old distinction. Paid presence on a surface you do not own behaves like every other rented channel: it works while you pay, and it stops when you stop. The difference between rented visibility vs owned visibility in AI search is not a philosophical one this year. It is now two different crawlers.

The opt-out question moved to a standards body

Through 2024 and 2025, telling an AI system what it could do with your content meant copying a vendor-specific line into robots.txt and hoping the next vendor used the same name. That is being replaced.

The IETF has a working group on AI preferences. Its vocabulary draft reached revision 08 on 14 September 2026, and the companion document on attaching those preferences to content reached revision 05 on 19 August 2026. The vocabulary separates three uses: train-ai, for changing a model's parameters; ai-use, for feeding your content to a model as input the user did not supply; and search, for selecting content and sending people to it. Each takes a value of y or n.

Drafts are not law, and an Internet-Draft can die quietly. The useful part is the shape of the question, which is now settled even if the syntax is not. You no longer get one answer. You get three, and they are genuinely different business decisions. A trade publisher may reasonably say no to training, no to grounding, and yes to search. A software company selling to developers may want yes to all three, because being quoted is the distribution.

Decide those three answers now, in a sentence each, and write them down. When the attachment mechanism stabilises, implementation becomes a config change rather than a quarter of internal argument.

Google ruled out the file shortcut

A lot of 2026 budget went into publishing special files for AI systems to read. Google's documentation on AI features, last updated 10 December 2025, addresses that directly: there are no additional requirements to appear in AI Overviews or AI Mode, and site owners do not need to create new machine readable files, AI text files, or markup to qualify.

The same page describes query fan-out, where one question is expanded into several related searches across subtopics before anything is written. That mechanism explains a reporting pattern that confuses people. Your page can be pulled into an answer about a question you never wrote for, because it answered a subquery the system generated on its way to the real one. It also explains why chasing a single phrase is a poor use of time, and why why AI search is not a ranking game is still the right frame. Fan-out also widens the field, which is good news if your coverage is deep and bad news if your site has one strong page and nine thin ones.

What did not change, and it is the expensive part

Measurement. In our own audit reruns, the same prompt put to the same engine hours apart can come back leaning on different sources, which is what you would expect from answers generated per request rather than read from a fixed list. Any test of a content edit without an untouched comparison page is weak, because a citation count that rises over a fortnight when every page on the site also rose tells you about the fortnight, not about the edit. We have watched that confuse careful people more than once.

Buyer behaviour shifted gradually rather than suddenly. The direction is consistent in the public datasets we have read: more commercial questions reaching assistants, more of the comparison work happening before a site visit. We have not read everything published, so read that as a direction we keep seeing, not a settled finding. The precise share depends on who classified the conversations and how, so treat single vendor percentages as estimates. AI search statistics worth quoting covers which of those numbers survive a second look, and how buyers are actually using AI search covers the behaviour underneath them. If you want to check your own position before reading anybody's figures, the self-check is ten tests you can run this afternoon with no tooling.

What we are still measuring

This post establishes what three sets of documents say, not what the engines as a group do. The crawler and timing detail is OpenAI's, the no-new-files and fan-out detail is Google's, and the three way vocabulary is an Internet-Draft that can still change or die; nothing here should be read as a rule the whole category follows. Our prefix count is one fetch of one file on 12 October 2026, and the file changes. We cannot tell you how much of an answer's source selection comes from a live fetch versus a cached index, because none of the engine documentation we have read publishes that split, and we have no clean attribution from a mention to revenue, because most of these surfaces send a referrer you cannot act on. The patterns we report from our own audits are what we keep running into in client work, which is a sample of the sites that hire us and not a sample of the web.

FAQ

Is there a new ranking factor for AI search this year? No. Google's own guidance says there are no additional requirements beyond standard indexability and snippet eligibility. The changes of 2026 are about access, advertising and permissions, not a new score to optimise.

Should I block AI crawlers? It depends which one and what you sell. Blocking training and blocking search are separate rules for separate agents, and blocking the search agent removes you from answers people are already asking. Treat it as three decisions, not one switch.

Does llms.txt do anything? Google's documentation says no new machine readable files are needed for its AI features. We have not found published documentation from another major engine committing to it, which is not the same as knowing none of them reads it. It costs little to publish and it is not a substitute for pages a crawler can fetch and parse.

How fast will a robots.txt fix show up? Allow about a day for the rule to be picked up, then a crawl cycle on top of that before anything changes in an answer. Measuring the same afternoon will only produce noise.

Sources

Related reading

AI search statistics worth quotingHow buyers are actually using AI searchWhy AI search is not a ranking gameRented visibility vs owned visibility in AI searchZero click, and what replaces the click, checked in your logs