Blog · Access and retrieval

Sitemaps, feeds and freshness signals that matter to AI

Published 4 May 2026 · Updated 8 October 2026 · 8 min read

Sitemaps, feeds and freshness signals that matter to AI
Photograph: camera obscura / unsplash

The short answer: Three things are worth your effort: an accurate lastmod in your XML sitemap, a page whose HTTP headers and visible date agree with it, and a push channel such as a feed or IndexNow for anything that changes faster than a crawl cycle. Google's sitemap documentation says it ignores <priority> and <changefreq> outright, and it is the one engine that documents how it treats these fields at all. Everything else in the freshness story is your server telling the truth about when a page actually changed.

Most of the sitemaps we open in audits get this wrong in a specific and boring way. They are technically valid and informationally worthless.

What is still read, and what is thrown away

Google's own sitemap documentation is blunt about two of the four optional elements: "Google ignores <priority> and <changefreq> values." They have been dead weight for years, and most generators we have opened still emit them by default.

lastmod is different. Google says it uses the value "if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate." Read that clause closely, because it describes a test your site either passes or fails as a whole. The engine is not evaluating one URL. It is deciding whether your sitemap, as a source, can be trusted to say when something changed.

Fail that test and, on Google's own account, you do not get a penalty. You get silence: the value stops being used. What the other engines do with an untrustworthy lastmod is not documented anywhere we can point you to, so treat the rest of this as reasoning from one published policy rather than from all of them.

The limits are worth knowing too: one sitemap file caps at 50MB uncompressed or 50,000 URLs, whichever comes first, and you chain past that with a sitemap index. A site of 200,000 pages needs at least four files.

The failure mode nobody looks for

Here is the pattern we see most often in audit work. A static site generator or a CMS rebuild stamps every URL in the sitemap with the time of the deploy. Change one typo on one page, ship, and 12,000 URLs now claim to have been modified this morning.

That sitemap is not lying on purpose. It is reporting build time instead of content time, and the effect is the same: the signal carries no information, so it gets discounted.

The check takes one command and no tooling:

`` curl -s example.com/sitemap.xml \ | grep -o '<lastmod>[^<]*' | cut -d'>' -f2 | cut -dT -f1 \ | sort | uniq -c | sort -rn | head ``

If the top line accounts for most of your URLs and matches your last deploy, your lastmod is a deploy stamp. Fix it at the source: the date should come from the content record in your database or the git commit that last touched that page's body, not from the build. A second check is cheaper still. Pick any page you have not edited in a year and compare its sitemap date to what you remember. One mismatch is a bug. A pattern of them is a discounted sitemap.

Three dates, one answer

A page carries its modification date in at least three places, and they are produced by different parts of your stack, which is why they drift.

The sitemap lastmod comes from your sitemap generator. The dateModified in your Article JSON-LD comes from your template. The Last-Modified HTTP header and the ETag come from your origin server or your CDN, and on a server-rendered page they are very often absent entirely.

That last one matters more than it looks. Crawlers send conditional requests, an If-Modified-Since or If-None-Match on a URL they already hold. A 200 only says the request succeeded and here is the body; it asserts nothing about whether the page changed. A server that answers 200 to every conditional request has given the fetcher no cheap way to tell old from new, so a fetcher that wanted to skip unchanged pages cannot. A server that can answer 304 Not Modified costs you nothing and buys crawl capacity for the pages that did change. Check yours with a HEAD request and look for whether Last-Modified is even present. On several of the server-rendered and edge-rendered stacks we have checked it is not, and we have not counted how common that is across the web.

When the three disagree, you are asking a reader or a crawler to pick one, and nothing published tells you which one gets picked. The visible "Updated" line on the page is the fourth date, and it is the one a human reader and a summarising model both see. Keep all four in sync or stop publishing dates at all. The middle option, a fresh-looking date on stale content, is the one we would avoid, because it is exactly the comparison Google's documentation says it makes.

None of this helps if the fetch never happens. Whether a given agent can reach and render the page at all is the prior question, and can AI crawlers actually read your site covers the access layer underneath everything here. If you have chosen to restrict some of those agents, read should you block AI crawlers first, because a blocked crawler makes your freshness work invisible by construction.

Feeds: a different contract

A sitemap is a pull channel. You publish a file, somebody fetches it eventually, and the latency between your change and their knowledge of it is out of your hands.

Feeds invert that. Google's documentation says plainly that if your CMS generates an RSS or Atom feed you can submit the feed's URL as a sitemap, and a feed has a property a sitemap does not: it is small, ordered by recency, and cheap to poll often. For a publication shipping several items a day, a short feed alongside the full sitemap is a better description of what is new than a 40,000-line file where the new rows are scattered.

IndexNow is the push version. One HTTP POST notifies participating engines that a URL changed, up to 10,000 URLs per request, authenticated by a text file hosted on the host you are submitting for, named for the key it contains. The key runs 8 to 128 characters. The specification calls them hexadecimal in one sentence and then permits lower case, upper case, digits and hyphens in the next, so read the character list rather than the adjective. The endpoint list published by the protocol names Bing, Yandex, Naver, Seznam, Amazon and Yep. Google is not on it. That list is the whole of what IndexNow buys you. Whether any given assistant reads one of those indexes is a separate question that the protocol's documentation does not answer, and we cannot see inside an assistant's retrieval step to check.

Commerce has the strictest version of this contract. The product feed an assistant reads is pushed by the merchant rather than crawled, and the fields are fixed: OpenAI's feed specification requires nine on every row of basic product data, including item_id, url, availability and price, and tells merchants to keep current price and availability up to date. It does not publish a cadence, and neither will we, but the reason to be quick is obvious: a wrong price in an answer is a wrong price at checkout. Product feeds for AI shopping surfaces goes into what that costs to operate, and agent payments and the trust signals in the way picks up where the answer ends and a transaction begins.

A refresh that counts as a refresh

Changing a date and nothing else is not a refresh, and it is detectable by anyone holding a previous copy of the page. Diffing is cheap. Trust, once discounted, is slow to come back.

The rule we apply in our own audit recommendations, which is an editorial judgement and not a measured result: a refresh is worth doing when at least one of these is true for that page. A fact or figure in it is now wrong. The question it answers has changed shape. Something has shipped since that the page does not mention. If none of those apply, leave the date alone and spend the hour on a page where one does.

Order the work by what the engines actually fetch, not by what your CMS lists first. The self-check includes the sitemap and header tests above if you would rather run the first pass yourself.

Honest limits

Almost everything documented above about how lastmod is treated comes from one engine's published guidance, Google's. We have not found equivalent published statements from the other engines, and the absence of a statement is not evidence that they behave the same way, or that they behave differently. The retrieval step that picks passages for an answer is not documented at that level by anyone, so nothing here tells you how recency is weighed once a page is already in an index.

The observations we attribute to ourselves are from audit work, not from a survey: which sitemaps stamp build time, which stacks omit Last-Modified, and the impression that price and availability questions reward recency while definitions do not. Those come from the prompt runs and crawls inside client audits, which are samples chosen by who hired us, and we have not counted them or held the sample constant long enough to publish a figure. Where you see a number in this post it is from a cited specification, not from us.

FAQ

Does changefreq or priority do anything? Not for Google, whose sitemap documentation states it ignores both. No other engine we could find publishes a position either way. They cost nothing to leave in and they buy nothing. If you are rewriting your generator anyway, drop them and spend the effort on accurate lastmod values.

Should every page have a visible updated date? Only if it is true and it matches your markup and your server headers. A date that contradicts the page's content is worse than no date, because it is a check an engine or a reader can run and watch you fail.

Is IndexNow worth implementing? It depends which engines matter to you. It reaches the engines that publish an IndexNow endpoint, a list that includes Bing, Yandex, Naver, Seznam, Amazon and Yep but not Google. For a news or commerce site the cost is one POST per change, so it is usually worth it. For a 40-page brochure site it is not.

How often should a sitemap regenerate? As often as content changes, with dates sourced from the content and not the build. A sitemap regenerated hourly that stamps everything with the build time is less useful than one regenerated weekly with honest per-page dates.

Sources

Related reading

Can AI crawlers actually read your siteShould you block AI crawlersProduct feeds for AI shopping surfacesAgent payments and the trust signals in the waySlow pages get dropped before they get readThe AI crawlers and user agents worth knowing