Blog · Running the work

Getting engineering to ship your AEO fixes

Published 25 September 2026 · Updated 12 October 2026 · 9 min read

Getting engineering to ship your AEO fixes
Photograph: Michaela / unsplash

The short answer: Stop sending findings and start sending tickets. The AEO fixes we see ship tend to carry four things: a command the engineer can run to see the problem, the output they should see instead, the blast radius if it stays broken, and the existing severity label the team already uses. Findings written as "we need better AI visibility" tend to sit in the backlog. Findings written as a failing check are much easier to point, and get picked up in a sprint.

The gap is usually not willingness. Engineers are not refusing to help. They are triaging a queue where some items are phrased as opinions, and an opinion is harder to prioritise than a reproducible bug.

Why the handover fails

A visibility report lands as a PDF with twenty recommendations. Marketing forwards it. The engineering lead reads three bullets, finds no reproduction steps and no definition of done, and files it under "when we have time". Nothing moves for a quarter.

Look at what the engineer was handed. "Our schema markup is incomplete" is not a defect, it is a judgement. "Improve crawlability for AI agents" names no agent, no URL and no expected response. Nobody can tell whether the work is ten minutes or ten days, so it cannot be pointed, so it never enters a sprint.

Contrast that with how a real bug arrives. Someone says: on this URL, with this header, we return a 403, and we should return a 200. That takes under a minute to confirm and the fix is usually obvious. AEO work has the same shape available to it. In the handovers we have reviewed, it is rarely used, which is why the first 90 days of AEO work are better spent building that translation habit than building a dashboard.

Sort the fixes by who actually owns them

Three buckets. Misfiling one is a common reason a fix stalls, and in the engagements we have run it is the reason we hit most often.

Edge and infrastructure. A firewall rule, a rate limit, a CDN cache header, a robots rule. These are often minutes of work sitting behind weeks of change-approval queue. They belong to whoever owns the WAF, who is frequently not the web team at all. Find that person before you write the ticket.

Template and build. Structured data in the page template, server-side rendering of the answer body, the canonical tag logic. This is normal application work, it gets pointed, and it goes in a sprint like anything else.

Content and copy. The answer paragraph, the pricing table, the corrected fact. No engineer required, and in most cases no engineer needs to be asked. In the backlogs we have been shown, a noticeable share of the queued AEO items were things marketing could have shipped in the CMS that afternoon. We have not counted this across a sample large enough to put a number on it. Clearing those out first buys you credibility for the tickets that do need code.

If you are deciding how to staff any of this, the split maps cleanly onto the argument in in-house or agency for AEO: the third bucket almost never justifies an external retainer, and the first one almost always needs someone internal with production access.

Write the ticket as a failing check

Here is the format that gets picked up. Four fields, no narrative.

Observed. One command and its output. Something like a request to your robots file with a named AI user agent, showing a 403 where a 200 belongs. Paste the real response.

Expected. The exact output that counts as fixed. A status code, a header value, a string present in the raw HTML before any JavaScript runs.

Repro. The command again, copyable, with no setup. If an engineer has to install anything to see your problem, you have lost them.

Blast radius. What stays broken until this ships, in plain terms, and limited to what you actually tested. Not "AI visibility suffers". Rather, as an example: the four user agents we fetched with were all refused on the pricing page, so none of those four can read our price. Name the agents you tried, and do not generalise to the ones you did not.

That last field is where the severity label comes from, and you should borrow the ladder engineering already uses rather than inventing a parallel one. If your team calls a customer-facing 5xx a P1, then robots.txt returning a 5xx is a P1 too, for the reason in the next section.

The failure mode that usually has no monitor

In the sites we have audited, robots.txt is treated as static config and we have not yet found one with a health check on it. That is a pattern in a handful of engagements, not a survey of the web, so read it as a prompt to check your own monitoring rather than a claim about everyone. It matters because, for any crawler that follows RFC 9309, a server error on that file does not mean "retry later", it means "stop".

RFC 9309, the robots exclusion protocol published in September 2022, is explicit: if the file is unreachable because of a server error, the crawler must assume a complete disallow until it is reachable again. That is what the standard requires of a conforming crawler. It is not a report of what each engine actually does, because most engines do not publish their robots.txt error handling at all, and we cannot see their fetch logic from outside.

Google is the one that documents its behaviour. Its robots.txt documentation, updated 31 August 2026, says crawling stops for the first 12 hours while retries continue, that for the next 30 days Google falls back to the last good copy, and that after 30 days it may proceed as though no robots file exists if the site is otherwise available, or stop crawling if the site looks unavailable. So a deploy that briefly 500s your robots file can mute you to Google for half a day, and your analytics will show nothing, because there was never a click to lose. Whether the other engines behave the same way is not something we can verify.

Two more numbers from the same Google page are worth putting into a test, and both are Google's documented behaviour rather than a rule binding on other crawlers. Google enforces a robots.txt size limit of 500 kibibytes and ignores content past it, where RFC 9309 only requires crawlers to parse at least that much. Google follows at least five redirect hops and then treats the result as a 404, where the RFC says a crawler may assume the file is unavailable after five. Teams that serve robots.txt through a redirect chain or append rules programmatically can cross both lines without noticing.

The adjacent one is rate limiting. Google's crawl budget guidance, updated 22 July 2026, states that server errors and rate-limiting signals such as HTTP 429 push the crawl limit down, while stable response times push it up. A WAF rule written to stop scrapers, applied bluntly to automated traffic, therefore has a compounding cost on that crawler: it blocks the fetch today and reduces how often anything is attempted tomorrow. Whether other engines back off the same way is undocumented. If your site sits behind Cloudflare, its AI Crawl Control documentation describes per-crawler allow and block rules and reporting on which crawlers ignore your robots directives, which turns part of this into a question with an answer rather than a guess. On other stacks you are back to reading your own access logs.

Put the assertions in CI

The durable version of all this is not a document. It is four or five tests next to your existing suite, running on every deploy.

Assert that /robots.txt returns 200, is under 500 kibibytes, and contains the allow lines you meant to publish. Assert that a fetch with each AI user agent you care about returns 200 on your three most important URLs, and keep the list of agents you tested next to the test, because that list is the limit of what the check proves. Assert that the answer paragraph appears in the raw response body, not only after hydration, since Google's JavaScript guidance, updated 4 March 2026, notes that pages are queued for rendering and that Google will not render JavaScript on pages blocked in robots.txt. Assert that your organisation JSON-LD parses and still carries the fields you depend on.

Each of those is a few lines. Once they exist, a regression becomes a red build rather than something a consultant rediscovers next quarter. That is also the moment the relationship changes: engineering is no longer absorbing requests from marketing, it is maintaining a check it owns. Our sample audit report shows the finding format these tests tend to be written from, including the evidence attached to each one.

Factual corrections are the exception. When an engine states something false about your company, a deploy on your own site will not reliably fix it, which is why getting a wrong fact about your brand corrected is a slower and messier job than anything in a sprint. Keep those items out of the engineering queue so the queue stays credible.

One aside. The naming argument around AEO vs GEO vs SEO: what the three terms actually mean rarely comes up in this conversation. In our experience engineers care less about what the programme is called than about whether the thing you sent them has a reproduction and a pass condition.

Honest limits

This post does not establish that the pattern we describe holds beyond the engagements we have run: our claims about unmonitored robots files, clogged backlogs and misfiled tickets come from a small number of audits, not from a sample anyone could call representative, and we have deliberately not attached counts to them. The timings, size limits and redirect behaviour here are Google's documented behaviour and RFC 9309's requirements, and we cannot confirm what ChatGPT, Perplexity, Claude, Copilot or Gemini crawlers do on a 5xx robots file, because none of them publish it and the behaviour is not visible from outside. A CI check proves only what it asserts against the agents you listed, so a green suite is evidence about those agents on those URLs and nothing more. Passing every check also does not mean you get cited: access is necessary and not sufficient, and the corroboration half of the work lives outside your repository.

FAQ

How much engineering time does AEO really need? Front-loaded and then small. The access and template work is concentrated in the first few weeks, and after that most months need a few hours, mainly to review schema changes and keep the checks green. If your ask is growing month over month, you are probably sending content tasks to the wrong team.

What if engineering has no capacity at all? Ship the content bucket yourself and reduce the engineering bucket to a single ticket: the robots and firewall check. That one item unblocks the rest, and it is small enough that it usually fits in whatever slack exists.

Should AEO findings go in the same backlog as product bugs? Yes, with the same labels and the same severity ladder. A separate board tends to signal that the work is optional, and optional work competes badly for sprint capacity.

Who should own the AI crawler allow list? Whoever owns the firewall, with a named reviewer from marketing. The rules change when vendors add or rename agents, so treat the list as a maintained artefact with a review date, not a one-time configuration.

Sources

All five were opened and checked on 12 October 2026.

Related reading

The first 90 days of AEO workIn-house or agency for AEOGetting a wrong fact about your brand correctedAEO vs GEO vs SEO: what the three terms actually meanWhat an AEO budget actually buysAn AEO roadmap you can defend in a planning meeting