Blog · Third-party sources

Forums, communities and user content in AI answers

Published 8 October 2026 · Updated 12 October 2026 · 8 min read

Forums, communities and user content in AI answers
Photograph: Zoshua Colah / unsplash

The short answer: Forums and user content feed the experience half of an AI answer, the part that says what real buyers found, while documentation and reference sites supply the facts. The catch is access. Of the large public forums we checked, Reddit, Stack Overflow and Quora, all three close their robots.txt to general crawlers, so engines that respect those files reach that content through licensing and search indexes rather than by fetching it. The only user content you can influence end to end is the community you host yourself.

Two kinds of claim in one answer

Across the buying-intent prompts in our own audit runs, the shape repeats. One or two sentences state what a product is and what it costs, sourced from a vendor page, a reference site or a directory. Then comes a sentence about what using it is like, and in the answers we read that sentence is the one most likely to carry a forum or community citation. That is a pattern in our sample, not a measured rate across engines.

The split matters because it tells you which pages to work on. A support thread will rarely win the definition of your category. It is very likely to win the sentence that begins "users report" or "people who switched said". If you want the first kind of sentence, see The third-party sources AI engines keep citing. If you want the second, you are in forum territory, and the rules are different.

Those rules are also harsher. Experience claims are where the wrong sentence does real damage, because a two-year-old thread about a bug you fixed last quarter reads to an engine as current testimony unless something newer contradicts it.

The access problem nobody mentions

Here is the part that gets left out of most writing on this topic. The big community sites are closing to crawlers, and you can verify it in a terminal in under a minute.

We fetched the three files again on 12 October 2026. https://www.reddit.com/robots.txt resolves to a comment block pointing at the Public Content Policy followed by two lines: User-agent: * and Disallow: /. Stack Overflow's file is four short lines: a licence link, User-agent: *, a Content-signal: search=no, ai-train=no declaration, and the same blanket Disallow: /. Quora's file is longer and more selective. On that date 17 named agents carried a bare full-site disallow, among them GPTBot, PerplexityBot, ClaudeBot, Claude-User, Applebot-Extended and Bytespider. OAI-SearchBot is disallowed site-wide with four narrow Allow exceptions for drinking-water paths, and the wildcard group is closed apart from a handful of marketing and sign-up paths. These are point-in-time reads of three files, and any of them can change the day after you read this.

So how do those domains keep showing up in citations? Two routes do not run over the public crawler path at all. A commercial licence or index deal is one. Citing a URL that sits in a search index the engine already queries is the other, since that is a different permission from fetching the page directly. We cannot see from the outside which route produced any given citation, and we are not claiming these are the only two. What we can say is that your ability to get a community post into an answer depends on arrangements you are not party to. Posting harder does not change the pipe.

That also explains a pattern people find confusing, where a thread is quoted accurately in one assistant and invisible in another on the same day. It is not ranking. It is access, and the user agents involved are listed in The AI crawlers and user agents worth knowing.

The community you host is the one you control

If the public forums are a rented channel on someone else's terms, the forum under your own domain is the owned one. It is also, in our experience auditing them, the most consistently broken.

Four failure modes come up again and again, and all four are checkable this afternoon.

The default robots.txt was never reviewed. Discourse ships a sensible file. As served from its own instance on 12 October 2026, the wildcard group blocks /search, /badges, /g, /admin/ and RSS variants, and leaves thread bodies open. Reasonable for crawl budget. Fine for an engine too, as long as nobody later pasted a blanket disallow over it while fighting a scraper. Fetch your own community's robots.txt and read it with the question "could an engine fetch a thread body from this".

The thread needs JavaScript. Several community platforms we have audited render the first post server side and the replies in the browser. An engine that takes the raw HTML gets the question without the answer, which is the worst possible excerpt. Run curl -sL <thread-url> | grep -c "<a distinctive phrase from reply 4>". Zero means the useful content is not in the document.

The good answer is behind a login or a customer portal. This is often correct policy. Just be aware you have chosen invisibility for that content and keep a public mirror of the answers that are not sensitive.

Nothing identifies the content as a discussion. Google's structured data guidance for DiscussionForumPosting, last updated 8 September 2026, describes it as markup for forum-style sites where people collectively share first-hand perspectives. It lists three required properties on the post: an author, a datePublished, and one of text, image or video. Its content guidelines say plainly not to use the markup for content primarily authored by the publishers of the site or their agents. Marking your marketing blog as forum discussion is the kind of shortcut that gets a site ignored rather than promoted.

What actually moves, and what does not

What follows is an impression from our audit work, not a controlled test. Each audit runs a prompt set of about 45 prompts across five engines, with retrieval on and retrieval off, and we are describing the pattern we keep seeing in those runs up to 12 October 2026. It is a sample of prompts on client categories, not a census of answers, and it has no held-out control, so read the three claims below as directional.

Recency beats volume on experience claims. A thread from this quarter that names a version number and describes an outcome gets pulled ahead of a popular thread from three years ago on the same subject, because the newer one is the only one that can answer a question about the current product.

Specificity beats enthusiasm. "It is great" is unusable to a model. "We moved 400 GB and the import took nine hours, mostly on index rebuild" is a quotable sentence with a fact in it, and facts are what get lifted.

Upvote count appears to matter much less than people assume. In our runs the quoted passage is often not the top-voted one in the thread, which is what you would expect if selection is matching text to a question rather than reading a scoreboard. We have not tested this against vote counts at scale, and we cannot rule out that score matters indirectly, through which threads reach an index in the first place.

And the thing we will not do: manufacture the conversation. Beyond the ethics, the major platforms prohibit coordinated inauthentic posting in their own rules, which is the same Public Content Policy Reddit's robots.txt points at, and a thread removed for manipulation takes its citations with it. We have not measured detection rates and are not claiming every such attempt is caught. It is a method risk, not a tactic.

Honest limits

We cannot tell you, from the outside, which engine holds which licence, because those terms are not public and they change. We also cannot separate a citation that arrived through a partnership from one that arrived through an index, since the answer shows a URL and nothing about the route. The robots.txt readings above are four files on one day, 12 October 2026, and a closed file tells you what a publisher asked for, not what a given crawler did with the request, which we have no way to observe from outside. The three claims in the section above come from our own audit prompt sets, roughly 45 prompts per audit across five engines, which is a sample of client categories rather than a census, with no control group and no counted rates, so a careful reader should treat them as what we keep seeing rather than as measured effects.

On timing, offsite work is the slowest layer. Access fixes on your own community can land within a crawl cycle. Anything that depends on other people writing about you runs on their schedule, as covered in How long AEO takes to show up in answers.

And none of this replaces structured review data, which behaves differently and has its own distribution. That is a separate job, described in Review sites, G2 and what AI repeats about your product.

If you want the shape of how we write these findings up, the sample audit report shows an access finding with its evidence attached, which is the same form a community finding takes.

FAQ

Should we block AI crawlers from our community to protect it? It is a business decision with a cost attached. Blocking removes those threads from the answers those systems write, which for a support community usually means losing the exact sentences you would want quoted. A paywalled archive is a different calculation.

Does posting on a public forum help our AI visibility? It can help the answer be correct, which is worth something on its own. It is not a reliable distribution channel, because whether that platform's content reaches a given engine is set by agreements you are not party to.

Is DiscussionForumPosting markup worth adding? Worth adding on a genuine community with real user posts, authors and dates. Not on anything your marketing team wrote. Google's content guidelines for the type say not to use it for content primarily authored by the site's publishers or their agents.

How do we stop an old thread giving wrong information about us? Answer it rather than chase it. Post a dated correction in the same thread, publish a current page that states the fact plainly, and let the newer, more specific text compete. In the cases we have worked on, deletion requests mostly produced a dead URL and a summary that outlived it.

Sources

Related reading

The third-party sources AI engines keep citingReview sites, G2 and what AI repeats about your productThe AI crawlers and user agents worth knowingHow long AEO takes to show up in answersHow to get a wrong fact about your brand corrected in AI answersLinkedIn as a cited source in AI answers