Outdated retrieval can overturn correct answers

On 28 September 2026, Md Shamim Ahmed, Lukas Galke, Poech, and Richard Röttger posted Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers (arXiv:2609.31342). Their claim is precise: retrieval-augmented generation is often sold as a fix for stale model weights, but outdated retrieved evidence can make a model wrong on items it already answered correctly without retrieval.

That is not adversarial corpus poisoning and not fabricated counterfactuals. The documents were once correct. They become harmful when the conditions that made them valid have changed.

Question. When a retrieval stack surfaces authentic but superseded evidence, how often does that evidence overturn an otherwise-correct answer, and what metadata actually controls deference: publication date, or an explicit validity boundary?

Method. Documentation-plane reading of arXiv:2609.31342 (abstract, Sections 1–3, 5–6, and the reported tables) on 28 September 2026. No local model runs. No claim that Casinokrisa reproduced the benchmark. All counts below are the authors’ reported results.

Result: poisoning is conditional on unaided correctness

The authors build 317 source-verified knowledge reversals across medicine (87), law (100), software/API (60), and platform policy (70). Poisoning is counted only when the model is correct without retrieval and becomes wrong after one outdated document is inserted. That isolates retrieval-induced failure from errors the model already makes alone.

The asymmetry matters. Models readily follow valid evidence. They do not reliably refuse superseded evidence. The failure is selective epistemic trust, not a general inability to use retrieval.

Result: dates are not validity boundaries

On a 50-item temporal-applicability control, the authors hold the historical evidence, question, options, and instructions fixed and change only the evaluation date. Dates alone produce only modest adaptation. When the same items add an explicit statement of when the old evidence stops applying (prose or table), the larger models (Qwen2.5-72B, Llama-3.1-70B) switch to the appropriate answer almost perfectly.

Causal interventions at the evaluation-date position move answer logits strongly; matched patches at the unchanged source date do not. Attention contributes early. The same heads also support ordinary date comparison and non-temporal thresholds, which suggests a general comparison mechanism recruited for temporal applicability rather than a dedicated “time circuit.”

Result: recency re-ranking helps only when dates are trustworthy

A fixed hybrid re-ranker (semantic relevance + document year + supersession cues) reduces downstream poisoning by about 4.6–10.0 percentage points when dates are accurate. Gains shrink when dates are missing and can reverse when dates are wrong. Recency is a weak proxy for applicability.

Interpretation for crawl–index–serve and AI search

In indexing-first terms, this paper is about inventory validity, not about writing longer prompts.

1. Fresh crawl is not the same as valid evidence. A page that was correctly indexed last year can still be the wrong citation today if policy, API, or medical guidance reversed. Index coverage answers “is this URL known?” Validity answers “does this claim still govern?”
2. Publication date ≠ effective date ≠ supersession relation. Operators who sort retrieval by lastmod or by crawl timestamp are sorting a different signal from “this recommendation was withdrawn on date T.”
3. AI Overview / RAG citation quality inherits the same failure. Google’s public eligibility rule for AI features on Search is still index + snippet eligibility (AI features and your website). That is necessary, not sufficient, for timely supporting links. A superseded page that remains indexed and snippet-eligible can still be retrieved into an answer stack that lacks validity arbitration.
4. “Follow the retrieved doc” product defaults amplify poisoning. The authors’ instruction gradient shows deference pressure makes selective trust worse. Systems that hard-code “always prefer the document” without an applicability check are optimizing for compliance, not correctness over time.

Limits

• Secondary note of one arXiv preprint (28 Sep 2026); not peer-reviewed journal version at time of writing.
• No Casinokrisa replication; numbers are authors’ reported estimates and should be checked against the paper’s tables and confidence intervals before citing in secondary work.
• Poisoning rates are conditional on unaided correctness and vary by domain and model; do not collapse them into a single universal percentage.
• Causal and head-localization analyses use restricted eligible subsets and constrained answer-logit tasks; they are mechanistic evidence, not a full deployed-RAG circuit claim.
• Medical source archival was partial in the authors’ capture pass; provenance URLs remain in their release.

Takeaway

RAG that only maximizes relevance can still inject stale authority. The operational unit for retrieval systems is not “newest URL” but whether the retrieved claim still applies. Treat supersession, effective dates, and validity intervals as first-class index metadata. Evaluate retrieval in both directions: does valid evidence correct an outdated model, and does superseded evidence harm a model that was already right?

Sources

• Md Shamim Ahmed, Lukas Galke, Poech, Richard Röttger. Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers. arXiv:2609.31342, 28 September 2026.
• Google Search Central. AI features and your website.

Suggested citation

Mikhail Drozdov. "Outdated retrieval can overturn correct answers." Casinokrisa Research, 28 September 2026. https://casinokrisa.com/blog/stale-document-poisoning-retrieval-validity-2026-09

Mikhail Drozdov is an AI Search & Indexing Systems Researcher. Casinokrisa is the research platform. Person URL: https://casinokrisa.com/person/mikhail-drozdov