Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering

Technical Summary: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering

Problem Statement
Large Language Models (LLMs) are increasingly adopted for legal research, yet their static training data and fixed knowledge cutoffs create a fundamental misalignment with the evolving nature of statutory law. This paper identifies two specific temporal failure modes in Legal Question Answering (QA):

1. Post-Cutoff Staleness: Models apply superseded legal rules to fact patterns occurring after legislative amendments, relying on outdated parametric knowledge.
2. Recency Bias: Models incorrectly prioritize newer legal provisions even when the fact pattern explicitly requires the application of a historical, superseded version of the law.

The authors note that while legal professionals increasingly rely on AI for research (with usage rising from 14% to 26% in 2024–2025), these tools often lack awareness of their limited temporal robustness. Unlike general knowledge, legal correctness is strictly dependent on the specific version of a provision and its temporal applicability.

Methodology
To diagnose and mitigate these issues, the authors constructed a novel benchmark and evaluated five LLMs across four inference settings.

• Dataset Construction: The authors created a dataset of 312 expert-validated, time-sensitive German statutory QA pairs. The data was synthetically generated using historical consolidated versions of six German statutes (BGB, StPO, AO, EStG, BauGB, HGB) sourced from buzer.de. The dataset is divided into three categories:
  • Post-Cutoff Amendment Questions (115 items): Designed to test staleness by presenting fact dates after the model's training cutoff, requiring knowledge of legislative changes that occurred post-cutoff.
  • Pre-Amendment Questions (113 items): Fact dates fall before an amendment, requiring the model to apply an older version of the law despite newer versions being available in its context or training.
  • Multi-Provision Pre-Amendment Questions (84 items): Complex scenarios requiring the synthesis of multiple provisions, where at least one must be applied in its historical version.
• Models Evaluated:
  • Post-Cutoff tasks: ChatGPT-5.1 and Claude Sonnet 3.7 (knowledge cutoffs precede the evaluation date).
  • Pre-Amendment tasks: ChatGPT-5.2, Claude Opus 4.5, and DeepSeek v3.2 (state-of-the-art models at the time of the experiment).
• Inference Settings:
  • Vanilla: Reliance solely on internal parametric knowledge.
  • Web Search: Models use native or external web search tools to retrieve current information.
  • RAG-kNN: Retrieval-Augmented Generation using k-nearest neighbor search on text embeddings, filtered by an extracted "as-of" date to ensure only valid paragraph versions are retrieved.
  • RAG-ToC: A retrieval strategy using the statute's table of contents to select relevant paragraphs, followed by the same temporal filtering and retrieval as RAG-kNN.
• Evaluation: An LLM-as-a-judge (Gemini 3 Flash Preview), validated against human expert ratings, scored answers on four criteria: Outcome Correctness, Legal Reasoning Correctness, Legal Basis Correctness, and Version Application Correctness.

Key Results

• Severe Staleness in Vanilla Models: In Post-Cutoff tasks, Vanilla models exhibited severe degradation. For example, Claude Sonnet 3.1 achieved outcome correctness scores as low as 24.35%, with legal reasoning correctness dropping to 0%. Models confidently applied superseded provisions rather than signaling uncertainty.
• Effectiveness of Temporal RAG: Both RAG-kNN and RAG-ToC substantially improved performance across all question types. In Post-Cutoff scenarios, RAG methods achieved outcome correctness scores between 78% and 94%, significantly outperforming Vanilla and Web search baselines. The two RAG variants showed no statistically significant difference in performance, suggesting that the temporal filtering is the critical factor rather than the retrieval mechanism (embeddings vs. table of contents).
• Instability of Web Search: While Web search improved performance over Vanilla in some cases, it yielded unstable gains and exhibited marked recency bias. In historically anchored tasks, Web-enabled models performed significantly better on recent questions than older ones, often retrieving current statutes even when historical versions were required.
• Multi-Provision Complexity: Performance generally declined slightly for Multi-Provision questions compared to single-provision tasks, particularly in Vanilla and Web settings. However, RAG methods remained robust, indicating that once the correct temporal context is provided, current LLMs can synthesize multiple provisions effectively.
• Recency Bias: Significant performance divergences based on the age of the question were observed exclusively in the Web search condition, suggesting that recency bias in this context is driven by retrieval pipelines favoring "fresh" content rather than inherent parametric biases in the models.

Significance and Claims
The paper claims that reliable legal QA requires treating temporal validity as a hard constraint. The authors argue that:

1. Static training is insufficient: The mismatch between static model parameters and dynamic statutory law creates a critical failure point for legal AI, posing significant liability risks when models confidently apply outdated laws.
2. RAG with Temporal Filtering is the solution: Simply retrieving "relevant" semantic matches is inadequate. Systems must explicitly filter retrieval spaces based on the "as-of" date of the legal fact pattern to ensure the model reasons within the correct legal regime.
3. Web search is not a panacea: Generic web search tools are prone to recency bias and cannot reliably replace curated, versioned statutory databases for temporally sensitive legal tasks.

The work fills a gap in existing benchmarks (which often lack civil-law focus or fail to test post-cutoff staleness) and provides empirical evidence that while retrieval can mitigate data staleness, the reasoning capabilities of LLMs regarding complex, multi-provision historical synthesis still face modest ceilings. The authors conclude that future research should explore how these findings generalize to common law jurisdictions and further investigate the mechanisms of recency bias.