How often do Answers Change? Estimating Recency Requirements in Question Answering

1. Problem Statement

Large Language Models (LLMs) frequently fail to handle time-sensitive questions accurately because they rely on static training data. Current benchmarks for Temporal Question Answering (TQA) suffer from two primary limitations:

• Binary Freshness: Existing datasets (e.g., FreshQA, RealTimeQA) typically treat temporal validity as a binary distinction (current vs. outdated), failing to model the frequency of answer changes (e.g., hourly vs. annually).
• Lack of Context Dependence: They do not distinguish between questions where the update frequency is constant (stationary) and those where the frequency varies based on the temporal context (non-stationary).
• Evaluation Gap: There is no principled framework to estimate a question's "recency demand" or to evaluate whether an LLM can dynamically adapt its reasoning when the context changes.

2. Methodology

A. The Recency–Stationarity Taxonomy

The authors propose a two-dimensional taxonomy to characterize temporal sensitivity:

1. Recency: Measures the expected time until an answer changes. This is defined across 12 discrete classes, ranging from "An-Hour" (highly volatile) to "Never" (permanent facts).
2. Stationarity: Measures whether the recency behavior is consistent over time.
   • Stationary: The update frequency is invariant (e.g., "Who is the US President?" changes every 4 years regardless of context).
   • Non-Stationary: The update frequency is context-dependent (e.g., "Who is leading the Olympic medal table?" changes hourly during the games but never in between).

B. The RECENCYQA Dataset Construction

The authors constructed RECENCYQA, a dataset of 4,031 open-domain questions annotated with recency and stationarity labels. The construction pipeline involved:

• Question Sampling: Sourced from existing datasets (FreshQA, PATQA, SituatedQA) and generated via LLaMA 3.3 (70B) based on real-world event descriptions.
• Label Generation:
  • Recency: LLaMA 3.3 (70B) was prompted 13 times per question to generate a distribution over the 12 recency classes. The majority class was assigned as the primary label.
  • Stationarity: Three state-of-the-art LLMs (GPT-5.2, Gemini 3 Flash, Claude Sonnet 4.5) voted on stationarity. This was cross-validated against the empirical recency distribution (e.g., if a question showed high variance in recency labels, it was forced to be non-stationary).
• Temporal Context Generation: For each question, specific temporal contexts were generated to ground the recency interpretation. These were rigorously verified by re-prompting the LLM to ensure the context consistently induced the intended recency label.

C. Experimental Setup

The study evaluated six LLMs (Qwen 2.5, LLaMA 3, Mistral, Gemma 3, Apertus) across three prompting strategies (Zero-shot, Few-shot, Chain-of-Thought) on three tasks:

1. Recency Classification: Predicting the update frequency without context.
2. Context Sensitivity: Comparing performance with and without explicit temporal context.
3. Dynamic Recency Transition (RL1 → RL2): Testing if a model can adapt its answer when the context shifts from one recency state (RL1) to another (RL2) for the same question.

3. Key Contributions

1. Novel Taxonomy: Introduced the first framework to jointly model answer volatility (Recency) and the stability of that volatility (Stationarity).
2. RECENCYQA Dataset: Released the first dataset of open-domain questions with granular recency distributions, stationarity labels, and verified temporal contexts.
3. Comprehensive Benchmarking: Provided a multi-faceted evaluation protocol that goes beyond simple "freshness" checks to analyze how models handle dynamic temporal shifts.

4. Results and Findings

A. Dataset Quality

• Human Evaluation: Human annotators achieved 76% accuracy on strict recency labeling and 78% on stationarity, confirming the reliability of the LLM-generated labels.
• Clarity: Questions were rated highly clear (4.66/5), with moderate difficulty in labeling (2.26/5).

B. Model Performance on Recency Classification

• General Difficulty: Recency classification is challenging for all models, with strict accuracy ranging from 24% to 52%.
• Best Performer: Gemma 3 (27B) using Few-shot prompting achieved the highest accuracy (52.05%) and F1 score (51.45%).
• Tolerance: Tolerant metrics (allowing adjacent class predictions) were significantly higher (>88%), suggesting models capture coarse temporal ordering but struggle with fine-grained distinctions.

C. Impact of Context

• Non-Stationary Questions: Providing temporal context significantly improved performance (gains >40% in some cases), indicating models can leverage cues when the update frequency is context-dependent.
• Stationary Questions: Surprisingly, adding context often degraded performance or had no effect. Models appeared to assign undue weight to temporal cues even when the underlying fact was stable, suggesting a lack of "temporal selectivity."

D. Dynamic Adaptation (Transition Task)

• Static Reasoning: Models struggled to adapt to changing contexts. For Gemma 3 (27B), while per-context accuracy was ~46%, the Transition Accuracy (correctly switching answers between contexts) was only 14.8%.
• Conclusion: Current LLMs exhibit predominantly static temporal reasoning and fail to dynamically revise predictions when the recency requirement shifts.

5. Significance

This work fundamentally shifts the paradigm of Temporal QA evaluation from a binary "old vs. new" check to a fine-grained, continuous analysis of temporal dynamics.

• Diagnostic Power: It reveals that LLMs are not just "outdated" but lack the mechanism to determine when to retrieve information versus relying on internal knowledge.
• Future Systems: The dataset and taxonomy provide a foundation for developing recency-aware retrieval systems, temporal confidence calibration, and dynamic multi-hop reasoning agents that can distinguish between stable facts and volatile, context-dependent information.
• Research Direction: It highlights the need for models that can explicitly reason about the rate of change of information rather than just the timestamp of the information.