Evaluating Temporal Consistency in Multi-Turn Language Models

Technical Summary: Evaluating Temporal Consistency in Multi-Turn Language Models

1. Problem Statement

While Large Language Models (LLMs) demonstrate high factual accuracy in single-turn settings, they often struggle with temporal scope stability in multi-turn interactions. In human conversation, once a temporal frame (e.g., "in 2010") is established, it is implicitly assumed to persist until explicitly changed.

Current LLMs frequently suffer from temporal drift: they fail to maintain this implicit context and instead revert to "present-day" assumptions. This is not a failure of factual knowledge (the model often knows the historical fact) but a failure of contextual coherence—the inability to bind correct knowledge to a specific, evolving temporal reference point across a dialogue.

2. Methodology

The authors introduce ChronoScope, a large-scale diagnostic benchmark designed to isolate and measure this specific failure mode.

• Dataset Construction:

  • Scale: Over 1.46 million deterministically generated question chains.
  • Source: Grounded in the Wikidata knowledge graph, ensuring precise temporal annotations (start/end times, point-in-time qualifiers).
  • Mechanism: The pipeline materializes "truth tables" from Wikidata snapshots and converts them into natural language using property-specific templates. This ensures the benchmark is reproducible and free from human or model-generated hallucinations.
  • Chain Families: The dataset is organized into 11 families that test different reasoning patterns, such as Carryover (implicit persistence), Scope Switch (explicit override), Cross-Entity Transfer (applying time to a new subject), and Temporal Narratives (tracking changes over a trajectory).

• Evaluation Settings: To disentangle different causes of failure, the authors use three settings:

  1. Gold Context: Prior turns are replaced with correct (gold) answers. This isolates whether the model can reason temporally if the dialogue state is perfect.
  2. Self-Conditioned: The model relies on its own previous answers. This captures error propagation and how a single mistake "poisons" the rest of the conversation.
  3. Questions Only: No prior context is provided. This serves as a baseline to see if the model can solve the query without any conversational memory.

• Key Metrics:

  • Acc@1 / Final@1: Turn-level and final-turn accuracy.
  • Strict-Chain Acc@1: The fraction of chains where every turn is correct.
  • Temporal Drift: A specialized metric measuring how often a model provides a factually correct "present-day" answer instead of the historically correct one.

3. Key Contributions

1. Formalization: Defines "temporal scope stability" as a measurable property of multi-turn LLM behavior.
2. ChronoScope Benchmark: A massive, deterministic, and multi-turn dataset that moves beyond the single-turn temporal QA limitations of prior work.
3. Diagnostic Insights: Identifies "present-day bias" as a dominant error mode, where models prioritize pre-trained recent knowledge over established conversational context.
4. Open Science: The dataset and evaluation suite are made publicly available.

4. Key Results

• The Collapse of Consistency: In the Self-Conditioned setting, strict chain accuracy collapses to near zero across all models. This indicates that temporal errors are cumulative; once a model loses the temporal thread, it rarely recovers.
• The Scale-Drift Paradox: Larger, more capable models (like GPT-4 or Gemini) often exhibit higher Drift rates. The authors suggest that stronger memorization of present-day facts during pre-training makes these models more resistant to implicit temporal overrides.
• Knowledge vs. Reasoning: Models often show higher Final@1 accuracy than Acc@1 under Gold Context, proving they can eventually find the right time frame, but they struggle to maintain it consistently throughout the interaction.
• Interaction Length: As the number of turns increases, temporal stability degrades significantly, especially when the model is conditioning on its own previous (potentially incorrect) outputs.

5. Significance

This research highlights a critical gap in the development of interactive AI. For applications in legal reasoning, historical research, education, and policy analysis, a model that "drifts" to the present is not just slightly inaccurate—it is fundamentally unreliable.

The paper concludes that improving factual density (more knowledge) is insufficient for interactive intelligence. Instead, future research must focus on temporal state tracking, objectives that reward context persistence, and decoding strategies that prevent the model from defaulting to its most recent pre-trained priors.