Temporal Fact Conflicts in LLMs: Reproducibility Insights from Unifying DYNAMICQA and MULAN

1. Problem Statement

Large Language Models (LLMs) encode factual knowledge in their parameters, leading to knowledge conflicts when training data contains outdated or evolving information. A specific subset of these conflicts involves temporal facts (e.g., "Who is the current US President?"), which change over time.

Two prominent studies have reached contradictory conclusions regarding how LLMs handle these conflicts when provided with external context:

• DYNAMICQA [12]: Concludes that temporal facts are more resistant to change (harder to update) than static facts. It uses a Question Answering (QA) format and measures "Coherent Persuasion" (distributional shift).
• MULAN [5]: Concludes that temporal facts are easier to update than static facts. It uses a fact-completion format and measures "Update Success Rate" (binary success of changing a memorized answer).

The paper identifies that these conflicting results may stem from differences in dataset design (Wikipedia edit counts vs. Wikidata relations), evaluation metrics (distributional shift vs. binary success), and context construction (natural passages vs. programmatically generated statements). The authors aim to resolve this discrepancy through a systematic reproducibility study.

2. Methodology

The authors conducted a three-phase reproducibility study to isolate the sources of disagreement:

A. Reproduction of Original Findings

• Datasets: Re-evaluated DYNAMICQA (5,689 samples, QA format) and MULAN (49,142 samples, fact-completion format).
• Models: Reproduced experiments using the original 7B models (Llama-2, Mistral, Qwen2, Falcon) and extended them to the Gemma 3 family (1B, 4B, 12B) to analyze model size effects.
• Metrics:
  • DYNAMICQA: Accuracy, Coherent Persuasion (CP) score, Semantic Entropy, and percentages of "stubborn" vs. "persuaded" instances.
  • MULAN: F1 score, confidence (first-token probability), and Update Success Rate.
• Code Correction: The authors identified and fixed a bug in the original DYNAMICQA evaluation code where "without-context" predictions were incorrectly compared against substitute answers rather than ground truth.

B. Cross-Dataset Evaluation (Methodological Unification)

To determine if the conclusions were driven by the dataset or the metric, the authors swapped methodologies:

1. DYNAMICQA on MULAN: Applied DYNAMICQA's CP score and QA format to MULAN data. Crucially, they replaced MULAN's simple programmatically generated prompts with LLM-generated natural language contexts (using Llama-3.3-70B) to match DYNAMICQA's style.
2. MULAN on DYNAMICQA: Applied MULAN's Update Success metric and fact-completion format to DYNAMICQA data.

C. Model Size Analysis

The study extended the evaluation to Gemma 3 models (1B, 4B, 12B) to investigate whether the findings on temporal fact updateability hold across different parameter scales, controlling for architectural variables.

3. Key Contributions

1. Systematic Reproducibility: Confirmed that both original studies' findings are reproducible under their specific original conditions (DYNAMICQA finds temporal facts hard to update; MULAN finds them easy).
2. Methodological Isolation: Demonstrated that the conflicting conclusions are highly dependent on dataset design and evaluation metrics rather than an inherent property of temporal facts.
3. Synthetic Context Generation: Introduced a method to generate realistic, natural language contexts for fact-completion tasks, bridging the gap between the "natural passage" style of DYNAMICQA and the "programmatic statement" style of MULAN.
4. Model Size Sensitivity: Revealed that the ease of updating temporal facts is non-monotonic with respect to model size, challenging the generalizability of findings derived solely from 7B models.

4. Key Results

Reproduction of Original Studies

• DYNAMICQA: Confirmed that temporal facts have lower accuracy, lower Coherent Persuasion scores, and higher "stubbornness" (resistance to change) compared to static facts.
• MULAN: Confirmed that mutable (temporal) facts have higher Update Success rates and lower confidence/F1 scores compared to immutable facts.

Cross-Dataset Findings (The Source of Conflict)

• MULAN findings generalize to DYNAMICQA: When applying DYNAMICQA's metrics to MULAN data, temporal facts still appeared easier to update (higher CP scores, lower stubbornness). This suggests MULAN's dataset design (Wikidata relations) inherently makes facts more malleable.
• DYNAMICQA findings do NOT generalize to MULAN: When applying MULAN's metrics to DYNAMICQA data, the results were mixed. While some models showed lower update success for temporal facts, others did not. Furthermore, models assigned higher confidence to temporal facts in this setting, contradicting the idea that they are inherently harder to update.
• Root Cause: The discrepancy arises because DYNAMICQA infers temporality via Wikipedia edit counts (often resulting in misclassification of static facts as temporal due to minor edits), whereas MULAN uses Wikidata relation types explicitly designed for time-varying facts. Additionally, the template sensitivity (single template in DYNAMICQA vs. multiple in MULAN) affects performance.

Model Size Effects (Gemma 3 Family)

• Non-Monotonic Behavior: The relationship between model size and updateability is not linear.
  • Gemma 4B: Showed the pattern where temporal facts are easier to update (consistent with MULAN).
  • Gemma 1B and 12B: Showed the opposite pattern, where temporal facts were harder to update than static facts.
• This indicates that conclusions about temporal fact updateability are sensitive to model scale and do not generalize even within the same model family.

5. Significance and Conclusion

The paper concludes that there is no consensus on whether LLMs are receptive to updating temporal facts via external context. The conflicting results in the literature are artifacts of:

1. Dataset Construction: How "temporality" is defined and verified (edit counts vs. structured relations).
2. Evaluation Metrics: Whether success is measured by distributional shift (CP) or binary answer replacement (Update Success).
3. Model Size: The behavior changes significantly across different parameter scales.

Implication: Neither DYNAMICQA nor MULAN provides a complete picture of temporal knowledge conflicts. Researchers must carefully consider dataset design and evaluation metrics when interpreting LLM behavior. The question of whether external context reliably resolves temporal conflicts remains open and highly contingent on experimental settings.

The authors have made their code and data publicly available to facilitate further research in this domain.