Technical Summary: Large Language Models Lack Temporal Awareness of Medical Knowledge

Problem Statement

Current methods for evaluating the medical knowledge of Large Language Models (LLMs) rely heavily on atemporal, examination-style benchmarks (e.g., MedQA, PubMedQA). These benchmarks treat medical knowledge as static, overlooking a critical dimension: temporal validity. In reality, medical knowledge is inherently dynamic, evolving continuously with new evidence, guidelines, and treatment approvals. A treatment appropriate under a 2018 guideline may be contraindicated in a 2024 update. Consequently, existing evaluations fail to assess whether LLMs can accurately reason about time-specific medical knowledge or distinguish between current and outdated recommendations. Furthermore, most medical data is historical and longitudinal, requiring models not only to recall facts but to know when those facts were correct.

Methodology: TempoMed-Bench

To address this gap, the authors introduce TempoMed-Bench, the first benchmark designed to evaluate the temporal awareness of LLMs in the medical domain using evolving guideline knowledge.

Dataset Construction Pipeline

1. Guideline Collection & Filtering: The authors collected over 23,000 papers from the PubMed Central (PMC) database. They employed a relaxed initial filter (keyword matching for "guideline" in titles or metadata) followed by a strict LLM-based verification to identify papers issued by professional societies as official clinical guidelines.
2. Trajectory Building: The system constructs "guideline trajectories" by linking a current guideline to its prior versions issued by the same organization on the same topic. This process involved:
   • Using LLMs to extract citations of prior guidelines from the current text.
   • Augmenting trajectories using PubMed's "Similar Articles" feature to recover missing prior versions.
   • Post-processing to restore missing PMIDs and consolidate trajectories.
   • The final dataset, TempoMed-Traj, contains 3,411 guideline trajectories.
3. Difference Extraction: For each trajectory, LLMs identified substantive differences between the up-to-date guideline and each outdated version. A separate verifier LLM scored these pairs to ensure they represented direct, head-to-head clinical comparisons (e.g., a change in recommended therapy for a specific condition) rather than minor clarifications. This yielded 721 difference pairs.
4. MCQ Generation: For each difference pair, the authors generated a multiple-choice question (MCQ) following National Board of Medical Examiners (NBME) guidelines. Each question includes:
   • A clinical vignette.
   • Five options: one aligning with the up-to-date guideline, one with the outdated guideline, two clinically plausible distractors, and an "I don't know" option.
   • The questions explicitly cite the guideline source and year.

Evaluation Framework

The authors evaluated over 10 frontier LLMs (including GPT-5, GPT-4.1, Llama-3.1, Qwen2.5, and MedGemma variants) on TempoMed-Bench through four Research Questions (RQs):

• RQ1: To what extent have LLMs acquired up-to-date medical knowledge?
• RQ2: Can LLMs correctly track and recall historical (outdated) medical knowledge?
• RQ3: Can LLMs maintain temporal consistency across different time periods?
• RQ4: Can agentic Retrieval-Augmented Generation (RAG) systems mitigate these failures?

Key Results

1. Performance on Up-to-Date Knowledge (RQ1)

• Linear Degradation: Model performance on up-to-date knowledge exhibits a gradual linear decline over time rather than a sharp drop at the model's knowledge cutoff date. This suggests that parametric medical knowledge is not strictly bounded by cutoffs; newer knowledge appears less frequently in pre-training corpora compared to well-established historical guidelines.
• Variable Accuracy: Even strong models like GPT-5 do not achieve perfect accuracy on up-to-date knowledge (70.69%), and smaller models perform significantly worse.

2. Performance on Historical Knowledge (RQ2)

• Significant Forgetting: LLMs consistently struggle to recall historical medical knowledge. The accuracy on historical (outdated) knowledge is only 25.37% to 53.89% of the accuracy on up-to-date knowledge.
• Implication: This disparity indicates potential "knowledge forgetting" effects during training, where models prioritize recent data over historical context.

3. Temporal Consistency (RQ3)

• Inconsistent Behaviors: LLMs exhibit temporally inconsistent behaviors, where predictions fluctuate irregularly across neighboring years without coherent transitions.
• Error Patterns: Analysis of prediction heatmaps revealed several dominant error patterns:
  • Only-Know-Latest: The model answers "Yes" to current statements and "No" to outdated ones regardless of the query year (ignoring temporal constraints).
  • All-True/All-False: The model accepts or rejects both statements uniformly across all years.
  • Wrong-Transition-Point: The model recognizes a shift in knowledge but places the transition at an incorrect year.
• Prevalence: The "Only-Know-Latest" and "All-True" patterns were the most frequent, indicating a lack of genuine temporal reasoning.

4. Impact of Agentic RAG (RQ4)

• Limited Improvement: Integrating agentic search tools (ToolUniverse and Biomni) with backend LLMs (GPT-4.1, GPT-5) provided only marginal improvements or, in some cases, degraded performance (ranging from -3.15% to +14.14%).
• Cause of Failure: The authors attribute this to information conflict. While the correct guideline might be retrieved, the presence of additional, potentially inconsistent documents from web searches increases cognitive load and causes interference, leading to incorrect predictions.

Significance and Contributions

The paper makes three primary contributions:

1. Temporal-Aware Medical Knowledge Evaluation: It establishes temporal context as a necessary dimension for evaluating medical LLMs, motivated by the evolving nature of real-world medical practice.
2. TempoMed-Bench: It provides a large-scale, rigorously constructed benchmark of 721 temporally grounded MCQs derived from 3,411 clinical guideline trajectories.
3. Empirical Evidence of Limitations: It provides empirical evidence that current state-of-the-art LLMs lack temporal awareness in medical knowledge. Crucially, it demonstrates that this is a non-trivial challenge that is not easily solved by simply integrating popular agentic RAG tools.

The authors conclude that passing TempoMed-Bench should be considered a necessary (though not sufficient) condition for deploying LLMs in evolving medical environments. The work highlights a critical gap in current AI capabilities, motivating future research into developing models that can better encode and reason over time-specific medical knowledge.