LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA

Technical Summary: LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA

Problem Statement
While Multimodal Large Language Models (MLLMs) have advanced significantly in general vision-language tasks and single-image medical analysis, they remain largely underexplored in the context of longitudinal medical visual reasoning. In clinical practice, patient assessment relies heavily on comparing sequential imaging studies over time to evaluate disease progression, treatment response, and prognosis. Existing medical Visual Question Answering (VQA) and report generation datasets are predominantly limited to single-visit, single-image scenarios. Furthermore, current multi-image datasets often aggregate images from different patients or modalities rather than capturing the temporal evolution of a single patient's condition. Consequently, there is a lack of a systematic benchmark to evaluate the ability of models to reason across temporal sequences in medical imaging.

Methodology
To address this gap, the authors introduce LoMeVQA, a comprehensive benchmark comprising 206,000 longitudinal VQA pairs. The dataset construction follows an automated pipeline designed to ensure clinical grounding and reduce hallucinations:

1. Longitudinal Data Construction: The pipeline organizes patient records from the MIMIC-CXR dataset chronologically. Using RadGraph, it extracts structured medical entities and their supporting sentence-level descriptions from radiology reports. These entities are aggregated across visits to construct "entity-level trajectories" that reflect the temporal evolution of specific clinical findings.
2. Difficulty-Controlled VQA Synthesis: The system employs a sliding window approach to sample consecutive studies. It identifies shared clinical entities across these time steps and uses a Large Language Model (LLM) to generate questions and answers based on the entity trajectories. This process allows for the control of difficulty by varying the temporal span and the number of involved entities.
3. Filtering and Validation: To ensure reliability, a two-stage filtering process is applied. First, an automated consistency check uses an LLM to verify that generated Q&A pairs align with the source clinical reports, labeling them as consistent, contradictory, or uncertain. Second, a human review by three clinicians with varying expertise levels is conducted to select high-quality samples for the test set.
4. Task Definition: LoMeVQA defines five distinct tasks:
   • Progress Classification (PC): Classifying disease progression as improved, stable, or worsened.
   • Progress Description (PD): Generating descriptive explanations of disease changes.
   • Progress Report Generation (PRG): Synthesizing integrated reports summarizing progression across multiple time points and regions.
   • Differential Region Grounding (DRG): Localizing regions with notable changes via bounding boxes.
   • Differential Region Description (DRD): Describing specific differences within a predefined region.

Key Contributions

• LoMeVQA Benchmark: The first large-scale, multi-task benchmark specifically designed for longitudinal medical image analysis, covering 206K samples across five diverse reasoning tasks.
• Automated Construction Pipeline: A novel method integrating medical knowledge graphs (via RadGraph) with LLMs to synthesize clinically grounded VQA pairs while mitigating hallucinations through trajectory modeling and consistency filtering.
• MedLong-8B: A specialized model fine-tuned from Qwen3-VL-8B on the LoMeVQA-dev set, designed to address the specific challenges of temporal medical reasoning.
• Systematic Evaluation: A comprehensive evaluation of both general-purpose and medical-domain MLLMs, revealing significant performance gaps in longitudinal reasoning.

Results
Extensive evaluations on the LoMeVQA-test set (2,500 high-quality samples) and out-of-distribution benchmarks (MMXU-test and MIMIC-CXR-T) yield the following findings:

• Current Limitations: Existing MLLMs, including state-of-the-art medical models like Lingshu and MedGemma, perform poorly on LoMeVQA. For instance, the best-performing models achieve only ~55% accuracy on classification tasks and approximately 25% Intersection over Union (IoU) on grounding tasks. General-purpose models often fail to detect subtle temporal changes or rely excessively on textual priors rather than visual evidence.
• Performance of MedLong-8B: The fine-tuned MedLong-8B model achieves state-of-the-art performance across all five in-distribution tasks, significantly outperforming both general and medical baseline models. It also demonstrates strong generalization capabilities on out-of-distribution datasets, improving performance by over 10-20% compared to its base model.
• Fine-tuning Dynamics: Experiments indicate that larger model capacities (8B vs. 4B) lead to better adaptation. Furthermore, multi-task fine-tuning (training on all tasks jointly) performs comparably to single-task fine-tuning, suggesting that sufficiently large models can learn multiple task-specific patterns without substantial interference.
• Qualitative Analysis: Attention map visualizations reveal that while base models often attend minimally to visual inputs during generation, MedLong-8B exhibits a strengthened alignment between visual evidence and linguistic reasoning, progressively focusing on relevant image regions as it generates answers.

Significance
The paper posits that LoMeVQA fills a critical void in medical AI research by providing the first systematic framework for evaluating longitudinal visual reasoning. The results highlight that current MLLMs lack the necessary temporal reasoning capabilities for real-world clinical applications where tracking disease evolution is paramount. By establishing MedLong-8B as a strong baseline and providing a rigorous dataset, the authors aim to facilitate future research into developing more capable medical MLLMs that can accurately interpret temporal sequences, thereby supporting more accurate and personalized clinical decision-making. The authors emphasize that their work is a step toward bridging the gap between static image analysis and dynamic, longitudinal clinical assessment.