Technical Summary: MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays

Problem Statement

Current medical Visual Question Answering (VQA) benchmarks for Chest X-rays (CXR) predominantly rely on simplified problem formulations, such as single-image recognition or pairwise image comparison. These settings fail to capture the reality of clinical workflows, where diagnostic reasoning requires comparing examinations across multiple visits to assess disease onset, progression, treatment response, and recurrence. Existing multi-visit benchmarks often focus on report generation or simple presence-absence judgments, lacking explicit evaluation of long-horizon longitudinal reasoning, temporal event ordering, and the integration of evidence across extended timelines. Consequently, there is a gap in evaluating whether Vision-Language Models (VLMs) can perform structured temporal decision-making over dependent, multi-interval visual evidence.

Methodology

The MI-CXR Benchmark

The authors introduce MI-CXR, a benchmark designed to evaluate Multi-Interval longitudinal reasoning over multi-visit CXR sequences. The benchmark consists of 5,311 five-way multiple-choice questions derived from patient timelines containing five temporally ordered visits (four consecutive intervals).

The benchmark is constructed using structured metadata and scene graph annotations from MIMIC-Ext-CXR-QBA and images from MIMIC-CXR-JPG. The construction pipeline involves:

1. Cohort Formation: Selecting patients with at least five temporally ordered visits.
2. Quality Filtering: Retaining only high-confidence, radiologist-annotated observations with clear temporal change labels (emergence, resolution, improvement).
3. Sliding Window Generation: Decomposing longer patient histories into fixed five-visit windows to ensure consistent temporal context.
4. Task Instantiation: Generating questions that explicitly require aggregating information across multiple visits, ensuring answers cannot be derived from a single image or isolated pair.

Task Families

MI-CXR formalizes longitudinal interpretation into three complementary reasoning capabilities:

1. Temporal Event Localization (TEL): Requires identifying the specific interval where a clinically meaningful event (e.g., abnormality emergence or resolution) occurs. This tests reasoning under temporal ordering and exclusivity constraints (e.g., distinguishing the first emergence from subsequent ones).
2. Interval-wise Change Reasoning (ICR): Focuses on interpreting visual changes between consecutive visits. Unlike standard pairwise tasks, the relevant interval is not pre-specified, requiring the model to first localize the described change within the timeline before interpreting its semantic content.
3. Global Trajectory Summarization (GTS): Requires integrating evidence across all five visits to characterize the overall disease course, testing the ability to compose local observations into a coherent global trajectory.

Evaluation Protocol

The authors evaluated 14 state-of-the-art VLMs, categorized into closed-source general-purpose, open-source general-purpose, and medical-specialized models.

• Zero-Shot Prompting: The primary protocol uses deterministic decoding (temperature = 0) to ensure reproducibility.
• Stage-wise Diagnostic Probing: To distinguish between visual perception failures and temporal reasoning failures, the authors employed a two-stage inference protocol. In the first stage, models generate structured interval-wise descriptions; in the second, they answer the original question based solely on these intermediate descriptions.
• Robustness Checks: The study included sensitivity analyses on decoding temperature, reasoning-style prompting ("Let's think step by step"), and one-shot demonstrations, as well as a pilot study on free-form report generation.

Key Results

Baseline Performance

Across all 14 models, performance on MI-CXR was consistently low, with an overall average accuracy of 29.3%, only modestly above the 20% random guessing baseline for five-way multiple-choice questions.

• Task Difficulty: Performance remained low across all task families (TEL, ICR, GTS), indicating that long-horizon temporal reasoning is a fundamental challenge for current VLMs, not limited to specific task designs.
• Model Categories: No single category (closed-source, open-source, or medical-specialized) demonstrated consistently robust longitudinal reasoning. While closed-source models generally performed slightly better, they still struggled significantly. Scaling (parameter size) showed inconsistent effects on performance.

Diagnostic Analysis

The stage-wise diagnostic probing revealed a critical disconnect between local understanding and global decision-making:

• Local Capability: When the relevant temporal interval was explicitly specified (ICR Variant), model accuracy improved substantially (e.g., from ~30% to ~60-70% for top models). This suggests models possess the capability to interpret interval-level changes when the temporal scope is constrained.
• Temporal Decision Failure: The primary bottleneck lies in temporal decision-making. Models often produce locally plausible interval descriptions but fail to:
  • Enforce temporal exclusivity (selecting the single decisive interval among multiple plausible candidates).
  • Bind events into ordered temporal structures.
  • Compose interval-level observations into coherent global trajectories without error propagation.

Error Patterns

Qualitative and quantitative error analysis identified distinct failure modes:

• TEL: Models struggle to resolve competing temporal hypotheses, often failing to enforce exclusivity constraints when multiple intervals appear consistent with an event.
• ICR: Models tend to over-commit to specific change directions even when visual evidence is marginal or ambiguous, leading to forced-choice errors.
• GTS: Errors are dominated by failures in global integration, where local inconsistencies or early misjudgments cascade into incorrect global summaries.
• Robustness: These failure patterns persisted across different decoding temperatures, prompting strategies (including chain-of-thought and few-shot), and output formats (multiple-choice vs. free-text generation), confirming that the limitation is intrinsic to longitudinal reasoning rather than prompt sensitivity or format familiarity.

Significance and Contributions

The paper claims three primary contributions:

1. MI-CXR Benchmark: A systematic evaluation framework for Temporal Event Localization, Interval-wise Change Reasoning, and Global Trajectory Summarization over multi-interval CXR sequences, moving beyond single-image and pairwise evaluations.
2. Formalization of Longitudinal Reasoning: A principled definition of longitudinal CXR interpretation as a global reasoning problem grounded in realistic clinical workflows, emphasizing temporally structured constraints (ordering, exclusivity, and trajectory-level consistency).
3. Diagnostic Framework: A stage-wise analysis revealing a systematic gap in current VLMs where locally correct observations do not reliably yield correct longitudinal decisions.

The authors emphasize that MI-CXR serves as a diagnostic tool to evaluate longitudinal reasoning capabilities in multimodal medical AI. The findings highlight that the current bottleneck in medical VLMs is not merely visual perception, but the ability to perform structured temporal decision-making over extended sequences. The benchmark is intended to motivate future work on representations and inference mechanisms that better support this form of reasoning, rather than serving as a tool for immediate clinical deployment.