💬 NLP

MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning

This paper introduces MedFrameQA, the first multi-image medical VQA benchmark derived from educational video transcripts to evaluate clinical reasoning, revealing that current advanced MLLMs struggle significantly with synthesizing visual information across image sequences and tracking pathological progression.

Original authors: Suhao Yu, Haojin Wang, Juncheng Wu, Luyang Luo, Jingshen Wang, Cihang Xie, Pranav Rajpurkar, Carl Yang, Yang Yang, Kang Wang, Yannan Yu, Yuyin Zhou

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Suhao Yu, Haojin Wang, Juncheng Wu, Luyang Luo, Jingshen Wang, Cihang Xie, Pranav Rajpurkar, Carl Yang, Yang Yang, Kang Wang, Yannan Yu, Yuyin Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student how to diagnose a patient. In the real world, a doctor rarely looks at just one X-ray or MRI scan and makes a decision. Instead, they look at a story told through pictures: a scan from last month, a scan from today, a view from the front, and a view from the side. They have to connect the dots between these images to see how a disease is growing, shrinking, or moving.

However, most current AI "doctors" (called Multimodal Large Language Models) are like students who only know how to look at a single snapshot. They are great at saying, "This is a lung," or "There is a shadow here," but they struggle when asked to compare two pictures and say, "The shadow moved here, which means the condition got worse."

This paper introduces MedFrameQA, a new "final exam" designed specifically to test if AI can handle this multi-image storytelling.

Here is a breakdown of what they did, using simple analogies:

1. The Problem: The "Single-Frame" Trap

Think of existing medical AI tests like a game of "Spot the Difference" where you are only shown one picture at a time. The AI just has to identify what's in the picture.

The Reality: Real doctors play "Connect the Dots." They look at a sequence of images to understand a timeline or a 3D structure.
The Gap: Current AI models fail at "Connect the Dots." They treat every image as an isolated island, missing the bridge between them.

2. The Solution: Building a New Exam (MedFrameQA)

The researchers wanted to build a test that forces the AI to look at multiple images at once. But where do you get questions that require this kind of deep thinking? You can't just make them up; they need to be medically accurate.

The "Video Library" Analogy:
Imagine a massive library of medical education videos on YouTube. In these videos, a professor is teaching a class. They show a slide, talk about it, then show the next slide and explain how it relates to the first one.

The Pipeline: The researchers built a robot assembly line to turn these videos into exam questions:
1. Harvest: They grabbed thousands of medical videos.
2. Extract: They pulled out the key slides (images) and the professor's voice (transcripts).
3. Match: They used AI to match the professor's words to the specific slide they were showing.
4. Group: They found slides that were part of the same "story" (e.g., Slide A shows a tumor, Slide B shows the tumor after treatment) and glued them together.
5. Quiz: They asked the AI to write a multiple-choice question that requires looking at all the glued-together slides to answer correctly.

The result is 2,851 new questions. Each question comes with 2 to 5 images and a "gold standard" explanation that proves why the answer is right, based on the video's original script.

3. The Results: The AI Got Stuck

The researchers took 11 of the smartest AI models available (including the latest "reasoning" models that are supposed to be super smart) and gave them this new exam.

The Scorecard:

The Grades: The AI models scored terribly. Most got less than 50% correct. That's barely a failing grade.
The "Reasoning" Models: Even the models designed to "think step-by-step" struggled. They got slightly better scores, but still far from perfect.
The Flaw: When the researchers looked at why the AI failed, they found a pattern:
- The "Amnesia" Effect: The AI would look at the first image, get an idea, and then "forget" it when looking at the second image.
- The "Island" Effect: The AI treated the images as separate, unrelated pictures rather than parts of a sequence.
- The "Direction" Error: In one example, the AI looked at a nerve and said it moved "left" when the images clearly showed it moved "right." Because it got that one detail wrong, the whole chain of logic collapsed.

4. What This Means (According to the Paper)

The paper concludes that while AI is getting better at looking at single pictures, it is currently not ready for the complex, multi-image reasoning that real clinical practice demands.

The Benchmark: MedFrameQA is now a strict ruler. If an AI can't pass this test, it can't yet be trusted to handle the "story" of a patient's medical history through images.
The Challenge: The paper highlights that we need to teach AI not just to see images, but to synthesize them—to understand how Image A changes into Image B.

In short: The paper built a new, harder test based on real medical video lessons. When they ran the smartest AI models through it, the models mostly failed, proving that current AI is still too "myopic" to handle the complex, multi-image reasoning doctors use every day.

Technical Summary: MedFrameQA

Problem Statement

Real-world clinical practice frequently relies on multi-image comparative reasoning, where clinicians synthesize findings across different views, modalities, and time points to diagnose pathology. However, existing medical Visual Question Answering (VQA) benchmarks are predominantly limited to single-frame interpretation. Current datasets (e.g., SLAKE, VQA-RAD, PMC-VQA) focus on isolated image analysis, often requiring only basic object recognition. Even newer multi-image benchmarks (e.g., MedXpertQA MM) often treat images as independent clues rather than interconnected components of a coherent clinical narrative, failing to test the ability to track pathological progression or cross-reference anatomical shifts. There is a critical gap in evaluating whether Multimodal Large Language Models (MLLMs) can perform clinically grounded, temporally coherent reasoning across multiple images.

Methodology

The authors introduce MedFrameQA, a benchmark constructed via a scalable, automated pipeline designed to generate high-quality multi-image VQA pairs from medical education videos. The methodology consists of four primary stages:

Video Collection: The team curated 3,420 medical education videos from YouTube using 114 combinatorial search queries pairing imaging modalities (e.g., MRI, CT, X-ray) with specific diseases or findings. The videos cover 9 human body systems and 43 organs.
Frame-Caption Pairing:
- Frame Extraction: Keyframes delineating scene boundaries were extracted using FFmpeg.
- Filtering: Frames were evaluated by GPT-4o based on image quality, medical content prominence, informativeness, and privacy (excluding human faces).
- Transcription & Alignment: Audio tracks were transcribed using Whisper. Frames were paired with narrative transcripts by defining a symmetric time margin ( $\Delta$ ) around frame intervals to capture relevant speech, followed by GPT-4o refinement to ensure precise clinical terminology and removal of unrelated content.
Multi-Frame Merging: Adjacent frame-caption pairs were merged into coherent "clips" (2–5 frames) if their captions described the same clinical concept. This was determined by asking GPT-4o to verify the correlation between consecutive captions, creating a sliding window process that respects the educational narrative flow.
Question-Answer Generation: For each merged clip, GPT-4o generated multiple-choice VQA items. The generation prompt enforced four constraints:
- Information Grounding: Questions must rely solely on visual evidence described in the captions.
- Educational Clinical Reasoning: Questions must probe skills like anatomical localization and differential diagnosis.
- Contextual Interaction: Questions must explicitly reference the image order (e.g., "in the first image... whereas in the third...").
- Distraction Options: Incorrect choices must be plausible but clinically distinct from the ground truth.
Filtering: The initial pool underwent difficulty filtering (removing items where all models agreed) and human evaluation (removing blurred frames or privacy violations), resulting in 2,851 high-quality VQA pairs.

Key Contributions

First Multi-Image Medical VQA Benchmark: MedFrameQA is the first benchmark explicitly designed to test multi-image reasoning in medical VQA using educationally validated diagnostic sequences.
Scalable Pipeline: The authors propose a novel pipeline for aligning audio narrations with visual frames at scale, transforming large video corpora into structured multimodal reasoning datasets.
Gold-Standard Rationales: Unlike previous benchmarks, MedFrameQA provides explicit, transcript-grounded reasoning chains that link specific frames to the final answer, facilitating error analysis.
Comprehensive Coverage: The dataset includes 2,851 questions with an average of 3.24 images per question, covering diverse modalities and anatomical structures.

Experimental Results

The authors evaluated 11 state-of-the-art MLLMs (including proprietary and open-source reasoning and non-reasoning models) on MedFrameQA.

Low Overall Performance: All assessed models demonstrated persistently low accuracy, with the peak average accuracy remaining below 55% (Gemini-2.5-Flash at 54.75%). Most models fell below 50%.
Reasoning Models vs. Non-Reasoning: Reasoning models (e.g., o1, o3, o4-mini, QvQ-72B) generally outperformed non-reasoning counterparts, suggesting that chain-of-thought reasoning aids in multi-image synthesis. However, even the best reasoning models struggled significantly.
Instability: Performance exhibited high instability across varying image counts (2 to 5 frames) and different anatomical systems/modalities. For instance, accuracy fluctuated by over 11% between the musculoskeletal and urinary systems for the best-performing model.
Error Analysis:
- Isolated Instance Treatment: Models often treated images as isolated instances rather than a sequence, failing to track pathological progression.
- Directional/Information Errors: Case studies (e.g., Figure 3 and Figure 4) revealed that models frequently made directional errors (e.g., misidentifying medial vs. lateral displacement) or neglected critical information in intermediate frames, leading to cascading reasoning failures.
- Modality Sensitivity: Performance varied significantly across modalities (CT, MRI, Ultrasound, X-ray), indicating a lack of robust generalization.

Significance and Claims

The paper claims that MedFrameQA provides a rigorous standard for evaluating the next generation of MLLMs in handling complex, temporally grounded medical narratives. The results expose severe deficiencies in current models' ability to perform multi-image synthesis, highlighting a critical gap between current AI capabilities and the demands of real-world clinical reasoning.

The authors modestly conclude that while MedFrameQA reveals the limitations of current models, effective strategies to enhance multi-image reasoning capabilities remain underexplored. They position the benchmark as a resource to foster the development of more capable diagnostic reasoning systems, noting that full-scale physician evaluation was limited to a subset of questions and that broader expert assessment is needed for future verification. The work does not propose specific new model architectures but rather establishes the necessary evaluation framework to drive such future research.