ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding

1. Problem Statement

Ultrasound acquisition is a highly operator-dependent skill requiring real-time, adaptive manipulation of the probe (position, orientation, pressure) and device settings to obtain clinically usable views. While Vision-Language Models (VLMs) show promise for autonomous medical systems, existing benchmarks primarily evaluate static image understanding (e.g., anatomy identification, pathology detection).

There is a critical gap in evaluating dynamic procedural reasoning. Current benchmarks fail to assess:

• How probe maneuvers map to imaging goals.
• How operators troubleshoot image artifacts in real-time.
• How protocols progress from one step to the next.
• Causal relationships between actions and image outcomes.

Furthermore, many leading commercial VLMs (e.g., GPT-4/5, Claude) lack native video support, and those that process image sequences often fail at temporal reasoning.

2. Methodology

A. Dataset Construction (ReXSonoVQA)

The authors constructed a new benchmark comprising 514 video clips paired with 514 questions (249 Multiple Choice Questions [MCQ] and 265 Free-Response).

• Source: Publicly available instructional ultrasound videos (YouTube), curated to ensure procedure-rich narration and continuous scanning segments.
• Preprocessing: Videos are cropped to isolate the ultrasound stream (removing UI overlays) and transcribed using WhisperX.
• Ground Truth: An LLM (GPT-5.2) converts time-aligned transcripts into structured Event Logs containing Actions (probe motion, adjustments) and Intents (goals, troubleshooting rationale).

B. Task Definition

The benchmark evaluates three specific cognitive competencies required for autonomous ultrasound:

1. Type 1: Action–Goal Reasoning: Identifying the probe maneuver and its immediate acquisition goal (e.g., "What view is being acquired?").
2. Type 2: Artifact Resolution & Optimization: Identifying image quality limitations (artifacts, occlusion) and the corrective adjustment made to resolve them.
3. Type 3: Procedure Context & Planning: Inferring the current position in the protocol and predicting the next logical step.

C. Quality Control Loop

To ensure questions cannot be answered via text-only priors (a common failure in video QA):

• Blind Solvability Screening: Questions are tested against VLMs without video input. Items solvable via text alone are flagged.
• Distractor Refinement (for MCQs): Distractors are constrained to be medically plausible, at the same abstraction level as the correct answer, and specific to the local anatomy/protocol to prevent elimination strategies.
• Rewriting: Flagged items are rewritten to remove linguistic cues or regenerated with stricter constraints.

D. Evaluation Protocol

• Models Evaluated: Gemini 3 Pro, Qwen3.5-397B, LLaVA-Video-72B, and Seed 2.0 Pro (all with native video support).
• Settings:
  • Video-Informed: Model receives the video clip (audio muted).
  • Text-Only (Blind): Model receives only the question and options.
• Scoring:
  • MCQ: Standard accuracy.
  • Free-Response: LLM-as-a-Judge using a 3-level rubric (Score 2: Correct visual evidence + reasoning; Score 1: Partial; Score 0: Incorrect).

3. Key Results

A. Overall Performance

• Video Advantage: Visual input significantly improves performance. MCQ accuracy rose from 42.2% (text-only) to 66.3% (with video). Free-response mean scores increased from 0.31 to 0.95.
• Model Comparison: Gemini 3 Pro achieved the highest overall performance. LLaVA-Video-72B performed near or below blind baselines, indicating limited procedural video understanding.

B. Task-Specific Challenges

• Type 2 (Artifact Resolution) is the Hardest: This category showed the lowest accuracy (60.8% for MCQ) and the smallest gains over text-only baselines. This suggests current models struggle with causal troubleshooting reasoning—understanding why an image is degraded and how a specific action fixes it.
• Duration Impact: Performance improves with clip duration. Longer clips (>20s) provide richer temporal context, aiding procedural reasoning.
• Failure Modes:
  • Many errors persist even with video input, indicating that visual perception alone is insufficient without deep causal reasoning.
  • In free-response, models often hallucinate anatomical structures or fail to link the visual evidence to the correct clinical conclusion.

C. Blind Baseline Analysis

• MCQ blind accuracy (42.2%) exceeded random chance (25%), revealing that some questions still contain exploitable linguistic patterns or weak distractors.
• Free-response questions showed much lower blind baselines, confirming they are better suited for evaluating genuine video-grounded reasoning.

4. Key Contributions

1. First Video-Based Ultrasound Benchmark: ReXSonoVQA is the first benchmark specifically designed to evaluate procedure-centric understanding in ultrasound, moving beyond static image recognition.
2. Rigorous Quality Control: The introduction of a "blind solvability" screening and distractor refinement loop ensures that questions genuinely require visual input, addressing a major flaw in previous video QA datasets.
3. Comprehensive Evaluation Framework: It defines three distinct cognitive tasks (Action-Goal, Artifact Resolution, Context Planning) that map directly to the requirements of robotic automation and real-time guidance systems.
4. Empirical Insights: The study exposes a fundamental limitation in current frontier VLMs: while they can describe what is happening, they struggle with the causal reasoning required for troubleshooting and optimization (Type 2 tasks).

5. Significance and Future Impact

• Robotics & Automation: ReXSonoVQA provides the necessary evaluation framework for developing autonomous robotic ultrasound systems that can adapt to patient constraints and resolve image quality issues in real-time.
• Clinical Training: The benchmark can drive the development of AI tutors that provide real-time feedback on probe manipulation and protocol adherence for novice sonographers.
• Model Development: The results highlight the need for future VLMs to improve temporal causal reasoning and troubleshooting logic, rather than just static frame interpretation. The authors note that extending this pipeline to in-hospital clinical footage is a key future direction.

In summary, ReXSonoVQA establishes a new standard for evaluating medical AI, shifting the focus from "what does the image show?" to "how is the image acquired and optimized?", which is critical for the next generation of autonomous medical devices.