FETAL-GAUGE: A Benchmark for Assessing Vision-Language Models in Fetal Ultrasound

1. Problem Statement

The paper addresses a critical gap in the application of Artificial Intelligence to prenatal care. While Vision-Language Models (VLMs) have shown promise in general medical imaging (e.g., CT, MRI, X-ray), there is no standardized benchmark to evaluate their performance in fetal ultrasound.

• Context: Ultrasound is the primary modality for fetal monitoring, but there is a global shortage of trained sonographers.
• Challenge: Fetal ultrasound is uniquely difficult due to:
  • Operator dependency: Image quality and orientation vary significantly based on the sonographer's skill.
  • Modality specifics: Images are noisy, lack standardized views compared to other modalities, and contain artifacts not found in adult imaging.
  • Data scarcity: There is a lack of large-scale, publicly available, annotated datasets specifically for fetal ultrasound VLMs.
• Consequence: Without a benchmark, it is impossible to systematically measure the capabilities or limitations of current VLMs in this domain, hindering the development of clinical decision-support tools.

2. Methodology: The Fetal-Gauge Benchmark

The authors constructed Fetal-Gauge, the first large-scale benchmark designed specifically for evaluating VLMs in fetal ultrasound.

A. Dataset Construction

• Scale: The benchmark comprises 42,036 images and 93,451 question-answer (QA) pairs.
• Data Sources: Aggregated from 13 publicly available fetal ultrasound datasets to ensure diversity in imaging conditions, equipment, and patient demographics.
• Standardization Pipeline:
  • Unification: Heterogeneous annotations (segmentation masks, bounding boxes, image-level labels) were converted into a unified Multiple-Choice Question (MCQ) format.
  • Visual Grounding: Segmentation masks were converted into red bounding boxes overlaid on images to create spatial reasoning tasks.
  • Vocabulary Normalization: Clinical abbreviations and synonymous terms were harmonized to reduce label noise.
• Phantom Data: Uniquely, the dataset includes 19,000 images from anatomical phantoms (simulated models used for training). This allows for the evaluation of model performance in controlled, repeatable environments versus real clinical data.

B. Task Design

The benchmark evaluates models across five distinct clinical tasks, all formulated as MCQs to ensure objective, automated evaluation:

1. Anatomical Plane Identification (PI): Identifying the specific view (e.g., trans-thalamic, abdominal).
2. Clinical View Conformity (VC): Determining if an image meets diagnostic standards.
3. Fetal Orientation Assessment (FO): Determining the fetus's position/orientation.
4. Clinical Diagnosis (CD): Classifying images as normal, benign, or malignant.
5. Visual Grounding (VG): Identifying the anatomical structure highlighted by a bounding box.

C. Evaluation Protocol

• Models Tested: 15 state-of-the-art VLMs were evaluated, including:
  • Medical-specific: Lingshu, MedVLM-R1, MedGemma, HuatuoGPT-Vision.
  • General-purpose: InternVL, Llama-3.2-Vision, Qwen2.5-VL, Aya-Vision, VIP-LLaVA.
  • Commercial: GPT-5.
• Metrics: Primary metric is Accuracy.
• Fine-tuning: Selected models (Llama-3.2 and Qwen2.5) were fine-tuned using LoRA to assess the impact of domain-specific training.

3. Key Results

A. Overall Performance Gap

Current VLMs perform poorly on fetal ultrasound tasks, falling far short of clinical requirements.

• Best Performer: GPT-5 achieved the highest overall accuracy at 55%.
• Open-Source Leaders: Lingshu-32B (46%) and Lingshu-7B (40%).
• Baseline: Most other models performed near random chance levels (~26%).
• Consensus: As shown in Figure 1 of the paper, very few questions were answered correctly by all models; the majority were only answered correctly by a small subset.

B. Task-Specific Findings

• Plane Identification (PI) & Visual Grounding (VG): Showed the most variation. GPT-5 led with 66% (PI) and 58% (VG).
• Orientation (FO) & Diagnosis (CD): All models performed near random chance, indicating a fundamental inability to reason about fetal positioning or pathology in this modality.
• Fine-Tuning Impact: Domain adaptation significantly improved performance.
  • Llama-3.2-11B: Improved from 33% to 85% after fine-tuning.
  • Qwen2.5-VL-7B: Improved from 24% to 52%.

C. Domain Adaptation & Structure Size

• Phantom vs. Clinical: Models generally performed worse on phantom images than real clinical images. This suggests a domain gap where models struggle with the specific artifacts and brightness variations of simulated training data.
• Scale Sensitivity: Performance dropped sharply as anatomical structures became smaller.
  • Large structures: >80% accuracy for top models.
  • Small/Medium structures: Accuracy often fell below 50%, highlighting a weakness in fine-grained spatial reasoning.

4. Key Contributions

1. Fetal-Gauge Dataset: The first and largest public benchmark for fetal ultrasound VLMs (42k images, 93k QAs), integrating 13 sources and including a unique phantom subset.
2. Comprehensive Evaluation: A systematic benchmark of 15 diverse VLMs (medical and general) under a unified framework.
3. Critical Analysis: Identification of specific failure modes, including poor performance on small structures, confusion between similarly shaped anatomical planes, and the inability to generalize from phantom to clinical data without specialized training.
4. Open Access: The benchmark is publicly available to drive future research.

5. Significance and Conclusion

• Clinical Reality Check: The study reveals that despite advances in medical AI, current VLMs are not yet ready for clinical deployment in fetal ultrasound, with the best model achieving only 55% accuracy.
• Research Direction: The paper highlights that generic medical training (on CT/MRI) is insufficient. Ultrasound-specific training data (as seen in the Lingshu models) is crucial for transferability.
• Future Path: The benchmark establishes a rigorous foundation for developing:
  • Domain-adapted architectures.
  • Improved spatial reasoning mechanisms.
  • Robust strategies for handling operator-dependent variability and phantom-to-clinical domain shifts.

In summary, Fetal-Gauge serves as a necessary "reality check" for the medical AI community, demonstrating that while VLMs show potential, significant architectural and data-driven innovations are required before they can reliably assist in prenatal care.