Beyond Benchmarks of IUGC: Rethinking Requirements of Deep Learning Methods for Intrapartum Ultrasound Biometry from Fetal Ultrasound Videos

1. Problem Statement

The paper addresses the critical need for automated, real-time monitoring of labor progress in low- and middle-income countries, where trained sonographers are scarce.

• Clinical Context: Approximately 45% of maternal and neonatal deaths occur during the intrapartum phase. Ultrasound parameters, specifically the Angle of Progression (AoP) and Head-Symphysis Distance (HSD), are vital for predicting delivery modes and reducing unnecessary Cesarean sections.
• Technical Challenges:
  • Data Complexity: Intrapartum ultrasound videos suffer from severe artifacts (acoustic shadows, speckle noise), rapid fetal movement, and significant anatomical deformation due to uterine contractions.
  • Task Interdependence: The workflow requires a cascaded pipeline: Standard Plane Classification (identifying valid frames), Segmentation (isolating the Fetal Head [FH] and Pubic Symphysis [PS]), and Biometry (calculating AoP and HSD). Errors in early stages propagate to final measurements.
  • Lack of Benchmarks: Prior work focused on single-center, static images. There was a lack of standardized, multi-center video benchmarks for end-to-end automated measurement.

2. Methodology: The IUGC Challenge

The authors organized the Intrapartum Ultrasound Grand Challenge (IUGC) in conjunction with MICCAI 2024 to establish a new benchmark.

A. Dataset

• Scale: The largest multi-center intrapartum ultrasound video dataset to date, comprising 774 videos (68,106 images) from three hospitals (JNU, SYSU, SMU) in China.
• Distribution:
  • Training: 434 videos (56,571 images).
  • Validation: 40 videos (2,870 images).
  • Testing: 300 videos (8,665 images) from three distinct devices (Esaote MyLab, GE Voluson P8, ObEye), introducing significant domain shift challenges.
• Annotations: Includes standard plane labels, pixel-level segmentation masks for FH and PS, and landmark annotations for biometry.

B. Task Definition

The challenge defined a multi-task framework:

1. Task 1 (Classification): Binary classification of standard vs. non-standard planes.
2. Task 2 (Segmentation): Semantic segmentation of the Pubic Symphysis (PS) and Fetal Head (FH).
3. Task 3 (Biometry): Regression of AoP and HSD based on segmentation outputs.

C. Participating Methods

Eight teams submitted solutions, ranging from baseline to state-of-the-art approaches:

• Baseline (T0): A two-stage UNet-based framework (Segmentation first, then Classification).
• Top Performers:
  • T1 (Ganjie): A novel Video Swin Transformer based end-to-end framework integrating local, temporal, and global features. It uniquely treated classification, segmentation, and biometry as a unified task.
  • T2 (ViCBiC): Used ResNet for classification and DeepLabV3 for segmentation, employing heavy data augmentation (AutoAugmentation) and test-time augmentation.
  • T3 (BioMedIA): Integrated pseudo-labeling (using nnUNet) to leverage unlabeled data and employed dynamic loss scaling.
  • T5 (nkdinsdale95): Utilized ensemble learning (combining EfficientNet, ConvNeXt, etc.) and sophisticated post-processing (ellipse fitting).
  • T4, T6, T7: Various adaptations including sparse attention mechanisms (DSSAU-Net), frequency-domain attention (MFA-UNeXt), and lightweight architectures (LinkNet/MobileNet).

3. Key Contributions

1. First Multi-Center Video Benchmark: Established the largest public dataset for intrapartum ultrasound, specifically designed to test robustness across different devices and centers.
2. Comprehensive Evaluation: Analyzed 8 top methods and 17 additional architectures (including SAM, nnUNet, and various Transformers) across classification, segmentation, and biometry.
3. Methodological Insights:
   • End-to-End vs. Two-Stage: Demonstrated that end-to-end video processing (T1) outperforms traditional two-stage image-based pipelines in overall robustness.
   • Post-Processing: Found that elliptical fitting (often used to smooth segmentation) actually degraded biometry accuracy for HSD, while direct contour-based measurement was superior.
   • Data Augmentation: Showed that complex augmentations (AutoAugmentation) and semi-supervised learning (pseudo-labeling) significantly boost performance.
   • Domain Shift: Highlighted that models struggle significantly with data from the Esaote MyLab device (SMU), indicating a need for device-adaptive learning.

4. Results

Performance was evaluated using Accuracy/F1/AUC (Classification), DSC/ASSD/HD (Segmentation), and Absolute Error (Biometry).

• Classification: T1 achieved the best results (ACC: 0.7441, F1: 0.7555), significantly outperforming others. However, all methods struggled to reach the >86% accuracy of human experts, with the best at ~74%.
• Segmentation: T2 led with a DSC of 0.8857. Notably, SAM (Segment Anything Model) and TransUnet also showed strong performance, suggesting foundation models are viable backbones.
• Biometry: T3 achieved the lowest error for AoP (9.16°), while T2 was best for HSD (10.39 pixels).
• Cross-Dataset Robustness: Performance dropped significantly on the SMU (Esaote) subset compared to JNU and SYSU, confirming that current models are sensitive to device-specific characteristics.
• Ranking Stability: T1, T2, and T3 maintained the top three positions in the comprehensive ranking, though T1 showed the most stability across different evaluation schemes.

5. Significance and Future Directions

• Clinical Impact: The study proves that automated intrapartum biometry is feasible but currently lags behind human expert performance, particularly in standard plane detection. It provides a roadmap for developing point-of-care AI tools to assist in labor management.
• Limitations Identified:
  • Standard Plane Detection: This remains the bottleneck; current models misclassify non-standard frames as standard due to low inter-class separability.
  • Error Propagation: Cascaded pipelines accumulate errors; end-to-end approaches are preferred but computationally expensive.
  • Unlabeled Data: Despite the availability of unlabeled data, most teams did not utilize semi-supervised learning effectively.
• Future Work: The authors call for:
  • Development of lightweight, real-time models for edge devices.
  • Integration of semi-supervised/self-supervised learning to utilize vast unlabeled clinical data.
  • Research into domain generalization to handle diverse ultrasound devices.
  • Exploration of direct regression or keypoint detection to bypass error-prone segmentation steps.

Conclusion: The IUGC challenge successfully established a rigorous benchmark for intrapartum ultrasound analysis. While deep learning has made significant strides, the gap between algorithmic performance and clinical expert reliability remains, necessitating further research in robustness, domain adaptation, and efficient architecture design before widespread clinical deployment.