Comparing label-free quality-control scores for automated angle-of-progression measurement in intrapartum ultra- sound: a retrospective benchmark study

Technical Summary: Comparing Label-Free Quality-Control Scores for Automated Angle-of-Progression Measurement

Problem Statement
Automated measurement of the Angle of Progression (AoP) from intrapartum ultrasound is generally accurate on average but suffers from a "long-tail" error distribution, where a small fraction of measurements fail by tens of degrees without any warning signal. For clinical deployment, the critical challenge is not merely average accuracy but identifying which specific outputs are unreliable enough to warrant manual review. This requires "label-free" reliability scores—metrics computed solely from model predictions without access to ground-truth labels—to trigger selective prediction (refusing the least reliable inputs). The study addresses three specific gaps: whether disagreement should be computed on the derived angle or the underlying landmark coordinates; the trade-off between review workload and the capture of gross errors; and the stability of thresholds when transferred from validation to test data.

Methodology
This retrospective benchmark study utilized the public IUGC2025 dataset (501 test files, 493 distinct images) and a held-out split of the ISBI 2015 cephalometric dataset for replication.

• Detector Pool: The study employed an ensemble of six heatmap-based landmark detectors (three based on HRNet-W48 and three on ConvNeXt-V2), including supervised models, fine-tuned variants, and a noisy-student detector trained on pseudo-labels. The deployed prediction for each frame was the coordinate-wise median of the six detectors.
• Label-Free Scores Compared: The study evaluated several uncertainty estimation strategies:
  1. AoP-space disagreement: Standard deviation of the six per-detector AoP values.
  2. Coordinate disagreement (E-CPV): Mean Euclidean distance of detector coordinates from their mean (adapted from Schobs et al.).
  3. Ensemble Heatmap Peak (E-MHA): The peak value of the averaged, normalized heatmaps.
  4. Input Consistency: The mean absolute change in AoP for a single detector (noisy-student) under Gaussian input perturbations.
  5. Heatmap Confidence: The mean peak sigmoid value across detectors.
• Evaluation Metrics: The primary endpoint was the retained error at 90% coverage (refusing the top 10% of suspicious frames). Secondary metrics included residual failure rates for errors >10°, >15°, and >20° at review budgets of 2–20%, and the transferability of thresholds fixed on validation data to the test set. A failure audit was conducted on all missed errors >20°.

Key Results

• Baseline Accuracy: The deployed median prediction achieved a mean AoP error of 4.62°. Errors exceeding 20° occurred in 1.2% of files (6 out of 501).
• Score Performance:
  • Referring the 10% most-disagreed frames reduced the retained error to 3.97°.
  • No significant difference was found between AoP-space disagreement and coordinate disagreement at the primary endpoint (90% coverage). The difference was +0.017° (95% CI −0.10 to 0.10), with coordinate disagreement showing slightly better performance in detecting the worst decile of frames (AUC 0.775 vs. 0.741).
  • E-MHA and Heatmap Confidence performed poorly as ranking scores for this heterogeneous ensemble, failing to effectively rank errors.
  • Input Consistency performed comparably to disagreement at 90% coverage but was less effective at catching the most extreme gross failures (>20°) at a 5% budget.
• Review Workload: At a 5% review budget, both disagreement scores caught 5 of the 6 errors >20°, reducing the residual failure rate from 1.2% to 0.2%. However, for moderate errors (>10°), the reduction was modest (from 9.4% to ~7.8%).
• Threshold Transfer: Thresholds calibrated on 100 validation frames did not maintain their intended workload on the test set. For example, a threshold intended to flag 5% of frames flagged 11.2% of test frames for AoP-space disagreement.
• Failure Audit: The single missed failure (>20°) at the 5% budget was a case where all six detectors shared a similar underestimation with little spread between them. Additionally, a systematic bias was observed where large reference angles (>140°) were frequently underestimated, and these were often not flagged by disagreement scores.
• Replication: On the ISBI 2015 cephalometric dataset, disagreement reduced the retained Mean Radial Error (MRE) by 0.28 mm at 90% coverage, confirming the ranking effect generalizes to a different anatomical task.

Significance and Claims
The paper claims to provide a rigorous, label-free evaluation of quality control mechanisms for automated obstetric ultrasound. Its primary contributions are:

1. Demonstrating the efficacy of ensemble disagreement: It confirms that disagreement among an ensemble of detectors is a robust signal for identifying gross measurement failures, catching 5 of 6 extreme errors with only a 5% review workload.
2. Clarifying the metric of disagreement: It finds no evidence that computing disagreement on the derived angle (AoP) is superior to computing it on the raw landmark coordinates, suggesting coordinate-based disagreement is preferable as it is independent of the specific measurement geometry.
3. Identifying limitations: The study highlights that disagreement scores fail to detect "shared errors" where all detectors agree on an incorrect value (e.g., systematic underestimation of large angles).
4. Highlighting calibration challenges: It demonstrates that thresholds derived from small validation sets do not reliably transfer to new data, necessitating local calibration for clinical deployment.

The authors explicitly state that these are exploratory, frame-level results that do not establish patient-level safety or clinical benefit, which would require prospective studies. The work serves as a benchmark for what label-free quality control can and cannot achieve in this specific domain.