Confidence Matters: Uncertainty Quantification and Precision Assessment of Deep Learning-based CMR Biomarker Estimates Using Scan-rescan Data

1. Problem Statement

Deep learning (DL) models for analyzing cine Cardiovascular Magnetic Resonance (CMR) data are typically evaluated based on accuracy (e.g., Dice scores, mean error against ground truth) rather than precision. While accuracy measures how close a prediction is to a manual reference, precision measures the consistency of the model across repeated measurements of the same patient (scan-rescan).

Current literature often relies on point estimates (a single value per scan) to assess precision. However, biomarker estimates inherently contain uncertainty. In longitudinal studies, where patient evolution is tracked over time, ignoring this uncertainty can lead to misleading conclusions about clinical stability. Existing methods fail to quantify the distribution of possible biomarker values, making it difficult to determine if changes in biomarkers are due to actual physiological changes or model variability.

2. Methodology

2.1. Core Pipeline

The authors utilized a state-of-the-art nnU-Net architecture trained to segment the Left Ventricular Blood Pool (LVBP), Right Ventricular Blood Pool (RVBP), and Left Ventricular (LV) myocardium from Short Axis (SAX) cine CMR data.

• Training Data: 2,793 clinical scans from Guy's and St Thomas' NHS Foundation Trust.
• Biomarker Calculation: Derived from segmentations to calculate Left Ventricular Ejection Fraction (LVEF), Right Ventricular Ejection Fraction (RVEF), and LV Mass (LVM).

2.2. Uncertainty Estimation Techniques

To move beyond point estimates and generate biomarker distributions, the study applied three uncertainty quantification techniques:

1. Deep Ensemble (DE): Utilized the 5-fold cross-validation inherent in the nnU-Net framework. Five separate models were trained and their predictions ensembled during inference.
2. Test-Time Augmentation (TTA): Applied spatial and intensity transformations (rotations, cropping, Gaussian blurring, intensity scaling) to the input image during inference. Ten augmented versions were generated to produce ten distinct estimates.
3. Monte Carlo (MC) Dropout: Kept dropout layers active (rate 0.2) during inference, running the model 10 times to generate 10 stochastic predictions.

2.3. Evaluation Datasets

Two external scan-rescan datasets were used to test precision (where the same patient is scanned twice on the same day with replanning):

• Roca-HC: 184 scans from 92 healthy volunteers.
• Strain-8: 40 scans from 20 healthy volunteers.

2.4. Proposed Metrics

The authors introduced three new distribution-based metrics to assess precision, moving beyond standard Coefficient of Variation (CoV):

• Confidence Precision Percentage (CPP): The percentage of Scan B values falling within the 95% confidence interval (CI) of Scan A (and vice versa).
• Confidence Intersection over Union (CIoU): The ratio of the intersection to the union of the confidence intervals between Scan A and Scan B.
• Precision Difference Percentage (PDP): The percentage of subjects where statistical tests (paired t-test or Wilcoxon signed-rank) rejected the null hypothesis that the Scan A and Scan B distributions are identical.

3. Key Contributions

1. Shift from Point Estimates to Distributions: Proposed a framework to estimate full biomarker distributions rather than single values using DL.
2. Novel Precision Metrics: Introduced CPP, CIoU, and PDP to rigorously quantify the agreement between scan-rescan distributions.
3. Comprehensive Uncertainty Analysis: Evaluated three distinct uncertainty techniques (DE, TTA, MC Dropout) specifically in the context of longitudinal CMR analysis.
4. Critical Insight on Metrics: Demonstrated that traditional accuracy/point-estimate metrics can mask poor precision, whereas distribution-based metrics reveal significant inconsistencies.

4. Results

4.1. Accuracy and Point-Estimate Precision

• The segmentation model achieved high accuracy with an average Dice score of 87%.
• Standard metrics (Mean difference and CoV) suggested satisfactory precision for all methods. For instance, MC Dropout showed the lowest standard deviation in mean differences, suggesting high consistency in point estimates.

4.2. Distribution-Based Precision (The Critical Finding)

Despite good point-estimate metrics, distribution-based analysis revealed significant issues:

• Low CI Overlap: For MC Dropout, the overlap between scan/rescan confidence intervals (CIoU) was >50% in less than 45% of cases. In many instances, the CIs did not overlap at all.
• Statistical Dissimilarity: Statistical tests (PDP) showed that for MC Dropout, over 90% of cases resulted in statistically significant differences between scan and rescan distributions. Even for DE and TTA, a large majority of cases (65–85%) showed significant differences.
• Method Comparison:
  • TTA and DE showed higher agreement in confidence intervals, likely because TTA captures aleatoric uncertainty (variations in patient positioning and scan planning) effectively.
  • MC Dropout produced narrow confidence intervals that failed to capture the variability between scans, leading to low overlap and high statistical rejection rates.

5. Significance and Conclusion

The study concludes that relying solely on point estimates and standard accuracy metrics is insufficient for assessing the reliability of DL-based CMR biomarkers in longitudinal studies.

• The "Illusion" of Precision: Aggregating multiple estimates can create a false sense of precision (low standard deviation) while the underlying distributions of scan and rescan results are statistically different.
• Importance of Aleatoric Uncertainty: The results highlight that aleatoric uncertainty (data noise, patient repositioning) is a dominant factor in scan-rescan variability. Techniques like TTA, which explicitly model input variations, perform better in capturing this reality than MC Dropout in this specific context.
• Clinical Implication: The proposed distribution-based metrics (CPP, CIoU, PDP) provide a more stringent and representative assessment of model repeatability. They suggest that current DL pipelines may need further optimization to improve the repeatability of cardiac functional biomarkers before they can be reliably used for monitoring disease progression over time.

In summary, the paper argues that confidence matters: without quantifying and assessing the full distribution of biomarker estimates, the precision of deep learning models in clinical cardiology remains overestimated.