🤖 machine learning

Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals

This study conducts a large-scale empirical analysis across 24 medical imaging tasks to evaluate the reliability and precision of various confidence interval methods, revealing how factors like sample size, metrics, and aggregation strategies influence uncertainty quantification and providing a practical decision tree to guide future reporting standards.

Original authors: Pascaline André, Charles Heitz, Evangelia Christodoulou, Annika Reinke, Carole H. Sudre, Michela Antonelli, Patrick Godau, M. Jorge Cardoso, Antoine Gilson, Sophie Tezenas du Montcel, Gaël Varoquaux

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pascaline André, Charles Heitz, Evangelia Christodoulou, Annika Reinke, Carole H. Sudre, Michela Antonelli, Patrick Godau, M. Jorge Cardoso, Antoine Gilson, Sophie Tezenas du Montcel, Gaël Varoquaux, Lena Maier-Hein, Olivier Colliot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of medical imaging, artificial intelligence is learning to see what the human eye might miss. These computer programs can scan X-rays, MRIs, and CT scans to detect tumors, identify fractures, or count cells with a speed and consistency that rivals expert radiologists. However, for these tools to be trusted in a hospital, doctors need more than just a single number saying how well the system works. They need to know how much that number might change if the system were tested on a different group of patients. Because every test set is just a small sample of the vast, complex reality of human disease, any performance score comes with a margin of error. To communicate this uncertainty, scientists use a tool called a confidence interval. Think of this as a safety net drawn around a performance score; it is a range of values that is likely to contain the true, real-world ability of the AI. If the net is too wide, it tells the doctor the system is unreliable; if it is too narrow, it might give a false sense of security. The question of how to draw this net correctly has long been a source of confusion and potential error in the field.

A large team of researchers from institutions across Europe set out to solve this confusion by conducting a massive, systematic investigation into how these confidence intervals actually behave in medical imaging. They did not rely on theoretical guesses or small, isolated examples. Instead, they built a virtual laboratory containing hundreds of real-world scenarios. They took twelve different medical tasks, ranging from identifying brain tumors in MRI scans to classifying skin lesions in photographs, and applied nineteen different AI models to each one. This created a vast collection of test cases covering both the task of finding specific shapes within an image, known as segmentation, and the task of sorting images into categories, known as classification. For every single combination of model and task, they simulated thousands of different test sets to see how the confidence intervals held up under pressure. They tested a wide variety of methods for calculating these intervals, including standard statistical formulas and computer-intensive resampling techniques, while also varying the size of the test sets and the specific metrics used to measure success.

The investigation revealed that there is no single, universal method for calculating these intervals that works well in every situation. The researchers found that the choice of method depends heavily on the specific details of the study. For instance, the type of task matters significantly. When the AI is asked to classify an image into a category, such as "disease" or "no disease," the confidence intervals require much larger test sets to be reliable compared to when the AI is asked to draw a precise outline of an organ. In the classification tasks, the researchers found that the intervals often needed hundreds of cases to be trustworthy, whereas segmentation tasks sometimes required far fewer. Furthermore, the way results are combined from multiple categories plays a crucial role. If a doctor wants to know how well the AI performs on rare diseases, they must look at the performance of each disease separately and then average the results. The study showed that this approach, known as macro-averaging, requires significantly more data to produce a reliable safety net than simply pooling all the data together, known as micro-averaging.

The specific metric used to measure performance also dramatically changes the behavior of the confidence intervals. Some metrics, which measure how well the AI overlaps with a reference shape, tend to produce stable and predictable intervals. Others, which measure the distance between the AI's outline and the true boundary, are much more erratic and require larger sample sizes to stabilize. The researchers discovered that the shape of the data distribution, specifically how skewed or lopsided it is, is a major driver of this instability. When the data is highly skewed, standard methods often fail to capture the true performance, leading to intervals that are either too narrow and misleading or too wide to be useful.

Perhaps most importantly, the study challenged the common practice of blindly following software defaults. Many popular computer programs for data analysis have a "default" setting for calculating confidence intervals, often favoring a specific complex method. The researchers found that this default is frequently the wrong choice for medical imaging. In many cases, a simpler method performed better, while in others, the default method failed catastrophically, especially when the data contained outliers or when the summary statistic was the median rather than the average. They also found that some widely used methods produced intervals that were so wide they were practically useless, while others were so narrow they missed the true value entirely.

To help the medical community navigate these complexities, the researchers translated their findings into a practical decision tree. This guide allows a researcher to look at their specific situation—whether they are doing segmentation or classification, what metric they are using, and how many patients they have—and immediately see which method for calculating the confidence interval is likely to be reliable. The study concludes that while calculating performance uncertainty is essential for the safe clinical use of artificial intelligence, it is not a one-size-fits-all problem. The reliability of the safety net depends entirely on the context, and choosing the right method requires understanding the specific characteristics of the data at hand. By mapping out these dependencies, the work provides a clear path forward for researchers to report their results with the honesty and precision that clinical practice demands.

Technical Summary: Performance Uncertainty in Medical Image Analysis

Problem Statement

The reliable validation and clinical translation of artificial intelligence (AI) in medical imaging are hindered by a lack of rigorous uncertainty quantification for performance estimates. While performance metrics are routinely reported, they are derived from finite test sets and are inherently subject to sampling variability. Confidence intervals (CIs) are the standard mechanism for quantifying this uncertainty, yet the medical imaging community lacks comprehensive guidance on which CI methods are appropriate for specific study characteristics. Existing literature often relies on parametric simulations or limited scopes that do not reflect the complex, non-Gaussian distributions and small-to-moderate sample sizes typical of medical image analysis. Consequently, researchers risk using inappropriate CI methods, leading to either overconfidence (intervals that are too narrow and miss the true value) or underconfidence (intervals that are excessively wide and imprecise).

Methodology

The authors conducted a large-scale empirical investigation to characterize the behavior of widely used CI methods across diverse medical imaging scenarios.

Experimental Design: The study utilized 456 benchmark instances, comprising 19 trained models applied to 12 segmentation tasks (using the Medical Segmentation Decathlon dataset) and 12 classification tasks (using a diverse set of imaging modalities).
Simulation Strategy: To avoid reliance on parametric assumptions that may not hold for medical metrics, the authors fitted Kernel Density Estimates (KDE) to the empirical distributions of performance metrics (for segmentation) and model logits (for classification). These fitted distributions served as the "true" underlying populations.
Sampling and Evaluation: From these true distributions, 10,000 synthetic samples were drawn at various sample sizes ( $n = 10$ to $250$, extending to $5,000$ for specific classification metrics). For each sample, CIs were computed, and empirical coverage (the proportion of intervals containing the true value) and average width were calculated.
Variables Investigated:
- CI Methods: Parametric methods (z, t, Wald, Wilson, Agresti-Coull, Clopper-Pearson), Non-parametric bootstrap methods (Percentile, Basic, BCa), and Concentration inequalities (Hoeffding, Empirical Bernstein).
- Metrics: 8 segmentation metrics (e.g., DSC, NSD, HD, ASSD) and 6 classification metrics (e.g., Accuracy, AUC, F1, MCC).
- Aggregation: Micro-averaging vs. Macro-averaging for classification; various summary statistics (Mean, Median, Trimmed Mean, STD, IQR) for segmentation.
Statistical Analysis: Claims regarding method performance were validated using permutation tests with False Discovery Rate (FDR) correction across thousands of configurations.

Key Contributions

Large-Scale Empirical Analysis: This is the first study to systematically evaluate CI behavior across 456 benchmark instances, covering a broad spectrum of tasks, models, metrics, and sample sizes.
Non-Parametric Simulation Framework: The authors developed a robust simulation pipeline using KDE to capture the true, often skewed, distributions of medical imaging metrics, avoiding the pitfalls of parametric assumptions.
Characterization of Study Dependencies: The study quantifies how reliability (coverage) and precision (width) of CIs depend on specific study characteristics, including the machine learning problem type, metric choice, aggregation strategy, and sample size.
Practical Decision Tools: The authors derived a decision tree to guide practitioners in selecting appropriate CI methods based on their specific study parameters.

Key Results

1. Sample Size Requirements Vary Wildly

The sample size required to achieve reliable and precise CIs ranges from tens to thousands of cases, depending heavily on the study context.

Classification vs. Segmentation: Classification tasks require substantially larger sample sizes than segmentation tasks to achieve comparable CI properties. For instance, achieving a CI width of 0.1 typically requires ~50 observations for segmentation but ~250 for classification. This is attributed to classification metrics conveying less information per sample (binary/limited classes) compared to segmentation metrics (pixel/voxel-level data).
Aggregation Strategy: Macro-averaged metrics require significantly more observations than micro-averaged metrics to achieve reliable coverage.

2. Metric Choice Drastically Impacts CI Behavior

Not all metrics behave equally. Distance-based metrics (e.g., ASSD, MASD) exhibit slower convergence to theoretical coverage and require larger sample sizes than overlap-based metrics (e.g., DSC, IoU).

Skewness Correlation: The study found a strong correlation between the skewness of a metric's distribution and the speed of CI coverage convergence. Highly skewed distributions (common in distance-based metrics) lead to slower convergence and greater sample size requirements.
Summary Statistics: Measures of central tendency (Mean) generally converge faster than measures of dispersion (STD, IQR). However, for highly skewed metrics, the Median or Trimmed Mean can offer more robust coverage than the Mean.

3. Performance of CI Methods

Bootstrap Methods:
- Percentile Bootstrap: Demonstrated the most robust behavior across a wide range of settings.
- BCa (Bias-Corrected and Accelerated): Generally excellent for means, particularly with small samples and skewed data, but fails catastrophically when applied to the Median (and other statistics invariant to single data point removal), leading to severe undercoverage as sample size increases.
- Basic Bootstrap: Consistently underperformed, often providing the worst coverage due to bias. It is only recommended for macro-aggregated Average Precision (AP) in specific contexts.
Parametric Methods: Parametric t-intervals and Wilson intervals (for proportions) outperform bootstrap methods for very small sample sizes ( $n < 25$ ) due to wider intervals that compensate for uncertainty. However, this advantage vanishes rapidly as sample size increases ( $n \ge 50$ ).
Concentration Inequalities: Hoeffding and Empirical Bernstein intervals produce extremely wide intervals (often 2x to 7x wider than bootstrap/parametric methods), reflecting the cost of distribution-free guarantees. They are generally too imprecise for practical use in this domain.

4. Failure Modes

Segmentation: Coverage failures are primarily linked to the presence of outliers in the empirical metric distributions.
Classification: Failures are predominantly associated with class imbalance, particularly when using macro-averaged metrics on small datasets.

Significance and Implications

The paper claims to bridge a critical gap in the literature by providing the first large-scale, data-driven evidence on how CI methods behave in medical imaging. The authors emphasize that:

No "One-Size-Fits-All": There is no single CI method that is optimal for all settings. Blind reliance on defaults (e.g., BCa in SciPy) can lead to misleading conclusions.
Actionable Guidance: The derived decision trees offer immediate, practical advice for researchers to select methods that balance reliability and precision for their specific task (segmentation vs. classification), metric, and sample size.
Foundation for Guidelines: These findings constitute essential building blocks for future consensus guidelines on reporting performance uncertainty in medical imaging AI, moving the field toward more trustworthy and clinically translatable AI models.

The study concludes that while quantifying performance uncertainty is computationally feasible (taking only seconds to hours for individual estimates), the systematic investigation required to understand these uncertainties is complex. The provided decision tools aim to prevent the community from falling into major pitfalls regarding overconfidence or underconfidence in model performance.