📊 statistics

Beyond Point Estimates: Distributional Uncertainty in Machine Learning Performance Evaluation

This paper advocates for a distributional perspective in machine learning evaluation by treating performance metrics as random variables and analyzing their empirical distributions through quantiles and confidence intervals, demonstrating that meaningful statistical inference on model variability is feasible even with small sample sizes to support more risk-oriented and differentiated model comparisons.

Original authors: Christoph Lehmann, Yahor Paromau

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Christoph Lehmann, Yahor Paromau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to perfect a new recipe. In the world of Machine Learning (ML), the "recipe" is the model, and the "taste test" is the performance metric (like accuracy or error rate).

The Old Way: The Single Taste Test
Traditionally, when chefs (or data scientists) evaluate a recipe, they cook it once, taste it, and say, "This soup is a 9 out of 10." They treat that single number as the absolute truth.

But here's the problem: Cooking isn't perfectly predictable. If you change the order you add ingredients, use a slightly different batch of spices, or stir the pot at a different speed, the soup might taste slightly different every time. In ML, these "ingredients" are things like how the data is split, how the computer starts its calculations, or how it tweaks its settings.

The old method ignores this. It assumes that one taste test represents the whole pot of soup.

The New Way: The "Flavor Distribution"
This paper proposes a new way of thinking: Instead of asking, "What is the score?", ask, "What does the range of scores look like?"

Imagine instead of tasting the soup once, you cook it 20 times with slight, random variations in your technique. You get 20 different taste scores.

Some might be 8.5.
Some might be 9.2.
One might be a disappointing 7.0.

The paper suggests we shouldn't just average these to get a "9.0." Instead, we should look at the distribution (the spread) of those 20 scores. This gives us a much clearer picture of reliability.

Key Concepts Explained with Analogies

1. The "Worst-Case" and "Best-Case" Scenarios (Quantiles)
Instead of just knowing the average, this method looks at the edges of the distribution.

The Analogy: Think of a weather forecast. A standard forecast might say, "It will be 75°F." But a distributional forecast says, "It will likely be between 70°F and 80°F, but there's a 10% chance it could drop to 65°F."
In the Paper: They look at specific points in the spread, called quantiles.
- The 90th percentile might tell you: "In 90% of cases, our model performs at least this well."
- The 10th percentile might tell you: "In 10% of cases, the model performs worse than this."
- This is crucial for "risk." If you are building a self-driving car, you don't just care about the average speed; you care about the worst-case scenario (the tail of the distribution).

2. The "Confidence Interval" (The Safety Net)
Since we can't cook the soup 1,000 times (it takes too much time and money), we usually only cook it a few times (say, 10 to 25 times). Because the sample is small, our guess about the "worst-case" scenario might be shaky.

The Analogy: Imagine you are guessing the height of a tree based on looking at it from a distance. You might guess "50 feet." But because you are far away, you aren't 100% sure. So, you say, "I'm pretty sure it's between 45 and 55 feet." That range (45–55) is your Confidence Interval.
In the Paper: The authors show that even with a small number of "cooking runs" (10–25), we can calculate these safety nets. They found that while the intervals get wider (less precise) if you look at the extreme tails (the very worst or very best cases), they are still useful for the middle ground.

3. The "Small Sample" Challenge
The paper acknowledges a major constraint: Running a machine learning model 1,000 times is expensive and slow. Most people can only afford to run it 10 to 25 times.

The Finding: The authors ran massive simulations (cooking the soup 1,000 times in a computer simulation) to see what the "true" distribution looked like. Then, they took small slices of 10, 15, or 25 runs and tried to guess the whole picture.
The Result: Even with just 15 or 20 runs, you can get a surprisingly good idea of the model's stability. You can tell if a model is "consistent" (all scores are close together) or "unstable" (scores jump wildly).

Why This Matters (According to the Paper)
The paper argues that looking at the spread of results is better than looking at a single average.

Two Models, Same Average: Imagine Model A and Model B both have an average accuracy of 87%.
- Model A is consistent: It always scores between 86% and 88%.
- Model B is wild: It sometimes scores 95%, but other times it crashes to 70%.
The Old Way: Would say, "They are the same."
The New Way: Would say, "Model A is safer. Model B is risky because of its wide spread."

Summary
This paper is a guide for moving from "What is the score?" to "How reliable is the score?" It teaches us to treat machine learning results not as fixed numbers, but as a cloud of possibilities. By using statistical tools to map out this cloud (even with limited data), we can make smarter, safer decisions about which models to trust, especially in situations where failure is not an option.

The authors provide practical methods to do this without needing complex math assumptions, making it a tool that can be used right now by engineers and scientists.

Technical Summary: Beyond Point Estimates: Distributional Uncertainty in Machine Learning Performance Evaluation

Problem Statement

Machine learning (ML) models are traditionally evaluated using point estimates of performance metrics (e.g., accuracy, F1 score, RMSE). This approach treats performance as a fixed value, failing to capture the inherent variability induced by stochastic elements of the training process, such as data splitting, weight initialization, and hyperparameter optimization. While these factors are often acknowledged as "confounding factors," they are typically analyzed unsystematically with a primary focus on achieving a single best metric value.

The paper argues that ML training should be viewed as a scientific experiment where the Target Metric of Interest (TMoI) is a random variable conditional on these confounding factors. However, practical constraints often limit the number of repeated training runs (sample size $n$ ) to small or medium ranges (e.g., $n \approx 10\text{--}25$ ) due to computational costs. Consequently, there is a gap in understanding how to perform meaningful statistical inference on the underlying performance distribution given these limited sample sizes, particularly for tail quantiles which are critical for risk-oriented applications (e.g., medicine, banking).

Methodology

The work proposes a distributional perspective where performance metrics are treated as random quantities. The methodology focuses on estimating quantiles and their corresponding Confidence Intervals (CIs) using frequentist statistics.

Statistical Framework

The study assumes an independent and identically distributed (i.i.d.) sample of TMoI values derived from seed-controlled training runs. It evaluates two main statistical tasks:

Point Estimation of Quantiles: The paper compares two estimators:
- Sample Quantile ( $\hat{Q}$ ): Based on order statistics (step function).
- Linear Interpolated Quantile ( $\hat{Q}_L$ ): A smoother estimator using linear interpolation of order statistics.
Interval Estimation (Confidence Intervals): Three nonparametric approaches are investigated for constructing CIs for quantiles, alongside the standard $t$ $t$ -interval for the mean as a baseline:
- Exact Nonparametric CIs: Based on order statistics and the binomial distribution. These are distribution-free but require specific minimum sample sizes to exist for a given confidence level.
- Asymptotic Nonparametric CIs: Based on the asymptotic normality of sample quantiles. These also have minimum sample size requirements, particularly for extreme quantiles.
- Semiparametric Bootstrap CIs: Uses a semi-parametric quantile estimator (Hut02) that extrapolates into the tails using a quantile function. This approach avoids strict parametric assumptions while addressing the instability of standard bootstrapping for quantiles near tails.

Experimental Design

The methodology is validated through two primary avenues:

Real-Data Use Cases: Three ML tasks were analyzed:
- Simpsons Characters: Image classification (20 classes).
- CIFAR10: Image classification (10 classes).
- Superconductors: Regression (predicting critical temperature).
  For each, approximately 500–1,000 seed-controlled repetitions were performed to generate "ground truth" empirical distributions. From these, smaller subsamples ( $n=10, 15, 25, 50$ ) were drawn to simulate realistic constraints and test the performance of the estimators.
Simulation Studies: Controlled simulations were conducted using distributions with bounded support (relevant for metrics like accuracy) and varying shapes (left-skewed Beta, normal, and normal mixture) to validate the robustness of the methods under known theoretical conditions.

Key Results

Point Estimation

Bias and RMSE: Both the sample quantile and linear interpolated estimators exhibit low relative bias (maximum $\approx 1\%$ ) and comparable Root Mean Square Error (RMSE).
Distribution Shape: Estimation is more reliable for central quantiles (around 0.5) and becomes more challenging for tail quantiles (e.g., 0.05, 0.95).
Estimator Choice: There is no substantial practical difference between the sample quantile and the linear interpolated estimator for the sample sizes and distributions tested.

Interval Estimation

Coverage and Sample Size:
- For sample sizes up to $n=25$ , a confidence level of 0.90 is recommended. Higher confidence levels (e.g., 0.95 or 0.99) result in excessively long intervals that reduce precision, particularly for small $n$ .
- Exact and Asymptotic CIs: These methods perform well when sample size requirements are met. However, they often fail to produce valid intervals for extreme quantiles with small $n$ (e.g., $n=10$ ).
- Bootstrap CIs: The semiparametric bootstrap provides a practical alternative when other methods are invalid due to sample size constraints. While they may exhibit slightly lower empirical coverage (dropping to $\approx 0.85$ for $n=10$ ) and longer intervals, they remain applicable where exact/asymptotic methods cannot be computed.
Asymmetry: Estimation uncertainty is asymmetric; lower quantiles generally require larger sample sizes than upper quantiles for the same confidence level. This can be mitigated by transforming the metric (e.g., using $-X$ for accuracy) to shift the quantile of interest to the upper tail.
Comparison to Mean: The $t$ -interval for the mean consistently produces shorter intervals with high stability. However, the paper notes that while the mean is stable, it fails to capture tail behavior and risk. The proposed quantile CIs, while wider, provide necessary information about variability and worst-case scenarios.
Distributional Impact: Skewness and multimodality in the underlying distribution significantly affect interval length. For instance, in skewed distributions, quantiles in the long tail exhibit higher uncertainty and longer CIs.

Significance and Claims

The paper claims to bridge the gap between statistical theory and ML practice by providing a framework for distributional evaluation that is feasible under realistic computational constraints.

Risk-Oriented Interpretation: The primary contribution is enabling a risk-oriented view of model performance. Instead of relying on averages, practitioners can make probabilistic statements (e.g., "accuracy falls below 90% in at most 10% of cases") supported by confidence intervals.
Feasibility with Small Samples: The study demonstrates that meaningful statistical inference is possible even with small sample sizes ( $n \approx 10\text{--}25$ ), which are common in ML due to the high cost of repeated training.
Practical Applicability: The proposed methods (particularly the nonparametric asymptotic CI and semiparametric bootstrap) are distribution-free, easy to implement, and applicable to both classification and regression metrics.
Differentiated Model Comparison: The approach allows for a more nuanced comparison of models. Two models may have similar mean performance, but their confidence intervals can reveal differences in stability and variability that point estimates miss.

The authors conclude that while standard point estimates are insufficient for high-stakes applications, the proposed distributional evaluation provides a practical extension to standard ML workflows, offering a more detailed characterization of uncertainty without requiring parametric assumptions.