Evaluating the Quality of the Quantified Uncertainty for (Re)Calibration of Data-Driven Regression Models

1. Problem Statement

In safety-critical applications (e.g., autonomous driving, medicine), data-driven regression models must not only be accurate but also provide reliable uncertainty estimates. This property is known as calibration. While numerous methods exist to generate uncertainty estimates (e.g., Deep Ensembles, Bayesian Neural Networks) and recalibrate them, the field lacks a consensus on how to evaluate the quality of these estimates.

The core problem identified by the authors is the inconsistency and lack of comparability among calibration metrics.

• Diverse Definitions: Metrics differ in mathematical foundations (probabilistic vs. interval-based), scaling (bounded vs. unbounded), and assumptions (e.g., Gaussian distribution).
• Conflicting Results: Different metrics often yield contradictory assessments of the same model or recalibration method. One metric may indicate a model is well-calibrated while another deems it poor.
• Cherry-picking Risk: This inconsistency allows researchers to selectively report metrics that favor their proposed method, creating misleading impressions of success.
• Evaluation Gap: While recalibration methods are routinely benchmarked against each other, the metrics used to evaluate them are rarely scrutinized or compared directly.

2. Methodology

The authors conducted a model-agnostic benchmark to evaluate 13 distinct calibration metrics independently of specific modeling techniques.

Experimental Setup:

• Models: Deep Ensembles (5 sub-models, MLPs) trained to minimize Negative Log-Likelihood (NLL).
• Datasets:
  1. 16 Real-World Datasets: Varied domains, sizes, and complexities.
  2. 10 Synthetic Datasets: Generated with controlled, noise-free functional relationships to isolate metric behavior from data noise.
  3. Controlled Miscalibration: Artificially generated "perfectly calibrated" data (Gaussian predictions centered on true targets) which were then perturbed with specific errors (constant/heterogeneous offsets in mean and/or standard deviation) to test metric sensitivity.
• Recalibration Methods Tested: Isotonic Regression, Variance Scaling, GP Normal, and GP Beta.
• Evaluation Strategy:
  • Rank Correlation: Calculated Spearman's rank correlation coefficients between metrics to see if they rank datasets/models consistently.
  • Normalized Performance: Metrics were normalized by their mean across datasets to allow comparison despite different scales.
  • Sensitivity Analysis: Measured the ability of metrics to detect specific types of miscalibration (e.g., under-estimation of uncertainty) in the synthetic controlled experiments.

Metrics Evaluated:
The study categorized metrics into three groups based on their behavior:

1. Group 1 (Proper Scoring Rules): RMSE, Sharpness, NLL, CRPS, Interval Score (IS), Pinball Loss. These assess the full predictive distribution.
2. Group 2 (Threshold/Binned Metrics): Calibration Score (CalS), ECPE, QCE, ENCE. These rely on discretizing confidence levels or standard deviations.
3. Group 3 (Others): PICP, CWC, UCE.

3. Key Contributions

1. Systematic Extraction & Categorization: The authors extracted and categorized a comprehensive set of 13 regression calibration metrics from the literature, clarifying the taxonomy between "calibration" (the state) and "recalibration" (the process).
2. Model-Agnostic Benchmark: They introduced a framework to benchmark metrics independently of the underlying uncertainty estimation method, isolating the behavior of the metrics themselves.
3. Discovery of Fundamental Inconsistencies: The study provides empirical evidence that calibration metrics frequently disagree, even on synthetic data where ground truth is known.
4. Identification of Robust Metrics: Through controlled experiments, the authors identified Expected Normalized Calibration Error (ENCE) and Coverage Width-based Criterion (CWC) as the most dependable metrics.
5. Dataset Size Constraints: They established that calibration metrics become unreliable on small datasets (fewer than 500–1000 samples) due to stochastic effects in binning and sampling.

4. Key Results

• Metric Clustering & Conflict:
  • Metrics naturally clustered into two main groups (Proper Scoring Rules vs. Threshold-based).
  • Strong Negative Correlation: These groups often produced opposite rankings of model performance. For example, a model deemed "best" by Group 1 was often ranked "worst" by Group 2.
  • Synthetic Data Persistence: Even on noise-free synthetic data, these inconsistencies persisted, proving the disagreement stems from the metrics' mathematical definitions, not data artifacts.
• Recalibration Evaluation:
  • The perceived success of a recalibration method varied drastically depending on the metric chosen. A method could appear to improve calibration under one metric while degrading it under another.
  • Proper scoring rules (Group 1) were often insensitive to recalibration adjustments, whereas threshold-based metrics (Group 2) showed larger fluctuations.
• Sensitivity to Miscalibration (Controlled Experiments):
  • ENCE and CWC demonstrated the highest robustness in detecting artificial miscalibration (both in mean and variance) across various scenarios.
  • NLL was effective at detecting mean bias but insensitive to uncertainty scaling alone.
  • ENCE outperformed other metrics in detecting degradation in the most complex scenario (simultaneous heterogeneous offsets of mean and variance).
• Sample Size Sensitivity: Metrics relying on binning (e.g., ECPE, QCE) showed high variability and instability on datasets with fewer than 500 samples.

5. Significance and Recommendations

• Critical Implication: The choice of calibration metric is not a neutral decision; it fundamentally dictates the conclusion of a study. Relying on a single metric can lead to false claims of success or failure in uncertainty quantification.
• Recommendation for Practitioners:
  • Primary Recommendation: Use ENCE (Expected Normalized Calibration Error) as the primary metric. It requires no hyperparameters, evaluates calibration across multiple bins (capturing local patterns), and is robust to both mean and variance errors.
  • Secondary Recommendation: Use CWC (Coverage Width-based Criterion) for its stability, though it requires a user-defined penalty parameter and focuses on a single confidence level.
  • Avoid: Relying solely on PICP (which ignores interval width) or metrics that show high variance on small datasets.
• Future Directions: The authors call for a shift toward simulation-based testing (where ground-truth uncertainty is known) to validate new metrics and suggest exploring relevance-weighted evaluation to handle heteroscedastic data more effectively.

In summary, this paper exposes a critical flaw in current uncertainty quantification research: the lack of a unified, reliable evaluation standard. It provides a rigorous benchmark demonstrating that metric selection can arbitrarily alter research conclusions and offers ENCE as a robust, practical solution for the community.