Recipes for Calibration Checks in Safety-Critical Applications

1. Problem Statement

Safety-critical systems (e.g., autonomous vehicles, medical monitors, weather forecasters) rely on probabilistic forecasters that output full probability distributions rather than point estimates. While these systems must be validated to ensure their uncertainty estimates are reliable, current validation practices have significant limitations:

• Focus on Accuracy: Many evaluations only check the mean accuracy of predictions, ignoring the distributional properties (variance, shape) which are crucial for decision-making under uncertainty.
• Continuous Scores vs. Decisions: Standard calibration metrics (e.g., Expected Calibration Error, CRPS, reliability diagrams) produce continuous scores. These require human interpretation and arbitrary thresholding, which is unsuitable for formal safety certification where a binary Pass/Fail verdict with a controlled false-rejection rate is required.
• Rigid Frameworks: Existing literature treats different aspects of calibration (data models, metrics, hypothesis types, testing procedures) in isolation. There is no unified framework that allows engineers to swap components (e.g., changing from offline to online testing) without re-deriving the entire methodology.
• Safety Asymmetry: Standard tests often treat over-confidence (intervals too narrow) and under-confidence (intervals too wide) symmetrically. In safety-critical contexts, over-confidence is dangerous, while conservatism is often acceptable.
• The "Large-N" Problem: As sample sizes grow, standard hypothesis tests reject models for infinitesimally small deviations, even if those deviations are operationally irrelevant.

2. Methodology: The Four-Slot Framework

The author proposes a modular framework that decomposes calibration checking into four independently swappable "slots." A specific "recipe" is formed by selecting one option from each slot.

Slot 1: Model & Data

Defines the form of the forecast and observations.

• Options: Univariate/Multivariate, Parametric (Gaussian) vs. Non-parametric (Particle clouds, Conformal sets).
• Adaptation: For non-parametric models (like particle filters) where a CDF is unavailable, the framework uses half-plane projections (sliced-Wasserstein approach) to reduce multivariate problems to 1D coverage checks.

Slot 2: Metric

Quantifies the discrepancy between forecasts and outcomes.

• Coverage: Checks if the fraction of outcomes falling within a prediction interval matches the nominal level (e.g., 95% of outcomes in the 95% interval).
• PIT (Probability Integral Transform): Transforms outcomes into uniform variables ($U_i = \text{CDF}(\hat{p}, y_i)$). Under perfect calibration, $U_i \sim \mathcal{U}(0,1)$.
• Folded PIT: To focus on dispersion (variance) rather than bias, the paper introduces folding: $V_i = |2U_i - 1|$. This maps the coverage curve directly to the empirical CDF of $V_i$.
• Statistical Power: The framework advocates for the Kolmogorov-Smirnov (KS) test on the folded PIT over per-level Binomial tests with Bonferroni correction. The KS test exploits the exact correlation structure (Brownian bridge) between levels, recovering statistical power lost by conservative Bonferroni corrections.

Slot 3: Hypothesis Formulation

Defines what constitutes "acceptable" calibration.

• One-Sided Tests: Instead of rejecting both over- and under-coverage, the framework allows one-sided hypotheses that only reject over-confidence (narrow intervals) while accepting conservatism (wide intervals).
• Tolerance Bands: To address the "Large-N" problem, the framework introduces equivalence testing. It defines a tolerance budget ($\epsilon_{tol}$), accepting models whose true coverage is within a small, engineering-specified margin of the nominal level. This prevents rejecting practically adequate models due to statistical noise in large datasets.

Slot 4: Testing Procedure

Produces the final decision.

• Offline (P-values): Uses standard p-values for fixed validation sets (e.g., Bonferroni or KS).
• Online (E-values): For sequential monitoring, the framework uses e-values (test martingales). E-values allow for "anytime-valid" testing: the test can be stopped at any time, and the Type-I error remains controlled. This is critical for runtime safety monitoring where data arrives continuously.

3. Key Contributions

1. Modular Grammar: A unified framework organizing calibration testing into four independent slots, enabling the creation of diverse "recipes" tailored to specific safety constraints without re-deriving theory.
2. Safety-Critical Refinements:
   • Introduction of one-sided tests that distinguish between dangerous over-confidence and benign conservatism.
   • Integration of tolerance bands to allow for operationally acceptable miscalibration, solving the asymptotic rejection problem.
3. Operational Decision Making: Shifts the paradigm from continuous scoring (requiring human thresholding) to rigorous Accept/Reject decisions with controlled false-positive rates, suitable for safety cases.
4. Statistical Efficiency: Demonstrates that the Folded KS test on the PIT is statistically more powerful than per-level Binomial tests with Bonferroni correction because it correctly models the correlation between coverage levels.
5. Anytime-Valid Monitoring: Integrates e-values to enable continuous, sequential calibration monitoring for deployed systems.

4. Experimental Results

The framework was validated on two distinct safety-critical problems, demonstrating the flexibility of the slot-swapping mechanism:

• Case 1: Weather Temperature Forecasting (Offline, Parametric)
  • Recipe: Univariate Gaussian model + Folded PIT metric + Two-sided/One-sided hypothesis + Offline KS p-value.
  • Result: The model showed a slight bias in the mean (detected by two-sided KS) but was well-calibrated regarding uncertainty width (passed one-sided folded KS with tolerance). This illustrates how different hypotheses answer different operational questions.
• Case 2: Robot Localization (Online, Non-Parametric)
  • Recipe: Particle Filter (non-parametric) + Half-plane coverage metric + One-sided hypothesis + Online E-value monitor.
  • Result: The system successfully detected "drift" (model mismatch) in real-time. As the magnitude of the unmodeled drift increased, the e-value monitor crossed the rejection threshold earlier and more frequently, demonstrating sensitivity to model degradation without requiring a fixed sample size.

Ablation Studies: Appendix experiments confirmed that swapping a single slot (e.g., changing the metric or the testing procedure) on the same dataset yields operationally distinct decisions, validating the modularity of the framework.

5. Significance

This paper provides a critical bridge between theoretical statistical calibration and practical engineering safety certification.

• For Practitioners: It offers a "recipe book" for constructing validation pipelines that are rigorous, interpretable, and adaptable to specific safety requirements (e.g., "we can tolerate 2% under-coverage but not over-coverage").
• For Safety Cases: It moves calibration from a "diagnostic" (a score to look at) to a "certificate" (a binary pass/fail with a known error rate), which is a prerequisite for deploying AI in autonomous systems.
• Theoretical Unification: It resolves the fragmentation between machine learning (coverage), forecast verification (PIT), and sequential analysis (e-values), showing they are different views of the same underlying probabilistic structure.

In summary, Valentin's work provides the necessary statistical tools to ensure that probabilistic AI systems do not just "look good" on average, but are rigorously certified to be safe under uncertainty.