PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework

Technical Summary: PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework

Problem Statement
As machine learning (ML) systems are deployed in high-stakes domains, the need for trustworthy uncertainty quantification (UQ) has become critical for safety and reliability. Traditional statistical inference relies on specifying a correct generative model, an assumption often violated by modern complex models (e.g., large language models) and massive, multi-modal datasets. While conformal inference offers a distribution-free alternative that guarantees marginal coverage under exchangeability, it suffers from three key limitations: (1) it lacks an explicit mechanism to screen for model adequacy, potentially producing unstable or unnecessarily wide prediction sets if the base model is poor; (2) its validity is marginal, often leading to coverage shortfalls on specific subgroups; and (3) empirical comparisons of conformal methods across diverse real-world conditions remain limited. Furthermore, the "garden of forking paths" in the data science lifecycle—choices regarding data cleaning, model selection, and hyperparameters—creates a hidden universe of uncertainty often ignored by standard UQ methods.

Methodology: The PCS-UQ Framework
The authors propose PCS-UQ, a framework grounded in the Predictability-Computability-Stability (PCS) principles for veridical data science. The method is designed to produce robust prediction intervals (for regression) and prediction sets (for classification) by integrating model screening, resampling, and adaptive calibration.

1. Predictability (Model Screening): The framework begins with a candidate set of algorithms. Data is split into training and validation sets. Algorithms are trained on the training set, and those performing poorly on the validation set (based on a loss function) are screened out. This ensures that uncertainty is quantified only using models that sufficiently capture the underlying data-generating process.
2. Stability (Bootstrapping): To assess finite-sample variability and algorithmic instability, the filtered set of algorithms is fitted on multiple bootstrapped datasets. This creates a "pseudo-population" of models. For regression, the method utilizes Out-of-Bag (OOB) samples to ensure efficient data usage without requiring a separate fixed validation set for every bootstrap iteration.
3. Computability (Calibration & Efficiency):
   • Multiplicative Calibration: Unlike standard conformal methods that use additive calibration (expanding intervals by a fixed constant), PCS-UQ employs a novel multiplicative calibration scheme. It scales the interval width based on the local spread of the bootstrap predictions. This allows the interval to adaptively widen for samples where the model ensemble exhibits high uncertainty, improving local and subgroup coverage.
   • Deep Learning Approximations: To address the computational cost of training multiple deep learning models, the authors propose two approximation variants: (1) Weighted Monte-Carlo Dropout, where nodes are dropped proportionally to activation, and (2) Additive Noise Perturbation, where Gaussian noise is added to weights. These methods generate perturbed models from a single trained network, bypassing expensive retraining.

Key Contributions

• Novel Framework: Introduction of PCS-UQ, which unifies model screening, bootstrap-based stability assessment, and multiplicative calibration to address the limitations of existing conformal methods.
• Benchmark Compilation: Creation of a comprehensive benchmark comprising 17 real-world regression datasets and 6 multi-class classification datasets. Crucially, the authors manually constructed natural subgroups for each dataset (based on feature distributions) to rigorously evaluate subgroup coverage, a metric often overlooked in conformal literature.
• Theoretical Connection: The paper establishes a formal link between PCS-UQ and conformal inference. It demonstrates that the multiplicative calibration step can be viewed as a novel conformal score function. Under a modified algorithm (splitting data into training, validation, and calibration sets), the authors prove that PCS-UQ preserves valid coverage under exchangeability, effectively bridging PCS principles with split conformal inference.
• Scalability: Development of computationally efficient variants for deep learning that maintain valid coverage while significantly reducing runtime compared to full bootstrap retraining.

Results

• Regression Performance: Across 17 real-world datasets, PCS-UQ maintained the target 90% marginal coverage while producing prediction intervals that were 10–20% narrower on average than "oracle-selected" conformal baselines (Split Conformal, Studentized Conformal, and Jackknife+-after-Bootstrap).
• Subgroup Coverage: PCS-UQ consistently achieved target coverage across constructed subgroups. In contrast, standard conformal methods (even those with oracle-selected models) frequently exhibited coverage shortfalls on specific subgroups. The multiplicative calibration was identified as the key driver for this subgroup adaptivity.
• Classification Performance: On 6 multi-class classification datasets, PCS-UQ reduced prediction set sizes by approximately 20% compared to conformal baselines (APS, RAPS, TopK) while maintaining 90% coverage.
• Deep Learning Efficiency: On three computer vision benchmarks (CIFAR-100, Caltech Birds, ImageNet-Small), the proposed approximation methods reduced prediction set sizes by ~20% over conformal baselines. These approximations were 30–100 times faster than the original bootstrap-based PCS-UQ, striking a balance between computational efficiency and interval quality.

Significance
The paper argues that trustworthy UQ requires a broader view of the data science lifecycle, moving beyond reliance on correctly specified generative models or marginal guarantees alone. By integrating the PCS framework, the authors demonstrate that it is possible to achieve both competitive interval widths and consistent subgroup coverage. The work highlights that explicit model checking (predictability) and adaptive calibration (multiplicative scaling) are essential for robust uncertainty quantification in complex, real-world settings. The theoretical connection to conformal inference further validates the methodological choices, suggesting that PCS-UQ offers a practical and theoretically grounded path forward for responsible AI decision-making.