Technical Summary: Generalized Conformal Predictive Systems Under Distributional Shifts

1. Problem Statement

Machine learning systems deployed in decision-making contexts require not only accurate point predictions but also statistically reliable uncertainty quantification. Specifically, predictive distributions must be calibrated, meaning predicted probabilities should match empirical frequencies, and prediction intervals should attain their target coverage.

A primary obstacle to achieving out-of-sample calibration is distributional shift. Classical conformal prediction methods rely on the exchangeability assumption (that data points are identically distributed and order-independent). This assumption is routinely violated in practice by:

• Covariate shift (changes in the input distribution).
• Label shift.
• Temporal drift.
• Adaptive or feedback-driven data collection (e.g., in biomolecular design).

While prior work has extended Conformal Predictive Systems (CPS) to specific shift regimes (e.g., known covariate shift via likelihood ratios), a general framework for Generalized Conformal Predictive Systems (GCPS) that handles arbitrary distributional shifts while accounting for the uncertainty in estimating shift weights has been lacking. The core challenge is to construct predictive envelopes (sets of Cumulative Distribution Functions, CDFs) that remain valid and calibrated even when the test point is drawn from a distribution different from the training data, and when the weights correcting for this shift are estimated rather than known.

2. Methodology

2.1. Theoretical Framework: Permutation Weights

The authors extend GCPS to non-exchangeable settings by modeling the ordering effects through permutation weights. Instead of assuming the test point $Z_{n+1}$ is exchangeable with the training set $\{Z_1, \dots, Z_n\}$, the method conditions on the unordered multiset of observed atoms. The identity of the test point among these atoms is modeled as a weighted draw with probabilities $w_i$.

• Weighted Empirical Law: For a candidate response value $y'$, the method constructs a weighted observed measure $\mu_{obs}$ and a hypothetical full measure $\tilde{\mu}$ incorporating the test point with its realized weight $w_{n+1}$.
• Shift-Aware Envelopes: The predictive system is constructed by inserting the test atom with its specific weight into the calibration set. Theorem 1 establishes that if the underlying distributional regression procedure $G$ is in-sample calibrated (probabilistic, isotonic, or auto-calibrated), the resulting predictive system $\Pi[X_{n+1}]$ contains a calibrated predictive CDF for $Y_{n+1}$ under the true (potentially non-exchangeable) law.

2.2. Handling Weight Uncertainty

In practice, shift weights are rarely known and must be estimated (e.g., via density ratios), introducing epistemic uncertainty. To address this, the authors introduce weight-uncertainty boxes:

• Definition: A coordinatewise multiplicative bound around estimated weights $\hat{w}_i$, defined as $L_i \hat{w}_i \leq w_i^* \leq U_i \hat{w}_i$.
• Confidence Region: This box $WWU_\alpha$ is treated as a $(1-\alpha)$ confidence region for the true weights.
• Robust Envelopes: The method constructs weight-robust CDF envelopes by optimizing the predictive system bounds over all weight vectors within the uncertainty box. This yields a set of CDFs that guarantees containment of a calibrated distribution with probability at least $1-\alpha$, even if the weights are misspecified within the box.

2.3. Efficient Computation

The paper derives efficient computation rules for three specific instances of $G$, avoiding the need for a naive sweep over all candidate $y'$ values:

1. Conformal Binning (CBIN): Uses weighted empirical CDFs within bins. The robust envelopes have closed-form expressions derived by pushing atoms with $Y_i \leq y$ to their upper weight bounds and those with $Y_i > y$ to their lower bounds.
2. Conformity-Measure CPS (CMCPS): Applies the same logic as CBIN but operates on conformity scores rather than raw outcomes.
3. Conformal Isotonic Distributional Regression (CIDR): Uses antitonic regression. The robust envelopes are computed by running the isotonic regression routine (PAVA) twice: once with weights set to their upper bounds and once with lower bounds, depending on the indicator values.

3. Key Contributions

The paper makes three primary contributions:

1. Extension to Non-Exchangeability: It extends GCPS to non-exchangeable sequences by encoding departures from exchangeability via permutation weights, constructing shift-aware predictive envelopes that remain valid under arbitrary distributional shifts.
2. Robustness to Weight Uncertainty: It develops robust predictive systems based on weight-uncertainty boxes. These provide finite-sample or asymptotic confidence guarantees that the envelope contains a calibrated predictive CDF, with the envelope thickness reflecting both finite-sample uncertainty and weight misspecification.
3. Efficient Algorithms and Empirical Validation: It provides efficient evaluation rules and closed forms for CBIN, CMCPS, and CIDR. It demonstrates empirically that shift-aware weighting improves predictive performance and preserves calibration under feedback covariate shift, while the band width effectively quantifies epistemic uncertainty.

4. Experimental Results

4.1. Synthetic Covariate Shift

Experiments were conducted on a 5D regression task with covariate shift.

• Calibration: Unweighted methods significantly under-covered (coverage $\approx 0.83-0.88$ vs. nominal $0.90$).
• Weighted Methods: Oracle and estimated weighting restored coverage to near-nominal levels ($\approx 0.91-0.93$).
• Uncertainty Propagation: The "Estimated CI weighting" (using the uncertainty box) was the most conservative, achieving coverage up to $\approx 0.97$. This came at the cost of slightly higher CRPS (Continuous Ranked Probability Score) and wider bands, correctly reflecting the additional epistemic uncertainty from weight estimation.
• Sample Size: As training and calibration sample sizes increased, the conformal bands tightened, indicating reduced epistemic uncertainty.

4.2. Feedback-Driven Biomolecular Design (AAV Capsids)

The method was applied to Adeno-associated virus (AAV) capsid packaging design, a scenario involving Feedback Covariate Shift (FCS) where test distributions are designed based on model predictions.

• Setup: Test distributions were generated with varying "inverse temperature" parameters ($\lambda$), shifting mass toward sequences with high predicted packaging fitness.
• Performance: Importance weighting restored near-nominal coverage across the range of shifts compared to unweighted variants.
• Extreme Shifts: At the most extreme shift ($\lambda=7$), weighted methods showed slightly worse CRPS and wider bands, consistent with a collapse in effective sample size and increased epistemic uncertainty.
• Reliability: Weighted procedures maintained good calibration for rare, high-value outcomes (extreme events), confirming their utility in high-stakes design settings.

5. Significance and Claims

The paper claims to establish the first out-of-sample calibration guarantee for Venn-Abers style predictors (a special case of conformal IDR) under distributional shifts and weight uncertainty within a unified predictive-systems framework.

The authors emphasize that the width of the predictive band is an informative metric of epistemic uncertainty. In their shift-aware setting, the band width captures both the uncertainty in learning the conditional distribution $Y|X$ and the uncertainty in estimating the shift weights. This allows practitioners to:

• Report a single representative forecast for routine use.
• Simultaneously report band-implied bounds on decision-relevant quantities (e.g., tail risks) to ensure decisions are robust to epistemic uncertainty.

The work positions GCPS not merely as a wrapper for conformity scores, but as a flexible calibration framework that can wrap any in-sample calibrated distributional regression procedure, extending its validity far beyond the exchangeability assumption.

6. Limitations

The authors note two specific caveats:

1. Joint Confidence Regions: Theorem 2 requires the weight-uncertainty box to be a valid joint confidence region for the true weights. In experiments, this was constructed asymptotically via a fitted propensity model.
2. Fixed Active Sets: The closed-form envelopes for CBIN and CIDR assume the active set (for binning) or the ordering (for IDR) is fixed across the uncertainty box. This restricts binning and ordering choices to those defined prior to weighting.