Anytime-Valid Confirmation of Label-Shift Corrections

Technical Summary: Anytime-Valid Confirmation of Label-Shift Corrections

1. Problem Statement

In small-batch scientific deployments, labeled target outcomes are often too scarce to reliably estimate label shifts (where the marginal distribution $p(y)$ changes while $p(x|y)$ remains stable) using traditional unlabeled-target methods like BBSE or MLLS. In these scenarios, practitioners often possess a pre-specified candidate correction derived from domain knowledge, regulatory specifications, or bridging experiments.

The core problem addressed is not estimating the shift, but confirming whether incoming labeled outcomes support a posited, fixed label-shift correction relative to a source predictive model. This is framed as a confirmation problem rather than an estimation problem, requiring a statistical tool that can handle sequential data arrival without a pre-specified sample size, while controlling Type I error under optional stopping.

2. Methodology

The paper proposes a sequential testing framework based on e-values and e-processes (Vovk & Wang, 2021; Shafer, 2021).

Core Construction:
Given a fixed source predictive $p_s(y|x, D_{tr})$ and a candidate weight $w(y)$ derived from domain knowledge, the authors define a tilted predictive:
$$\tilde{p}_w(y|x, D_{tr}) \propto w(y) p_s(y|x, D_{tr})$$
The test accumulates the per-observation likelihood ratio between the tilted predictive and the source predictive:
$$e_i = \frac{\tilde{p}_w(y_i|x_i, D_{tr})}{p_s(y_i|x_i, D_{tr})} = \frac{w(y_i)}{Z_w(x_i)}$$
where $Z_w(x_i)$ is the normalization constant.

Theoretical Guarantees:

• Conditional E-Value: Under the predictive null hypothesis $H_0^{pred}$ (where outcomes follow the source predictive $p_s$), the expectation of $e_i$ given past data and the current input is 1. Thus, $e_i$ is a valid conditional e-value.
• Martingale Property: The running product $M_t = \prod_{i=1}^t e_i$ forms a nonnegative martingale (an e-process).
• Anytime-Valid Rule: By Ville's inequality, the stopping rule $\tau^* = \inf\{t : M_t > 1/\alpha\}$ controls the Type I error at level $\alpha$ simultaneously for all sample sizes $t$, without correction for optional stopping.
• NLPD Gap Identity: The log-wealth $\log M_t$ equals the cumulative difference in Negative Log-Predictive Density (NLPD) between the source and the tilted predictive:
  $$\log M_t = \sum_{i=1}^t (\text{NLPD}_i(p_s) - \text{NLPD}_i(\tilde{p}_w))$$
  Rejection occurs when the tilted predictive accumulates a log-score advantage exceeding $\log(1/\alpha)$.

Closed-Form for Gaussian Processes:
For Gaussian Process (GP) sources with Gaussian label-shift ratios, the authors derive closed-form expressions for the normalization constant $Z_w(x)$ and the per-sample log e-value. This allows for efficient computation using only the GP posterior outputs ($\mu_s, \sigma^2_s$) and the shift parameters.

Multiple Testing:
For parallel testing of $K$ streams, the paper recommends the e-BH procedure (Wang & Ramdas, 2022) using batch e-values to control the False Discovery Rate (FDR), rather than conservative Bonferroni corrections.

3. Key Contributions

1. NLPD-Gap Identity: Connects Kelly-optimal betting to model evaluation, showing that the cumulative NLPD gap serves as the log-wealth of a valid e-process.
2. Anytime-Valid Confirmation Test: Provides a formal sequential test for a fixed posited shift that requires no pre-specified sample size and controls Type I error under optional stopping.
3. Closed-Form GP E-Values: Derives analytical solutions for e-values under Gaussian label-shift tilts, enabling practical implementation in GP regression.
4. Growth-Rate Characterization: Proves that under the alternative (correct specification), the test has asymptotic power one, with a growth rate equal to the average KL divergence between the tilted and source predictives.
5. Empirical Validation: Demonstrates Type I control, finite-sample power, sensitivity to miscalibration, and the advantage of using a reliable prior over noisy re-estimation in small batches.

4. Experimental Results

The authors validate the method using end-to-end GP regression simulations:

• Type I Control: Under the predictive null, the empirical false-alarm rate remains strictly below the nominal $\alpha$ (0.05) across all horizons, confirming the anytime-valid guarantee.
• Power vs. Fixed-Sample LRT: While fixed-sample Likelihood Ratio Tests (LRT) calibrated for specific sample sizes show slightly higher power at large horizons, the e-process maintains validity under arbitrary stopping times, offering a fundamental trade-off between power and flexibility.
• Small-Batch Advantage: In scenarios with small target batch sizes ($n_t=10$), a reliable prior correction (oracle) significantly outperforms adaptive re-estimation in terms of held-out NLPD. The confirmation test provides formal evidence to support using such priors.
• Miscalibration Sensitivity: The test is highly sensitive to source predictive miscalibration. Variance underestimation by the source GP (e.g., factor $c=2$) causes a sharp inflation of Type I error (up to 36.45%), highlighting that the validity guarantee relies on the source model being well-calibrated.
• Misspecification Robustness: When the posited shift is misspecified (wrong magnitude or direction), the test's power decreases, but Type I error remains controlled as long as the weight is fixed before testing. The test does not falsely reject the null due to misspecification; it simply fails to reject.

5. Significance and Claims

The paper claims to provide a necessary tool for a specific, often overlooked setting in model deployment: confirmation rather than estimation.

• Distinction from Estimation: The authors emphasize that rejection confirms the posited correction is consistent with the data relative to the source, but it does not quantify the exact degree of shift. For quantitative estimation, unlabeled-target methods remain the appropriate tool.
• Practical Utility: The method allows practitioners to formally validate domain-knowledge-based corrections (e.g., regulatory specs) as data arrives, avoiding the unreliability of re-estimating shifts from scarce labeled data.
• Limitations: The authors explicitly state that the Type I control guarantee is conditional on the predictive null (i.e., the source model must be well-calibrated). The test is not robust to arbitrary model misspecification without additional calibration layers. Furthermore, the candidate weight must be fixed before the testing stream begins; using data from the testing stream to estimate the weight invalidates the e-value property.

In summary, the paper offers a rigorous, anytime-valid framework for confirming pre-specified label-shift corrections, bridging the gap between domain knowledge and sequential statistical evidence in data-scarce environments.