Estimate Level Adjustment For Inference With Proxies Under Random Distribution Shifts

Technical Summary: Estimate Level Adjustment For Inference With Proxies Under Random Distribution Shifts

Problem Statement
In scientific and industrial experimentation, researchers often rely on proxy outcomes (e.g., short-term metrics, surrogate endpoints, or model predictions) to draw inferences about primary outcomes that are difficult, expensive, or delayed to measure. While proxies facilitate faster decision-making, statistical guarantees derived from proxy data (such as unbiasedness or valid confidence intervals) do not automatically transfer to the primary estimand.

Existing methods for correcting proxy bias—such as surrogacy assumptions, covariate shift adjustments, or prediction-powered inference (PPI)—rely on strict identifying assumptions. These assumptions are often difficult to validate and may be violated by unobserved distribution shifts (e.g., concept drift). When such assumptions fail, proxy-based confidence intervals can be systematically miscalibrated, leading to biased parameter estimates and under-coverage of the true primary parameter. A critical practical constraint is that individual-level historical data required to re-fit or re-calibrate these models is often unavailable; typically, only aggregate metric estimates and uncertainty summaries are retained.

Methodology
The authors propose a generic, estimate-level framework to empirically calibrate proxy-based inference without requiring individual-level historical data. The core approach treats the residual bias between the proxy and the primary metric as a latent random variable.

1. Latent-Bias Model:
   The method assumes that across $K$ domains (where $K-1$ are historical training domains and $K$ is the target domain), the discrepancy between the proxy estimator ($\hat{\theta}^*_k$) and the primary estimator ($\hat{\theta}_k$) follows an additive Gaussian bias model:
   $$\phi_k := \theta^*_k - \theta_k \sim N(\rho, \gamma^2)$$
   Here, $\rho$ represents the mean bias and $\gamma^2$ represents the between-domain variance of the bias. This formulation allows the method to model the distribution of residual proxy bias using only historical aggregate estimates and their covariance structures.

2. Estimation via Method-of-Moments:
   Using historical domains, the authors define the within-domain difference $d_k = \hat{\theta}^*_k - \hat{\theta}_k$. They derive a method-of-moments estimator for the bias distribution parameters $(\rho, \gamma^2)$:

   • $\hat{\rho} = \frac{1}{K-1} \sum d_k$
   • $\hat{\gamma}^2 = \frac{1}{K-1} \sum (d_k - \hat{d})^2 - \frac{1}{K-1} \sum \tilde{\sigma}^2_k$
     where $\tilde{\sigma}^2_k$ is the estimated variance of the difference $d_k$. The estimate $\hat{\gamma}^2$ is truncated at zero to handle finite-sample noise.

3. Inference and Adjustment:

   • Large $K$ (Plug-in): When the number of historical domains is large, the estimated bias parameters are treated as known. The method constructs a Wald-style confidence interval for the target primary metric $\theta_K$ by centering the interval at the debiased estimate $(\hat{\theta}^*_K - \hat{\rho})$ and inflating the variance to include the estimated between-domain bias variance ($\hat{\gamma}^2$).
   • Small $K$ (Domain Bootstrap): When the number of historical domains is limited, uncertainty in $(\hat{\rho}, \hat{\gamma}^2)$ can affect coverage. The authors propose a domain bootstrap procedure that resamples historical domains to propagate this uncertainty, generating an empirical distribution for the target parameter.

4. Evaluation Protocol:
   The paper introduces a "Leave-One-Domain-Out" (LOO) diagnostic to evaluate interval calibration. By treating each historical domain as a target (hiding its primary outcome) and checking if the adjusted proxy interval overlaps with the true primary interval, the method provides a conservative, observable proxy for non-miscalibration.

Key Contributions

1. Generic Framework for Proxy Bias: The paper formalizes the bias inherent in proxy estimators as a latent variable problem, assuming joint normality of sampling distributions. This allows for the characterization of residual bias without stringent identifying assumptions.
2. Scalable Adjustment Procedure: The authors introduce a closed-form method-of-moments estimator and a domain bootstrap procedure that rely solely on historical aggregate estimates. This enables scalable deployment in settings where individual-level data is unavailable.
3. Selective Inflation and Empirical Validation: The method acts as a post-processing step that selectively inflates confidence intervals only when historical diagnostics indicate under-coverage. Extensive simulations and real-world applications demonstrate that this approach improves calibration and coverage across various proxy procedures and distribution shifts.

Results

• Simulations: In multi-domain simulations involving covariate shift and concept drift, unadjusted proxy estimators consistently undercovered. While standard corrections (like weighted PPI) improved calibration in stable settings, they degraded under strong concept drift. The proposed random-bias adjustment stabilized coverage across all drift levels. Notably, for estimators with high residual bias, the interval lengths plateaued as sample size increased, correctly reflecting that uncertainty was dominated by cross-domain bias rather than sampling noise.
• Real-World Applications:
  • Civil Comments Dataset: In estimating the prevalence of toxic comments across web publications, the adjusted proxy intervals increased the overlap rate with true primary intervals from 0.64 to 0.96 (at $\alpha=0.05$), with a corresponding increase in normalized interval width from 0.5 to 1.3.
  • Long-Term Experiments: In estimating Average Treatment Effects (ATE) using model-predicted proxies, the adjustment increased the overlap rate from 0.68 to 1.0. The method effectively inflated uncertainty in experiments where residual bias was large relative to sampling variability.

Significance and Claims
The paper claims to offer a practical, modular solution for improving the reliability of proxy-based inference in the absence of individual-level historical data. By modeling residual bias as a random effect, the method provides a "safety net" that adjusts for violations of standard identifying assumptions (such as covariate shift or surrogacy).

The authors emphasize that this approach does not improve the proxy estimator itself but rather improves the uncertainty quantification associated with it. It is designed to be layered on top of existing correction methods (like PPI or importance weighting). The significance lies in its ability to prevent false positives from overconfident proxy inference, thereby enabling safer and more rigorous decision-making in large-scale experimentation and scientific domains where primary outcomes are delayed or unobservable. The authors modestly note that the method's effectiveness depends on the informativeness of historical domains and that the Gaussian bias assumption is a working approximation that may require richer models for complex bias patterns.