Technical Summary: proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference

Problem Statement

In scientific and industrial settings, primary outcomes of interest are often difficult to measure due to delays (e.g., long attribution windows), high costs (e.g., human labeling), or limited observability. Practitioners frequently substitute these with readily available proxy outcomes (surrogate endpoints, short-term signals, or model predictions). However, valid inference on a proxy does not guarantee valid inference on the primary outcome. Proxy-based estimates can suffer from systematic biases that are difficult to predict, leading to improperly calibrated confidence intervals and flawed decision-making.

Existing literature typically addresses either adjustment (correcting proxy flaws) or validation (diagnosing proxy adequacy) in isolation, rarely connecting the two. Furthermore, most frameworks assume controlled experimental settings, making them ill-suited for generic measurement scenarios like population-level estimation or observational monitoring. There is a persistent gap in effectively using proxy outcomes: a lack of a unified diagnostic layer to determine whether and which adjustment is warranted before application.

Methodology: The proxymate Framework

The paper introduces proxymate, a framework and open-source Python package that organizes proxy validation and adjustment into four distinct levels. Each level corresponds to a specific statistical question, data structure, and set of diagnostics and adjustments.

1. Representativity Level (Population Validity)

• Question: Should the validation data be reweighted before use?
• Context: Validation data ($V_k$) containing paired primary ($Y$) and proxy ($Y^*$) observations may be non-representative of the target population ($P_k$) due to sampling bias or missingness.
• Diagnostics: Checks for covariate distribution gaps (Standardized Mean Differences, Kolmogorov-Smirnov), stability of importance weights (Effective Sample Size), weight extremity, and coverage.
• Adjustments:
  • No Gap: Proceed without correction.
  • IW Adjustment: Apply Importance Weighting (IW) to reweight the validation data to the target population distribution.
  • Gap Too Severe: Flag the validation as unreliable; recommend targeted label acquisition or narrowing the target population.

2. Unit Level (Measurement Quality)

• Question: Does the proxy track the primary at the individual level?
• Context: Evaluates the discrepancy between $Y_i$ and $Y^*_i$ within a historical domain.
• Diagnostics:
  • Accuracy: Systematic bias (Welch's t-test).
  • Precision: Noise levels ($R^2$, RMSE, Pearson correlation).
  • Agreement: Identity line alignment (Bland-Altman, Lin's CCC).
  • Calibration: Predicted probabilities vs. observed frequencies (ECE, Brier score).
  • Discrimination: Ranking ability (ROC AUC, Kendall's $\tau$).
  • Distributional Fidelity: Marginal distribution matching (KS test, TV distance).
• Adjustments:
  • Systematic Bias: Additive bias adjustment.
  • Miscalibration: Platt scaling, Isotonic regression, or Multicalibration (MCGrad).
  • Poor Discrimination: Proxy replacement or model retraining (cannot be fixed post-hoc).

3. Estimate Level (Decision Validity)

• Question: Does the proxy lead to the same aggregate conclusion as the primary within a single domain?
• Context: Assesses reliability for aggregate inference (point estimates and confidence intervals). Individual errors may compound or cancel unpredictably.
• Diagnostics: Aggregate bias (Relative Absolute Error), Inferential validity (CI coverage), and Sign agreement.
• Adjustments:
  • Aggregate Bias: Prediction-Powered Inference (PPI) corrects the proxy estimate using validation data: $\hat{\theta}_{PPI} = \hat{\theta}^*_{all} + (\hat{\theta}_V - \hat{\theta}^*_V)$.
  • CI Miscalibration: PPI confidence intervals explicitly account for bias adjustment uncertainty.
  • No Bias: No adjustment needed (Theorem 1 notes that applying PPI to an already-adequate proxy strictly increases MSE).

4. Domain Level (Cross-Domain Transportability)

• Question: Does the proxy-primary relationship hold across many domains?
• Context: Tests the stability of the proxy-primary bias ($\phi_k$) across historical domains ($k=1, \dots, K-1$) to ensure transportability to a target domain ($K$).
• Diagnostics: Aggregate association ($R^2$), Heterogeneity (Cochran's $Q$, $I^2$, $\tau^2$), and Prediction accuracy (Leave-One-Out MSE).
• Adjustments:
  • Systematic Cross-Domain Bias: Regression Calibration fits a linear mapping ($\hat{\theta}_{cal} = \hat{\gamma}_0 + \hat{\gamma}_1 \hat{\theta}^*$) across historical domains.
  • Residual Bias: Estimate-level adjustment models residual bias as a random effect.
  • High Heterogeneity: Widen prediction intervals; point prediction is deemed unreliable.

Key Contributions

The authors identify four primary contributions:

1. A Four-Level Validation Framework: Organizing proxy validation into Representativity, Unit, Estimate, and Domain levels, ensuring distinct statistical questions are addressed with appropriate diagnostics.
2. A Theoretical Diagnostic-to-Adjustment Bridge: Mapping specific failure modes to targeted adjustments whose assumptions are satisfied, formally closing the gap between diagnosis and production correction.
3. Empirical Application at Scale: Demonstrating the framework's value across multiple production applications at Meta, identifying proxy failures, selecting candidates, and guiding adjustments.
4. An Open-Source Package: A modular Python library exposing validation checks, configurable planners, custom estimators, and structured reports.

Results and Applications at Meta

The framework was applied to six production use cases at Meta, spanning experimentation, prevalence monitoring, and ML model training. These applications addressed four primary bottlenecks: long maturation windows, restricted measurement, low detectability, and low sensitivity.

• Case Study A (Nonpayment Prediction): Validated an ML model predicting 60-day nonpayment outcomes. While unit-level precision failed (expected for heavy-tailed revenue), the proxy showed strong aggregate association ($R^2=0.82$) and directional alignment across 22 experiments. The proxy was approved for directional decision-making (go/no-go) but not for precise magnitude estimation, enabling ~300 annual experiments.
• Case Study B (Scam Prevalence): Evaluated proxies for scam detection against human labels. The Representativity Level flagged a severe data-quality issue (low Effective Sample Size due to over-representation of flagged content). After reweighting, all candidate proxies failed Unit Level validation (low sensitivity on the broader population). Consequently, all proxies were rejected, preventing the deployment of a biased personalization model and redirecting development toward an ensemble approach.

Overall Impact:

• Validated proxies enabled ~7,500 experiments annually across four workstreams.
• In two cases, the framework successfully rejected biased proxies, preventing downstream harm.
• Millions of proxy-primary unit comparisons were assessed and corrected.

Significance and Claims

The paper claims that proxymate fills a critical practical gap by providing a unified "diagnose-then-correct" workflow. Its significance lies in:

• Preventing Blind Adjustment: By enforcing a diagnostic layer, it prevents the application of adjustments (like PPI) when they are unwarranted or when the underlying data is non-representative, which could otherwise inflate Mean Squared Error (MSE) or amplify bias.
• Modularity and Generalizability: The framework is applicable uniformly to A/B surrogates, prevalence classifiers, and monitoring proxies, regardless of the specific domain.
• Production Readiness: It moves beyond theoretical methodology to a deployed system that handles real-world data constraints (missingness, covariate shift, long maturation) and integrates with existing experimentation stacks.

The authors maintain modesty, noting that the framework validates statistical agreement rather than causal mechanisms and currently validates one proxy against one primary outcome at a time. Future work includes sequential validation and automated proxy selection.