Technical Summary: Bias-Aware External-Model-Assisted Inference in High-Dimensional Regression

Problem Statement

Modern statistical practice often faces an asymmetry of information: abundant covariates are available, but gold-standard labeled responses are scarce due to the cost of expert annotation or experimentation. Simultaneously, machine-learned predictors (often transfer-learned or foundation models) can impute missing responses cheaply but imperfectly. The inferential challenge is to leverage these abundant unlabeled covariates and external predictors to sharpen conclusions about target parameters without inheriting the predictor's bias.

Existing frameworks, such as Prediction-Powered Inference (PPI) and its power-tuned variant PPI++, attempt to correct predictor outputs using a "rectifier" estimated on the small labeled sample. However, this paper identifies a structural weakness in PPI for linear regression: when the working model and the predictor are both linear, the rectifier exactly cancels the predictor's contribution. Consequently, PPI and PPI++ reduce to ordinary least squares (OLS) on the labeled data alone, regardless of predictor quality. Furthermore, if the predictor is estimated and close to the oracle function, the rectifier can inject noise, inflating variance above the labeled-only floor. Genuine efficiency gains in high-dimensional settings must therefore arise from a mechanism other than prediction rectification.

Methodology: The DEAL Estimator

The authors propose the Debiased External-model-Assisted Lasso (DEAL), a four-stage procedure designed to extract efficiency from the unlabeled design through the variance of a debiased estimator rather than through a prediction rectifier.

1. Bias-Aware Initialization:
   The procedure begins with an external estimator $\hat{\beta}_{ext}$ (potentially from transfer learning). Instead of directly using this estimator, DEAL constructs a "one-step correction" $C$ based on the labeled data. A data-driven shrinkage level $\hat{t} \in [0, 1]$ is estimated using a separate tuning subsample. This shrinkage treats the correction as a noisy observation of an unknown bias vector. The initializer is formed as $\tilde{\beta}_{init} = \hat{\beta}_{ext} + \hat{t}C$.

   • If the external estimator is uninformative, $\hat{t} \to 0$, reverting to a target-only approach.
   • If the estimator is near-oracle, $\hat{t} \to 0$ (switching off the correction to avoid noise injection).
   • If the estimator is biased but informative, $\hat{t} \to 1$, utilizing the correction.

2. Pseudo-Label Imputation:
   Using the initializer $\tilde{\beta}_{init}$, pseudo-labels $\tilde{f} = \tilde{X}\tilde{\beta}_{init}$ are imputed for the unlabeled covariates $\tilde{X}$.

3. Stacked Lasso:
   A Lasso estimator is fitted on the stacked dataset comprising the original labeled data $(X_0, y_0)$ and the pseudo-labeled data $(\tilde{X}, \tilde{f})$. This step leverages the enlarged sample size ($n_0 + N$) to achieve a sharper initial fit and a more stable precision matrix estimate.

4. Final Debiasing:
   A standard debiasing step is applied to the stacked Lasso estimator using an approximate inverse covariance matrix (constructed via Javanmard-Montanari or nodewise Lasso methods). This restores asymptotic normality and valid confidence intervals.

Key Adaptations:

• Variance-Balance Rule: The unlabeled sample size $N$ is not fixed but chosen from the data to balance the reduction in leading noise variance against the introduction of residual bias variance.
• Covariate Shift Handling: A one-sided substitution rule for the debiasing matrix is introduced to restore coverage when the unlabeled covariate distribution is more dispersed than the target (covariate shift).
• Non-Linear Labelers: The framework extends to non-linear labelers by linearizing the predictor on an auxiliary sample drawn from the target marginal, ensuring the bias-aware step remains valid under model misspecification.

Key Contributions and Theoretical Results

1. Structural Limitation of PPI: The paper proves (Proposition 1) that under a correctly specified linear model, the PPI rectifier annihilates the predictor's contribution, reducing PPI to labeled-only OLS. This establishes that efficiency cannot come from the rectifier mean in linear settings.
2. Variance Channel Efficiency: DEAL routes the unlabeled covariates into the variance of the debiased estimator. Theoretical analysis shows that under a linear labeler, DEAL achieves a strictly shorter confidence interval than target-only debiased Lasso, PPI, and PPI++. The asymptotic variance ratio is given by $G(\kappa, t_0) = (\frac{1+t_0\kappa}{1+\kappa})^2$, where $\kappa = N/n_0$. When the external estimator is informative ($t_0 < 1$), this ratio is strictly less than 1.
3. Robustness to Misspecification and Non-Linearity:
   • Under model misspecification, DEAL provides valid inference for the population least-squares projection parameter $\beta^*_{proj}$.
   • Under non-linear labelers, DEAL strictly dominates optimally-tuned PPI++ at full unlabeled saturation, whereas PPI++ is capped by the irreducible labeled noise.
4. Covariate Shift Validity: The paper introduces a shift-aware modification that preserves nominal coverage even when the unlabeled covariates are drawn from a distribution with a larger covariance (shift-up) than the target, a regime where standard pooled estimators fail.
5. Adaptive Shrinkage: The data-driven shrinkage parameter $\hat{t}$ adapts to the quality of the external estimator, ensuring validity across regimes ranging from uninformative to near-oracle.

Empirical Results

The authors validate the theory through Monte Carlo simulations and a portfolio of six real-data applications spanning astronomy, materials chemistry, proteomics, and oncology.

• Simulation: In high-dimensional settings ($p \gg n_0$), DEAL confidence intervals were found to be between 0.49 and 0.87 of the length of target-only debiased Lasso intervals at nominal coverage. In contrast, PPI and PPI++ intervals remained close to the debiased Lasso baseline (ratios $\approx 1.0$).
• Real-Data Applications: Across six diverse applications (including one using a Large Language Model as a zero-shot oracle), DEAL consistently tightened confidence intervals. The median ratio of DEAL intervals to debiased Lasso intervals ranged from 0.23 to 0.53.
  • Reproducibility: DEAL's selected significant coordinates showed higher bootstrap stability (selection frequencies of 0.66–0.97) compared to debiased Lasso (0.26–0.63) and PPI++ (0.26–0.61).
  • Generalization: In applications with out-of-sample fit (galaxy morphology and band gap), DEAL's discoveries achieved positive out-of-sample $R^2$, whereas other methods yielded negative or near-zero $R^2$.
• Labeler Independence: Crucially, the efficiency gain was observed even when the external labeler had very low predictive accuracy (e.g., correlation $\approx 0.09$ in proteomics). When labelers were replaced by permuted (no-information) surrogates, DEAL's intervals collapsed back to the debiased Lasso baseline, confirming that the gain stems from the unlabeled design sharpening the precision matrix, not from the labeler's accuracy.

Significance and Claims

The paper claims that the primary contribution is a relocation of the external model's role: from the mean (via a rectifier) to the variance (via a debiased estimator). This shift allows the method to bypass the structural cancellation inherent in PPI for linear models.

The authors emphasize that DEAL is particularly relevant for the era of foundation models and synthetic data, where analysts often lack control over or the ability to audit the calibration of external predictors. By treating the external model as a "black box" and protecting inference against its bias via a shrinkage safeguard, DEAL converts an uncontrolled labeler into a harmless or beneficial tool. The method is claimed to be the sharper tool specifically within the high-dimensional sparse-linear regime, offering valid inference and strictly shorter intervals where prediction-powered methods fail to improve upon the labeled-only baseline.

The paper remains modest regarding its scope, acknowledging that:

• Under misspecification, it infers the best linear projection, not the true non-linear conditional mean.
• The efficiency gain is contingent on the labeled design being the binding constraint; if the labeled sample is already large or well-conditioned, the benefit recedes.
• The method relies on high-dimensional sparse-linear machinery (debiased Lasso, nodewise Lasso) and is not a general-purpose M-estimator like PPI.