Dependable Exploitation of High-Dimensional Unlabeled Data in an Assumption-Lean Framework

1. Problem Statement

The paper addresses the statistical inference problem in semi-supervised learning (SSL) settings characterized by:

• Data Structure: A small set of labeled data $(X_i, Y_i)_{i=1}^n$ and a large set of unlabeled data $(X_i)_{i=n+1}^{n+N}$.
• High-Dimensionality: The number of covariates $p$ is much larger than the sample size $n$ ($p \gg n$).
• Assumption-Lean Framework: The true conditional mean function $f(X) = E[Y|X]$ is unknown and potentially complex (nonlinear). The working model is a linear regression $Y \approx X^T\theta$, which may be misspecified.
• Goal: To estimate a linear functional of the regression parameter, $v^T\theta^*$, where $\theta^*$ is defined as the $L_2(P)$ projection of $f(X)$ onto the linear space spanned by $X$ (i.e., $\theta^* = \arg\min_\theta E[(f(X) - X^T\theta)^2]$).
• The Core Challenge: Existing semi-supervised estimators often rely on consistent estimation of the conditional mean function $f(X)$. If $f(X)$ cannot be estimated consistently (a common issue in high dimensions with complex data), these estimators may become less efficient than standard supervised estimators (using only labeled data), rendering the unlabeled data counterproductive. The authors seek an estimator that is guaranteed to be at least as efficient as the supervised baseline, regardless of whether $f(X)$ is correctly specified or consistently estimated.

2. Methodology

The authors propose a novel Dependable Semi-Supervised Estimator ($\hat{\theta}^{d}_{S,\psi}$) that avoids direct reliance on the consistency of $f(X)$.

A. Theoretical Foundation

Instead of estimating $f(X)$ directly to correct the bias, the method constructs a set of unbiased estimating functions.

1. Score Function: In the linear working model, the score function is $X(Y - X^T\theta)$.
2. Auxiliary Function: They introduce a $p$-dimensional function $m(X)$ (e.g., a flexible non-parametric fit or a specific basis expansion) and define a centered covariate $Z = X(m(X) - \mu)$, where $\mu = E[X m(X)]$.
3. Projection: They postulate a working regression model where the score function $X(Y - X^T\theta^*)$ is projected onto the space spanned by $Z$. This yields a coefficient matrix $B$:
   $$X(Y - X^T\theta^*) = B^T \{X m(X) - \mu\} + E$$
   Here, $E$ represents the residual. The key insight is that even if $m(X)$ is not the true $f(X)$, the term $X m(X) - \mu$ helps explain the non-linear part of the score function, reducing variance.

B. The Estimator Construction

The proposed estimator is a one-step debiased update of the supervised Dantzig selector ($\hat{\theta}_D$):
$$\hat{\theta}^{d}_{S,\psi} = \hat{\theta}_D + \hat{\Omega} \left( \hat{\xi}_{S,\psi} - \hat{\Sigma}_n \hat{\theta}_D \right)$$
Where:

• $\hat{\Omega}$ is an estimator of the precision matrix $\Sigma^{-1}$.
• $\hat{\Sigma}_n$ is the sample covariance matrix of the labeled data.
• $\hat{\xi}_{S,\psi}$ is a modified moment estimator that incorporates unlabeled data:
  $$\hat{\xi}_{S,\psi} = \frac{1}{n}\sum_{i=1}^n X_i Y_i - \psi \hat{B}^T \left( \frac{1}{n}\sum_{i=1}^n X_i \hat{m}(X_i) - \frac{1}{n+N}\sum_{i=1}^{n+N} X_i \hat{m}(X_i) \right)$$
• Tuning Parameter $\psi$: A scalar that balances the contribution of the labeled and unlabeled data. The authors prove that $\psi = 1$ minimizes the asymptotic variance.

C. Key Technical Features

• Cross-Fitting: To avoid overfitting and ensure asymptotic normality, the method uses sample splitting (cross-fitting) to estimate the nuisance parameters ($\hat{m}(X)$ and $\hat{B}$).
• No Consistency Requirement for $f(X)$: The estimator does not require $\hat{m}(X)$ to converge to the true $f(X)$. It only requires that the estimation error of $\hat{m}(X)$ is sufficiently small (controlled by a sequence $c_n$) and that the coefficient matrix $B$ is sparse.
• Variance Estimation: A consistent estimator for the asymptotic variance is constructed, allowing for valid confidence intervals and hypothesis testing.

3. Key Contributions

1. Dependability Guarantee: The primary contribution is an estimator that is provably no less efficient than the supervised debiased lasso estimator, even under model misspecification or when $f(X)$ cannot be consistently estimated. This solves the "efficiency loss" problem plaguing previous SSL methods.
2. High-Dimensional Inference: The method extends semi-supervised inference to the high-dimensional regime ($p \gg n$) under an assumption-lean framework, where the target parameter is defined via a linear projection rather than a true linear model.
3. Optimal Variance Reduction: Theoretical analysis shows that with $\psi=1$, the asymptotic variance is strictly smaller than the supervised estimator when the unlabeled data size $N$ is large relative to $n$ (specifically when $\lim n/(n+N) = \rho < 1$).
4. Generalization to M-Estimation: The framework is extended beyond least squares to general M-estimation problems.

4. Results

Theoretical Results

• Asymptotic Normality: The estimator $v^T \hat{\theta}^{d}_{S,\psi}$ is shown to be asymptotically normal.
• Efficiency Gain:
  • If the linear model is correct ($f(X) = X^T\theta^*$), the estimator achieves the same efficiency as the supervised debiased estimator.
  • If the model is misspecified, the estimator is strictly more efficient than the supervised estimator provided $0 < \psi < 2$ (optimal at $\psi=1$) and $N$ is sufficiently large.
  • Crucially, unlike "straightforward" debiased estimators (which fail if $f(X)$ is hard to estimate), this method guarantees efficiency gains without needing a consistent estimate of $f(X)$.

Numerical Simulations

• Setup: Simulations were conducted with $p \in \{200, 500\}$, $n \in \{100, 300, 500\}$, and $N \in \{n, 4n, 8n\}$. Two data generating models were used: a non-additive model (misspecified) and an additive model.
• Performance:
  • S-SSL (Proposed): Consistently achieved the lowest Standard Deviation (SD) and Root Mean Squared Error (RMSE) across scenarios.
  • Coverage: The 95% confidence intervals for S-SSL maintained coverage rates close to the nominal level (0.95), even under misspecification.
  • Comparison: The "straightforward" debiased estimator (D-SSL) often suffered from poor coverage and higher variance when the conditional mean model was misspecified. The proposed method remained robust.

Real Data Application

• Dataset: MIMIC-III (Intensive Care Unit electronic health records).
• Task: Predicting albumin levels using 162 clinical features.
• Setup: $n=500$ labeled, $N=4000$ unlabeled.
• Findings: The proposed S-SSL method successfully identified clinically relevant biomarkers (Total Calcium, Free Calcium, Iron Binding Capacity, RDW) with shorter confidence intervals compared to supervised methods. It detected signals (e.g., Free Calcium) that the supervised method missed, demonstrating superior power and efficiency.

5. Significance

This paper provides a paradigm shift in how unlabeled data is utilized for statistical inference in high-dimensional settings.

• Risk Mitigation: It removes the risk associated with using unlabeled data. Practitioners can now incorporate massive amounts of unlabeled data without fear of degrading inference quality if their predictive models are imperfect.
• Model Agnosticism: By operating in an assumption-lean framework, the method is applicable to complex, real-world data (like EHRs or genomics) where the true data-generating process is unknown and likely non-linear.
• Practical Utility: The method offers a "safe" way to improve statistical power and precision in fields where labeling is expensive but data collection is cheap, bridging the gap between machine learning (prediction) and statistics (inference).

In summary, the authors successfully constructed a dependable semi-supervised inference procedure that guarantees efficiency gains over supervised learning, regardless of the complexity of the underlying data distribution, making it a robust tool for modern high-dimensional data analysis.