Debiased Estimators in High-Dimensional Regression: A Review and Replication of Javanmard and Montanari (2014)

1. Problem Statement

In high-dimensional statistical settings where the number of covariates ($p$) significantly exceeds the number of observations ($n$), classical inference methods (confidence intervals, p-values) fail.

• The LASSO Limitation: The LASSO estimator is widely used for variable selection in $p \gg n$ settings due to its ability to induce sparsity via an $\ell_1$ penalty. However, this penalty introduces inherent bias and non-linearity, making the sampling distribution of the estimator difficult to characterize. Consequently, standard hypothesis testing and confidence interval construction are invalid.
• The Goal: To develop a framework that corrects the bias of the LASSO estimator to restore asymptotic normality, thereby enabling valid statistical inference (confidence intervals and hypothesis tests) in high-dimensional regimes.

2. Methodology

The paper reviews and replicates the framework proposed by Javanmard and Montanari (2014), which constructs a Debiased LASSO (also referred to as a debiased estimator).

A. Theoretical Framework

1. Model: Linear regression $Y = X\theta_0 + W$, where $W \sim N(0, \sigma^2 I)$.
2. Debiasing Mechanism: The method starts with the standard LASSO estimator $\hat{\theta}_n$ and adds a correction term proportional to the subgradient of the $\ell_1$ penalty.
   $$\hat{\theta}_u = \hat{\theta}_n + \frac{1}{n} M X^\top (Y - X\hat{\theta}_n)$$
   Here, $M$ is a "de-biasing matrix" designed to approximate the inverse of the sample covariance matrix ($\hat{\Sigma} = X^\top X / n$).
3. Optimization of $M$: To minimize the variance of the estimator, $M$ is constructed by solving a convex optimization problem for each column $m_i$:
   $$\min_{m} m^\top \hat{\Sigma} m \quad \text{subject to} \quad \|\hat{\Sigma}m - e_i\|_\infty \leq \mu$$
   This ensures that $M\hat{\Sigma} \approx I$ (the identity matrix), effectively "undoing" the correlation structure of the design matrix.
4. Theoretical Guarantees:
   • Error Decomposition: The estimation error is decomposed into a Gaussian noise term and a bias term ($\Delta$).
   • Asymptotic Normality: Under sparsity assumptions ($s_0 = o(\sqrt{n}/\log p)$) and compatibility conditions, the bias term vanishes asymptotically. The estimator $\sqrt{n}(\hat{\theta}_u - \theta_0)$ converges to a normal distribution $N(0, \sigma^2 M \hat{\Sigma} M^\top)$.
   • Inference: This normality allows for the construction of valid confidence intervals and p-values using standard normal approximations.

B. Empirical Approach

The author conducts a partial replication of the original study and extends it:

• Simulation: Replicates 6 scenarios using a circulant symmetric covariance matrix (low off-diagonal correlation).
• Comparators: Compares the Javanmard-Montanari (JM) method against:
  • Multisample splitting (Meinshausen et al.).
  • Ridge-type projection estimator.
  • LASSO Projection Estimator (van de Geer et al., Zhang and Zhang): A closely related method referenced but not implemented in the original 2014 paper.
• Real Data: Analyzes the riboflavin dataset ($n=71, p=4088$) to test performance on complex, real-world genomic data.

3. Key Contributions

1. Replication and Verification: Confirms the theoretical claims of Javanmard and Montanari (2014), demonstrating that the debiased LASSO achieves near-optimal confidence interval coverage and controls Type I error rates.
2. Introduction of the LASSO Projection Estimator: The paper implements and benchmarks the LASSO projection estimator (desparsified LASSO), which was previously only cited in the original work.
3. Identification of a Critical Trade-off: The study reveals a divergence in performance between idealized simulations and real-world data:
   • Simulations: The LASSO projection estimator demonstrated superior statistical power (True Positive rate) in low-signal settings compared to the JM method.
   • Real Data: The JM method (with its optimized $M$ matrix) outperformed the projection estimator in terms of precision and signal detection on the riboflavin dataset.
4. Robustness Analysis: Highlights that the JM method's explicit convex optimization for the de-biasing matrix adapts better to complex correlation networks found in biological data, whereas projection estimators excel in simpler, structured covariance environments.

4. Results

• Simulation Study:
  • Coverage: Both the JM method and the LASSO projection estimator achieved coverage probabilities close to the nominal 95% level.
  • Power: In low-signal regimes (e.g., signal strength $b=0.1$), the LASSO projection estimator showed significantly higher True Positive (TP) rates (e.g., 0.82 vs. 0.70 for JM) while maintaining controlled False Positive (FP) rates.
  • Interval Length: The replication produced tighter confidence intervals than the original 2014 study.
• Riboflavin Dataset Analysis:
  • Multisample Splitting: Produced overly conservative intervals covering the entire covariate range.
  • JM vs. Projection: The JM method yielded the narrowest confidence intervals and successfully isolated significant genes (e.g., ARGF_at, LYSC_at) that the Ridge and LASSO projection estimators missed or identified with less precision.
  • Conclusion: The JM method's ability to optimize the de-biasing matrix $M$ provided superior precision in the presence of complex, non-ideal correlation structures.

5. Significance

This paper provides a critical nuance to the field of high-dimensional inference:

• Theoretical Validation: It solidifies the theoretical foundation that debiasing LASSO allows for classical inference in $p \gg n$ settings.
• Practical Guidance: It challenges the assumption that one method is universally superior. The choice between the Javanmard-Montanari debiased estimator and the LASSO projection estimator depends on the data structure:
  • For idealized, low-correlation data, projection estimators may offer higher power.
  • For complex, real-world data (like genomics) with intricate correlation networks, the optimized de-biasing matrix of the JM method is more robust and precise.
• Future Directions: The author suggests that future simulation studies should utilize design matrices with stronger correlations (violating restricted eigenvalue conditions) to better reflect real-world challenges and further validate the necessity of the JM optimization approach.

In summary, the paper confirms that while simpler projection methods are powerful in theory, the computationally intensive optimization of the de-biasing matrix in the Javanmard-Montanari framework offers a distinct advantage in practical, noisy, high-dimensional applications.