High-Dimensional Single-Index Models: Link Estimation and Marginal Inference

Technical Summary: High-Dimensional Single-Index Models: Link Estimation and Marginal Inference

Problem Statement
This study addresses the statistical challenges of estimation and hypothesis testing in high-dimensional single-index models (SIMs) where the sample size $n$ and the dimension of the coefficient vector $p$ are large and comparable ($p/n \to \kappa$). The model is defined as $E[y_i | X_i = x] = g(\beta^\top x)$, where $X_i \sim N_p(0, \Sigma)$, $\beta$ is an unknown coefficient vector, and $g(\cdot)$ is an unknown link function.

While single-index models offer advantages over nonparametric regression by mitigating the curse of dimensionality and avoiding model misspecification regarding the functional form, existing high-dimensional literature faces two primary gaps in this regime:

1. Link Function Estimation: Previous approaches often overlook the consistent estimation of the unknown link function $g(\cdot)$, or assume it is known (as in Generalized Linear Models).
2. Marginal Inference: There is a lack of rigorous methods for establishing the marginal asymptotic normality of individual coordinates of $\beta$ when $g(\cdot)$ is unknown, which is necessary for constructing valid confidence intervals and p-values for variable selection.

Methodology
The authors propose a novel three-step estimation procedure combined with a fourth step for inferential parameter estimation. To manage the complex dependency structures arising from data reuse, the dataset is theoretically split into two disjoint subsets, $I_1$ and $I_2$, though the authors note that in practice, using all data in each step is recommended.

1. Index Estimation (Pilot Step): Using the first subset $(X^{(1)}, y^{(1)})$, a pilot estimator $\tilde{\beta}$ (e.g., ridge regression, least squares, or MLE depending on the response type) is constructed. An index estimator $W_i$ for the true index $\beta^\top X_i$ is then derived using observable adjustment techniques (Bellec, 2022). This estimator corrects for bias and variance, yielding $W_i \approx \beta^\top X_i + \mathcal{N}(0, \tilde{\sigma}^2/\tilde{\mu}^2)$.
2. Link Function Estimation: Using the pairs $(y^{(1)}_i, W_i)$, the authors estimate $g(\cdot)$. Since $W_i$ contains Gaussian noise relative to the true index, this constitutes an "errors-in-variables" problem. The authors employ a deconvolution kernel technique (Stefanski and Carroll, 1990) combined with a Nadaraya-Watson estimator to asymptotically remove the attenuation bias. A monotonization operator (e.g., rearrangement operator) is applied to ensure the estimated link function $\hat{g}(\cdot)$ is monotonic, consistent with the model assumptions.
3. Coefficient Estimation: Using the second subset $(X^{(2)}, y^{(2)})$ and the estimated link $\hat{g}(\cdot)$, the authors minimize a surrogate loss function $\ell(b; x, y, \bar{g}) = \bar{G}(x^\top b) - y x^\top b$ (where $\bar{G}' = \bar{g}$). This yields the final coefficient estimator $\hat{\beta}(\hat{g})$.
4. Inferential Parameter Estimation: To facilitate statistical inference, estimators for the asymptotic bias $\mu^*$ and variance $\sigma^2_*$ are constructed using observable adjustments based on the estimated link function, following the framework of Bellec (2022).

Key Contributions

• Consistent Link Estimation: The paper provides a consistent estimator for the unknown link function $g(\cdot)$ in the high-dimensional proportional regime, addressing the errors-in-variables problem via deconvolution.
• Marginal Asymptotic Normality: The authors rigorously establish the marginal asymptotic normality for any finite subset of the coordinates of the coefficient estimator $\hat{\beta}(\hat{g})$. This result holds even when the ratio $\kappa = p/n > 1$, a regime where previous marginal normality results for SIMs were limited.
• Efficiency Improvement: By utilizing the consistently estimated link function, the proposed estimator is shown to be more efficient than estimators relying on potentially misspecified link functions or those that ignore the link structure entirely.
• Technical Extension: The proof strategy extends the arguments of Zhao et al. (2022) from logistic regression (MLE) to a broader class of unregularized M-estimators and incorporates the estimated link function.

Theoretical and Empirical Results

• Theoretical Guarantees:
  • Theorem 1: Establishes the uniform consistency of the link estimator $\hat{g}(\cdot)$ with a convergence rate of $O_p((\log n)^{-m/2})$, which is shown to be a lower bound.
  • Theorem 2 & 4: Prove that for any coordinate $j$, the statistic $T_j = \frac{\sqrt{p}(\hat{\beta}_j - \mu^*\beta_j)}{\sigma^*/\tau_j}$ converges in distribution to a standard normal $N(0, 1)$. This holds for both regularized ($p > n$) and unregularized ($p < n$) settings.
  • Corollary 3: Demonstrates that confidence intervals constructed using these asymptotic results achieve the correct coverage probability asymptotically.
• Numerical Experiments:
  • Simulations across various models (Logit, Poisson, Cubic, Piecewise) confirm that the index estimator follows the predicted Gaussian distribution.
  • The link function estimator shows decreasing squared loss as sample size increases.
  • The marginal distribution of the coefficient estimator closely matches the standard normal density, validating the theoretical asymptotic normality.
  • Efficiency comparisons indicate the proposed method outperforms standard estimators (like Least Squares or MLE with fixed links) in most scenarios, particularly when the true link is unknown or complex.
• Real Data Application: The method was applied to two UCI datasets (DARWIN and speech data) related to Alzheimer's disease. The proposed estimator consistently demonstrated lower estimated effective asymptotic variance compared to standard logistic regression.

Significance and Claims
The paper claims to provide the first rigorous framework for simultaneous link function estimation and marginal statistical inference in high-dimensional single-index models where $n$ and $p$ are comparable. By establishing marginal asymptotic normality, the authors enable valid hypothesis testing ($H_0: \beta_j = 0$) and confidence interval construction for individual features, which was previously unresolved for SIMs with unknown links in this regime. The methodology leverages observable adjustments and deconvolution techniques to handle the bias introduced by estimating the index and the link function, offering a robust tool for high-dimensional econometrics and statistical learning. The authors note that future work could extend these results to non-Gaussian covariates and multi-index models.