A Collinearity Test Statistic for Hypothesis Testing in High-Dimensional Generalized Linear Models

Technical Summary: A Collinearity Test Statistic for Hypothesis Testing in High-Dimensional Generalized Linear Models

Problem Statement
The paper addresses the challenge of hypothesis testing in Generalized Linear Models (GLMs) regarding the exclusion of relevant predictors. Specifically, given a set of $p$ predictors and a sample size $n$, the authors consider testing the null hypothesis $H_0$ that a hypothesized subset of predictors $S$ (where $|S| < n$) contains all non-zero regression coefficients, against the alternative $H_A$ that there exists a set of false exclusions $A$ (predictors with non-zero coefficients omitted from $S$).

The difficulty of this problem varies across three asymptotic regimes:

1. Classical ($p = o(n)$): Standard Likelihood-Ratio Tests (LRT) and Wald tests are effective.
2. Moderate ($p/n \to \kappa \in (0, 1/2)$): Classical tests fail; Sur and Candès [5] showed LRT statistics require a non-trivial rescaling factor derived from solving nonlinear equations.
3. High-Dimensional ($p > n$): Classical methods are inapplicable. Existing solutions involve partial penalized likelihood ratio tests (e.g., using SCAD/MCP penalties), which require solving difficult nonconvex optimization problems with no guarantees of finding the global solution.

The authors aim to propose a single test statistic that is computationally simple, avoids nonconvex optimization and nonlinear equation solving, and remains valid across all three regimes, including the $p > n$ case.

Methodology
The authors propose the Collinearity Test Statistic, denoted as $T_n$. The methodology is built upon the canonical exponential family GLM framework.

1. Definitions:

   • Let $\eta(\beta) = X\beta$ be the natural parameter.
   • Define the weight matrix $D(\beta) = \text{diag}(b''(\eta_i(\beta)))^{1/2}/\phi^{1/2}$ and the standardized response $V(\beta)$.
   • Define an adjusted response vector $Z(\beta) = D(\beta)\eta(\beta) + \sqrt{n}V(\beta)$.
   • Let $\hat{\beta}_S$ be the Maximum Likelihood Estimator (MLE) under the null hypothesis (restricted to $S$).
   • Define the projection matrix $\Pi_S(\beta)$ onto the span of the weighted design matrix $D(\beta)X_S$, and its orthogonal complement $\Pi_S^\perp(\beta)$.

2. The Statistic:
   For each predictor $v_i$ not in $S$ (i.e., $i \in S^c$), the authors define a collinearity metric $R_i$ measuring the cosine of the angle between the projection of the predictor and the projection of the adjusted response onto the orthogonal complement of the null model space:
   $$R_i = \frac{\langle \hat{\Pi}_S^\perp \hat{D}v_i, \hat{\Pi}_S^\perp \hat{Z} \rangle}{\|\hat{\Pi}_S^\perp \hat{D}v_i\| \|\hat{\Pi}_S^\perp \hat{Z}\|}$$
   The test statistic is the maximum squared collinearity across all excluded predictors:
   $$T_n = \max_{i \in S^c} n R_i^2$$

3. Distribution and P-value Estimation:

   • Theoretical Limit: Under $H_0$, $n R_i^2$ converges in distribution to $\chi^2_1$.
   • Critical Values:
     • For fixed $p$, a Bonferroni correction using the $(1 - \alpha/(p-s))$ quantile of $\chi^2_1$ provides a conservative rejection region.
     • For high dimensions ($p > n$), the authors derive an asymptotic approximation for the quantile: $q^{(n)} \approx 2 \ln(p_n/\alpha\sqrt{\pi}) - \ln(\ln(2p_n/\alpha))$. Theorem 2 establishes that this approximation holds under specific technical assumptions (e.g., $(\ln p_n)^4/n \to 0$).
   • Monte Carlo: For precise p-values (especially when $p$ is small to moderate), the authors recommend a Monte Carlo approach where the adjusted response $\hat{Z}$ is replaced by random Gaussian vectors to generate an empirical distribution of $T_n$.

Key Contributions

• Unified Statistic: The paper introduces a single test statistic applicable across classical, moderate, and high-dimensional regimes without modification.
• Computational Simplicity: Unlike competitors in the high-dimensional regime, the collinearity test requires neither the solution of nonlinear systems (as in the rescaled LRT) nor nonconvex optimization of penalized likelihoods. It relies on the same Iterative Reweighted Least Squares (IRLS) machinery used to compute the MLE.
• Theoretical Validity in High Dimensions: The authors prove that the $\chi^2_1$ approximation for the tail of the statistic remains valid even when $p > n$, provided specific regularity conditions are met.
• Sparsity Sensitivity: Theoretical analysis (Theorem 3) and numerical results indicate the test is particularly powerful when the alternative hypothesis is sparse (i.e., few predictors are missing).

Results
The authors provide numerical experiments using logistic regression with simulated data ($n=100$) across varying dimensions ($p$) and sparsity levels ($|A|$):

• Classical Regime ($p=12, n=100$): The collinearity test performs competitively with the LRT and Wald tests. It produces small p-values for sparse alternatives ($|A|=1$).
• Moderate Dimensions ($p=49, n=100$): In this regime, the LRT and Wald tests (computed with Jeffreys' prior penalties) show reduced power compared to the collinearity test, particularly for sparse alternatives.
• High Dimensions ($p=1000, n=100$): The collinearity test functions without modification. It demonstrates high power for sparse alternatives ($|A|=1$, median p-value $< 10^{-6}$) but lower power for non-sparse alternatives ($|A|=10$), consistent with its nature as a max-type ($L_\infty$) statistic.
• Overall Performance: The Wald test generally performed the worst. The collinearity test was not always the most powerful (e.g., LRT was superior in the classical regime with non-sparse alternatives) but was consistently competitive and superior in sparse, high-dimensional settings.

Significance and Claims
The paper claims that the collinearity test offers a practical, robust alternative for GLM hypothesis testing, particularly in scenarios where $p \ge n$. Its primary significance lies in its computational tractability and universality across asymptotic regimes.

The authors are modest in their claims, acknowledging:

1. Limitations: The test is not the most powerful across all regimes; tests designed specifically for a given regime (e.g., classical LRT for $p=o(n)$) may outperform it when assumptions are met.
2. P-value Estimation: While analytical bounds exist, precise p-value estimation often requires Monte Carlo simulation, incurring a computational cost of $O(nmp)$.
3. Scope of Theory: The theoretical proof of the Type 2 error decay rate (Theorem 3) is established for the Gaussian linear model. Extending these power results to non-Gaussian GLMs (like logistic or Poisson) is identified as an open problem for future investigation.

The authors conclude that the test is a viable, easy-to-implement tool that avoids the computational bottlenecks of existing high-dimensional methods while maintaining strong statistical power, especially in sparse settings.