Improving the adjusted Benjamini--Hochberg method using e-values in knockoff-assisted variable selection

1. Problem Statement

The paper addresses the problem of variable selection in high-dimensional linear regression ($Y = X\beta + \varepsilon$) where the number of predictors ($m$) is large, potentially comparable to or exceeding the sample size ($n$). The goal is to identify non-zero coefficients ($\beta_j \neq 0$) while controlling the False Discovery Rate (FDR).

• Context: Standard Benjamini-Hochberg (BH) procedures fail here because test statistics derived from Ordinary Least Squares (OLS) estimates often exhibit complex, arbitrary dependence structures.
• Existing Solutions:
  • Knockoff Filter (Barber & Candès): A powerful, p-value-free method that constructs "knockoff" variables to create a valid control mechanism. However, it can be conservative (low power) when signal strengths are weak or the target FDR is low.
  • Knockoff-Assisted BH (Sarkar & Tang): A two-stage method that uses knockoff variables to generate two sets of p-values ($P^{(1)}$ and $P^{(2)}$). It uses $P^{(1)}$ for screening and $P^{(2)}$ for the final BH decision. While more powerful than the standard knockoff filter in some regimes, it relies on a specific, rigid "all-or-nothing" calibration strategy.

The Gap: The authors aim to generalize the Sarkar & Tang approach by introducing e-values (a modern inferential tool) to create a more flexible, continuous, and powerful weighting scheme for the BH procedure.

2. Methodology

The proposed framework reinterprets the two-stage variable selection process through the lens of p-to-e calibration.

Core Framework

1. Data Generation: Two sets of statistics are generated using the original design matrix $X$ and the knockoff matrix $\tilde{X}$:

   • $\hat{\beta}^{(1)}$ and $\hat{\beta}^{(2)}$ are derived from linear combinations of $X$ and $\tilde{X}$.
   • These yield two sets of p-values: $P^{(1)}$ (from $\hat{\beta}^{(1)}$) and $P^{(2)}$ (from $\hat{\beta}^{(2)}$).
   • Crucially, under certain conditions (known $\sigma$ or specific degrees of freedom), $P^{(2)}$ is independent of $P^{(1)}$.

2. p-to-e Calibration:

   • The first set of p-values, $P^{(1)}$, is transformed into e-values ($S^{(1)}$) using a calibrator function $g$: $S^{(1)}_j = g(P^{(1)}_j)$.
   • An e-value $S$ satisfies $E[S] \leq 1$ under the null hypothesis. Large values indicate evidence against the null.
   • Innovation: Instead of the binary "all-or-nothing" calibrator used by Sarkar & Tang, the authors propose using bounded admissible calibrators (e.g., $g(t) = C(1 - t^a)$). This allows for a continuous, smooth weighting of hypotheses rather than a hard threshold.

3. Proposed Procedures:
   The paper introduces three specific algorithms:

   • Method 1 (ep-BH): Uses $S^{(1)}$ as unnormalized weights. The adjusted p-values are $\tilde{P}_j = P^{(2)}_j / S^{(1)}_j$. The standard BH procedure is applied to $\tilde{P}$. This offers exact finite-sample FDR control.
   • Method 2 (Adaptive ep-BH): Estimates the proportion of true nulls ($\pi_0$) using $P^{(2)}$ and incorporates this into the weighting. It uses adjusted p-values $P^*_j = \frac{\hat{\pi}_0 P^{(2)}_j}{I(P^{(2)}_j \leq \lambda) S^{(1)}_j}$. This offers asymptotic FDR control and improved power.
   • Method 3 (Adaptive Weighted BH): Standardizes the e-values to sum to $m$ to form weights $W_j$. It uses a weighted Storey estimator for $\pi_0$ and applies an adaptive weighted BH procedure. This also offers asymptotic FDR control.

3. Key Contributions

1. Unified Framework: The authors demonstrate that the Sarkar & Tang (Bon-BH) method is a special case of their general framework, corresponding to a specific, extreme "all-or-nothing" calibrator.
2. Bounded Calibrators: They introduce the concept of using bounded admissible calibrators for multiple testing weights. This avoids the variance inflation caused by extreme e-values and provides a more stable, flexible mechanism for incorporating prior evidence.
3. Theoretical Guarantees:
   • Method 1 provides exact FDR control at level $\pi_0 \alpha$ in finite samples.
   • Methods 2 and 3 provide asymptotic FDR control at level $\alpha$.
4. Generalization: The framework extends the applicability of knockoff-assisted methods to scenarios with low signal strength and sparse signals where traditional knockoff filters struggle.

4. Results

Simulation Studies

The authors compared their methods (M3, M4, M5) against the original Knockoff Filter (M0) and the Sarkar & Tang methods (M1, M2) across various settings ($n, m, k$, signal strength $\gamma$, correlation $\rho$).

• FDR Control: All proposed methods maintained FDR control at or below the nominal levels ($\alpha = 0.05, 0.1$). Method 1 showed strict control; Methods 2 and 3 showed strong empirical control.
• Power:
  • In low signal strength and low target FDR scenarios, the proposed methods (especially M3, M4, M5) significantly outperformed the original Knockoff Filter (M0), which was overly conservative.
  • The proposed methods also consistently outperformed the Sarkar & Tang methods (M1, M2) in terms of power, demonstrating that the continuous calibrator is superior to the binary one.
  • In high signal strength regimes, the proposed methods remained competitive with M0.

Real-World Application (HIV-1 Drug Resistance)

The methods were applied to a dataset analyzing HIV-1 protease inhibitor resistance (846 samples, 99 genetic positions).

• Performance: At $\alpha = 0.05$ and $\alpha = 0.1$, the proposed methods (M3, M4, M5) identified significantly more mutation positions than the standard Knockoff Filter (M0), which failed to detect any significant positions at $\alpha=0.05$.
• Validity: The discoveries included a high proportion of positions with prior biological support (TSM panel) as well as novel candidates, validating the method's ability to find true signals without excessive false positives.

5. Significance and Conclusion

This paper makes a significant contribution to the field of high-dimensional statistics by bridging knockoff inference and e-value theory.

• Practical Impact: It offers a robust alternative to the standard Knockoff Filter for practitioners dealing with sparse signals or strict FDR requirements, where the standard filter often lacks power.
• Theoretical Advancement: By identifying the "all-or-nothing" nature of previous knockoff-assisted BH methods as a specific case of a broader calibrator framework, the authors open the door for optimizing calibrator functions to maximize power while maintaining validity.
• Future Directions: The authors suggest extending this framework to Generalized Linear Models (GLMs) and exploring k-FDR control, indicating the versatility of their e-value weighting approach.

In summary, the paper proposes a flexible, e-value-weighted Benjamini-Hochberg procedure that leverages knockoff variables to achieve superior power in variable selection tasks, particularly in challenging regimes of weak signals and sparse data, while rigorously controlling the False Discovery Rate.