Two-Stage Multiple Test Procedures Controlling False Discovery Rate with auxiliary variable and their Application to Set4Delta Mutant Data

1. Problem Statement

The paper addresses the challenge of multiple hypothesis testing where researchers have access to auxiliary variables (covariates) in addition to the primary test statistics (e.g., p-values or log-fold changes).

• Limitations of Existing Methods: Traditional covariate-assisted methods (e.g., AdaFDR, IHW) often rely on restrictive assumptions, such as the p-value following a uniform distribution under the null hypothesis regardless of the covariate, or satisfying a "mirroring property" (symmetry around 0.5). These assumptions often fail in real-world biological data where the auxiliary variable (e.g., standard deviation of measurements) is dependent on the primary variable.
• Goal: To develop flexible methodologies that incorporate auxiliary variables to increase statistical power (detect more true positives) while strictly controlling the False Discovery Rate (FDR), without imposing rigid distributional assumptions on the p-values.

2. Methodology

The authors propose a framework based on modeling the joint distribution of the primary variable ($\beta$) and the auxiliary variable ($y$) using Copula functions.

A. Data Modeling

• Primary Variable ($\beta$): Modeled as a mixture of null ($f_0$) and alternative ($f_1$) distributions. The null distribution $f_0$ is estimated from the data (e.g., using a mixture of normals).
• Auxiliary Variable ($y$): Treated as a random variable with an estimated marginal distribution (often non-parametric, e.g., empirical CDF).
• Joint Distribution: Instead of assuming independence or specific conditional symmetries, the dependence structure between $\beta$ and $y$ is modeled using a Copula ($G_0$). The marginal distributions are transformed into uniform p-values ($p_{1i}$ for auxiliary, $p_{2i}$ for primary), and the Copula captures their dependency.

B. Proposed Two-Stage Procedures

The paper introduces two distinct approaches to aggregate the auxiliary and primary information into a single test statistic while controlling FDR:

1. Two-Stage FDR(H) (Hard Thresholding):

   • Mechanism: A screening process. A tuning parameter $\gamma_1$ is set.
   • Step 1: If the auxiliary p-value $p_{1i} > \gamma_1$, the hypothesis is immediately accepted (not rejected), regardless of the primary p-value.
   • Step 2: If $p_{1i} \le \gamma_1$, the primary variable is tested. The combined p-value $p^H_i$ is calculated using the Copula distribution $C(\gamma_1, p_{2i})$.
   • Characteristics: Acts as a "filter." It creates a rectangular rejection region. It requires selecting an optimal $\gamma_1$ to maximize discoveries.

2. Two-Stage FDR(S) (Soft Thresholding):

   • Mechanism: A continuous adjustment process.
   • Step: The rejection criteria for the primary variable are adjusted based on the specific value of the auxiliary variable using the conditional distribution of the primary p-value given the auxiliary p-value: $p^S_i = P(P_{2i} \le p_{2i} | P_{1i} = p_{1i})$.
   • Characteristics: No hard cutoff is applied to the auxiliary variable. The rejection region is curved (non-rectangular), allowing genes with "weaker" auxiliary signals to still be rejected if their primary signal is strong enough, and vice versa. This method does not require tuning a parameter like $\gamma_1$.

C. Theoretical Guarantees

• Uniformity: The authors prove (Theorems 3.1 and 3.2) that the aggregated p-values ($p^H_i$ and $p^S_i$) follow a Uniform(0,1) distribution under the global null hypothesis ($G_0$), regardless of the dependence structure modeled by the Copula.
• FDR Control: Because the aggregated p-values are uniform under the null, standard FDR control procedures (like the Benjamini-Hochberg procedure or Storey's method) can be applied to these new p-values to guarantee FDR control.

3. Key Contributions

1. Relaxation of Assumptions: The methods do not require the p-value to be uniform under the null conditional on the covariate, nor do they require the mirroring property. This makes the approach applicable to a wider range of real-world data where such symmetries do not exist.
2. Copula-Based Dependence Modeling: By explicitly modeling the joint distribution via Copulas, the methods capture complex dependencies between experimental variability (auxiliary) and effect size (primary).
3. Two Distinct Strategies: The introduction of both "Hard" (screening) and "Soft" (continuous adjustment) thresholding provides flexibility depending on the scientific context (e.g., whether a strict biological precondition is required).
4. Robustness: Simulation studies show that even if the Copula is misspecified (e.g., using a Joe copula instead of the true Clayton copula), the methods generally maintain FDR control, though power may vary.

4. Results

Simulation Studies

• Setup: Simulated 8,000 genes with varying degrees of dependence (Kendall's $\tau$) between primary and auxiliary variables.
• Performance:
  • FDR Control: Both proposed methods effectively controlled FDR at the nominal level (e.g., $\alpha=0.05$).
  • Power (TPR): The proposed two-stage methods significantly outperformed existing one-stage methods (locfdr, Storey) and other covariate-assisted methods (IHW, Boca & Leek) in terms of True Positive Rate (TPR), especially as the dependence between variables increased.
  • Comparison: Two-stage FDR(S) generally achieved the highest power, followed by FDR(H).

Real Data Application (Set4∆Mutant Data)

• Context: Analysis of Saccharomyces cerevisiae (yeast) gene expression under oxidative stress, comparing Wild Type (WT) vs. Set4 Knockout (KO).
• Variables:
  • Primary: Log-fold change ($\beta$).
  • Auxiliary: Standard deviation of log-fold changes across replicates (representing experimental variability).
• Findings:
  • Selection of Copula: The Clayton Copula was selected as the best fit for the data based on AIC/BIC criteria.
  • Gene Discovery:
    • One-stage methods (locfdr, Storey) identified ~250–424 genes.
    • Two-stage FDR(H) identified 485 genes.
    • Two-stage FDR(S) identified 582 genes.
  • Biological Validation: The two-stage FDR(S) method successfully identified the entire HXT gene family (hexose transporters), which are known to be crucial for stress response and glucose transport under hypoxia. Other methods missed several members of this family.
  • Conclusion: The proposed methods identified a biologically relevant cluster of genes regulated by Set4 that were missed by traditional approaches, demonstrating superior sensitivity without inflating false discoveries.

5. Significance

• Statistical Advancement: The paper provides a rigorous theoretical framework for incorporating auxiliary information into multiple testing without relying on the often-unrealistic assumption of conditional uniformity or symmetry.
• Practical Impact: In genomics and other high-throughput fields, experimental variability (noise) is a critical factor. By using the standard deviation (or other quality metrics) as an auxiliary variable, researchers can filter out noisy data or down-weight unreliable signals more intelligently than simple filtering.
• Reproducibility: The methods enhance the credibility of gene expression analysis by ensuring that discoveries are not just statistically significant but also robust against experimental variability, leading to more reproducible biological findings.
• Generalizability: While demonstrated on yeast data, the methodology is applicable to any domain involving multiple testing with auxiliary covariates (e.g., clinical trials, finance, imaging).