Resampling-based multi-resolution false discovery exceedance control

Technical Summary: Resampling-based Multi-resolution False Discovery Exceedance Control

Problem Statement
Multiple testing procedures are essential for controlling error rates when testing large numbers of hypotheses. While Familywise Error Rate (FWER) control ensures no false positives, it is often overly conservative. False Discovery Exceedance (FDX) control offers a more flexible alternative, ensuring that the False Discovery Proportion (FDP)—the fraction of false discoveries among all rejections—exceeds a prespecified threshold $\gamma$ with probability at most $\alpha$.

Existing resampling-based FDX methods, such as the Romano-Wolf procedure, are powerful under dependence but typically lack simultaneous guarantees. Conversely, existing simultaneous FDP methods (e.g., Hemerik et al., 2019) provide confidence envelopes for FDPs across various thresholds but are not optimized for a specific FDX target $\gamma$, often resulting in lower power for FDX control. Furthermore, many simultaneous methods rely on confidence envelopes with shapes that must be chosen independently of the data, a constraint that limits their adaptability and power.

Methodology
The paper proposes a resampling-based method that generalizes the popular maxT procedure to achieve multi-resolution FDX control. The method requires the same input as maxT: a matrix of resampled test statistics (via permutation or bootstrap).

1. Single-Step Procedure:
   The core of the method defines a rejection threshold $q$. For a set of transformations $\mathcal{G}$ (e.g., permutations), the method calculates a statistic $s_g$ for each transformation $g \in \mathcal{G}$:
   $$s_g = \sup \left\{ t \in \mathbb{R} : \frac{R(t, gX)}{R(t, X) \vee 1} > \gamma \right\}$$
   where $R(t, X)$ is the number of rejections at threshold $t$ for the observed data, and $R(t, gX)$ is the number of rejections for the transformed data. The term $R(t, gX)$ serves as a conservative estimate of the number of false positives. The final threshold $q$ is the $(1-\alpha)$-quantile of the values $\{s_g\}_{g \in \mathcal{G}}$.

2. Sequential Procedure:
   A sequential variant is introduced to uniformly improve power. It iteratively refines the threshold by identifying subsets of hypotheses that are likely to be true nulls and recalculating the threshold based on these subsets. This process converges to a limit threshold $q_{lim} \leq q$.

3. Multi-Resolution Guarantee:
   The method provides a simultaneous guarantee: with probability at least $1-\alpha$, for all thresholds $t \geq q$, the FDP is bounded by $\gamma$. This allows researchers to "zoom in" on subsets of the most significant hypotheses (e.g., the top $k$ rejections) and claim that their FDP is also bounded by $\gamma$, without needing to adjust $\alpha$ or $\gamma$. Notably, if the number of rejections is small enough (specifically $< \gamma^{-1}$), the method guarantees FWER control (FDP = 0) simultaneously with the FDX control.

Key Contributions

• Simultaneous FDX Control: The paper introduces the first method to provide simultaneous FDX control across multiple resolutions (thresholds) while maintaining power comparable to non-simultaneous methods like Romano-Wolf.
• Data-Dependent Envelopes: The method is formulated as a generalization of previous simultaneous approaches (Hemerik et al., 2019). A key theoretical innovation is the use of confidence envelopes whose shape depends on the data (specifically the observed rejection counts $R(t)$). Previous methods required data-independent envelope shapes to maintain validity, which restricted their power.
• Connection to maxT: The proposed method is shown to coincide with the single-step and sequential maxT procedures when $\gamma = 0$. As $\gamma$ increases, the method's power increases, allowing for a controlled fraction of false positives.
• Practicality: The method requires no tuning parameters beyond $\alpha$ and $\gamma$ (unlike Hemerik et al., 2019, which requires candidate envelopes and other parameters). It is implemented in the R package rFDP.

Results

• Simulations: In simulations with correlated data, the proposed method demonstrated power comparable to the Romano-Wolf procedure, which is considered the most powerful non-simultaneous FDX method. It significantly outperformed the Lehmann-Romano method (which is conservative under dependence) and the Hemerik et al. (2019) method when tuned for FDX control.
• Real Data Analysis:
  • Riboflavin Production: Applied to gene expression data, the method rejected 186 hypotheses at $\gamma=0.1$ with 95% confidence. Simultaneously, it provided guarantees that the top 19 hypotheses contained at most one false discovery, and the top 9 contained zero false discoveries (FWER control).
  • Music Reviews: Applied to album review scores, the method identified trends in artist scores. It demonstrated the ability to reject hypotheses at different resolutions (e.g., rejecting 22 hypotheses at $\gamma=0.2$ while simultaneously guaranteeing that the top 4 contained no false discoveries).

Significance and Claims
The paper claims that FDX methods can often be made simultaneous "almost for free" in terms of power loss. By allowing the confidence envelope shape to depend on the data, the method resolves a major limitation of previous simultaneous inference techniques. The authors emphasize that the method is valid under the same assumptions as the Romano-Wolf method (specifically, an invariance assumption on the test statistics of true null hypotheses). While the method assumes a conditional invariance for finite-sample validity, the authors note that this assumption may be unnecessary asymptotically and that simulations suggest the weaker unconditional invariance is often sufficient in practice.

The work bridges the gap between powerful, non-simultaneous FDX control and simultaneous FDP inference, offering a tool that allows researchers to explore data at varying levels of granularity (resolutions) while maintaining rigorous statistical guarantees.