Confidence envelopes for the false discoveries with heterogeneous data

1. Problem Statement

In the field of multiple hypothesis testing, selective inference aims to provide statistical guarantees on the number of false discoveries (FD) for any subset of hypotheses selected by a user, even if that selection is data-dependent (a phenomenon known as "data snooping" or "p-hacking").

• The Goal: Construct a confidence envelope $\hat{V}(S)$, which is a simultaneous upper bound on the number of false discoveries $|S \cap H_0|$ for any subset $S \subseteq \{1, \dots, m\}$.
• The Limitation of Existing Methods: Most state-of-the-art methods (e.g., Genovese 2006, Blanchard et al. 2020, Katsevich & Ramdas 2020) assume homogeneity: that under the null hypothesis, all $p$-values follow a Uniform distribution $U[0,1]$.
• The Reality: In many applications (e.g., discrete data, binomial tests, weighted p-values), $p$-values are heterogeneous and discrete. Their distributions under the null are known but differ across hypotheses ($F_i \neq U[0,1]$).
• The Consequence: Applying homogeneous methods to heterogeneous data leads to conservative bounds (over-estimation of false discoveries), resulting in a significant loss of statistical power (fewer true discoveries detected).

2. Methodology and Framework

The paper bridges two major frameworks for constructing confidence envelopes:

1. The Inversion Procedure (Genovese 2006): Based on local tests for intersection hypotheses.
2. The Joint Error Rate (JER) Approach (Blanchard et al. 2020): Based on reference families and interpolation.

The authors develop a unified approach to handle heterogeneity through the following steps:

A. General Shortcuts for the Inversion Procedure

The inversion procedure is generally NP-hard to compute. The authors introduce generic shortcuts for local tests that can be expressed as a maximum over a threshold $t$ (e.g., $\phi_A = \max_t \mathbb{I}(\text{violation at } t)$).

• They define efficient statistics $\phi_{S,n,t}$ that allow computing bounds in polynomial time.
• They prove that for specific structures (like homogeneous Simes-like tests), these shortcuts are exact (coincide with the theoretical inversion procedure), providing optimal estimators for the number of true nulls ($m_0$).

B. Heterogeneous Probabilistic Inequalities

To handle non-uniform $p$-values, the authors derive new concentration inequalities tailored to heterogeneous distributions $F_i$:

1. Heterogeneous Bretagnolle Inequality: A refinement of the DKW inequality for independent variables with different known CDFs. It uses the mean of the individual CDFs ($\bar{F}$) rather than a worst-case uniform bound.
2. Heterogeneous Simes-like Inequality: Derived from step-up FDR control procedures (Döhler 2018). It constructs adaptive thresholds based on the vector of CDFs $H(s,t) = (F_i(t)/(1-F_i(s)))_i$.

C. Construction of Heterogeneous Envelopes

Using the new inequalities as local tests, the authors construct confidence envelopes in three standard settings:

1. Top-k Setting: Bounds on false discoveries in the set of the $k$ smallest $p$-values.
2. Simultaneous k-FWER: Bounds on false discoveries in sets defined by thresholds.
3. Deterministic Setting: Bounds for pre-specified subsets of hypotheses.

They propose both non-adaptive bounds (using a fixed worst-case estimator) and adaptive bounds (plugging in an over-estimator $\hat{m}_0$ derived from the data).

3. Key Contributions

1. Theoretical Unification: The paper establishes a formal equivalence between the inversion procedure and the JER approach in the homogeneous case, proving that adaptive JER methods (like those in MBR2024) are exact shortcuts.
2. New Heterogeneous Tools:
   • Introduction of a Heterogeneous Bretagnolle inequality for the Top-k setting.
   • Introduction of a Heterogeneous Simes-like inequality for the k-FWER setting.
3. Adaptive Procedures: Development of data-driven estimators for $m_0$ (the number of true nulls) that leverage the known heterogeneity ($F_i$). These estimators are plugged into the JER framework to create adaptive confidence envelopes.
4. Computational Efficiency: The proposed methods are computable in $O(m^2 \log m)$ or $O(m^3)$ time, making them feasible for large-scale problems (e.g., genomics), whereas exact inversion is exponential.
5. Handling Discreteness: The methods explicitly handle discrete $p$-values and ties, which often cause issues in continuous approximations.

4. Results (Numerical Experiments)

The authors validate their methods via simulations (using Fisher's exact tests on binomial data with varying sample sizes, creating heterogeneous $p$-value distributions).

• Power Gain: The heterogeneous envelopes consistently outperform their homogeneous counterparts (DKW and Simes).
  • In the Top-k setting, the Bretagnolle-based adaptive bound detects more true positives, especially when the signal is weak or the proportion of nulls is high.
  • In the k-FWER setting, the heterogeneous Simes bound is less conservative than the vanilla Simes bound when the signal is moderate to strong.
• Comparison:
  • For very small $p$-values, the heterogeneous Simes bound is the least conservative.
  • For larger $p$-values, the Bretagnolle bound often provides the tightest envelope.
  • The adaptive versions significantly outperform non-adaptive versions, demonstrating the value of estimating $m_0$ using heterogeneity.

5. Significance and Impact

• Bridging the Gap: This is the first work to systematically extend confidence envelope theory from the homogeneous (uniform) case to the heterogeneous (known distribution) case.
• Practical Utility: Many real-world datasets (e.g., genetic association studies with discrete phenotypes, clinical trials with small sample sizes) generate discrete, heterogeneous $p$-values. Current tools force researchers to use conservative homogeneous bounds, wasting statistical power. This paper provides the first rigorous, powerful alternative.
• Methodological Rigor: By proving that the new shortcuts are exact in the homogeneous limit and providing tight bounds in the heterogeneous case, the paper offers a robust theoretical foundation for selective inference in complex data environments.
• Software Implications: The methods are compatible with existing R packages (like DiscreteFDR) and can be integrated into workflows for fMRI analysis and gene expression studies where data snooping is prevalent.

In summary, the paper provides a mathematically rigorous and computationally efficient framework to recover statistical power lost due to data heterogeneity in selective inference, ensuring that confidence envelopes for false discoveries are both valid and sharp.