📊 statistics

FDP control in multivariate linear models using the bootstrap

This paper proposes a bootstrap-based method for post hoc inference of the False Discovery Proportion in multivariate linear models, which offers simultaneous asymptotic control over all hypothesis subsets, simpler theoretical justification, and greater statistical power than existing parametric approaches, as demonstrated through simulations and real-world applications in neuroimaging and transcriptomics.

Original authors: Samuel Davenport, Bertrand Thirion, Pierre Neuvial

Published 2026-09-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samuel Davenport, Bertrand Thirion, Pierre Neuvial

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern science, researchers often face a problem not of finding a single needle in a haystack, but of finding thousands of needles scattered across a field, while ensuring they do not mistake a piece of straw for a needle. This challenge is central to fields like brain imaging and genetics, where scientists measure thousands of signals simultaneously. In a brain scan, for instance, a computer might look at every tiny cube of tissue to see if it lights up during a specific task. In a genetic study, it might check thousands of genes to see if their activity changes with a disease. The standard way to handle this flood of data has been to control the "false discovery rate," a statistical safety net that limits the average number of mistakes across the entire study. However, this safety net has a blind spot: it guarantees the average error rate but does not promise that any specific group of findings is actually correct. A researcher might identify a cluster of active brain regions or a set of disease-related genes, only to find that a large, unpredictable portion of that specific group is actually noise. To truly trust a discovery, scientists need a way to say, "We are 95 percent sure that at least this many items in this specific group are real," regardless of how they chose that group.

This is the precise gap that Samuel Davenport, Bertrand Thirion, and Pierre Neuvial address in their recent work. They have developed a new method to provide these specific, reliable guarantees for groups of findings in complex statistical models. Their approach focuses on the "false discovery proportion," which is the actual percentage of mistakes within a chosen set of results, rather than just the average across all possible sets. To solve this, they turned to a technique called the bootstrap. Imagine a scientist who has collected data from a group of people and wants to know if a pattern they see is real or just a fluke. Instead of relying on rigid mathematical formulas that assume the data behaves in a perfectly predictable way, the bootstrap method creates thousands of fake versions of the dataset by randomly reshuffling the original data points. By analyzing these fake versions, the researcher can build a picture of what random noise looks like in their specific situation. The authors adapted this technique for the complex, multi-variable models used in brain and gene studies, proving that it works reliably even when the data points are deeply interconnected, as they often are in biology.

The researchers tested their method through rigorous computer simulations, creating artificial datasets that mimicked the messy, interconnected nature of real-world brain scans and gene expression data. They compared their bootstrap approach against the current standard methods, which rely on strict assumptions about how data points relate to one another. The results were clear: the new bootstrap method provided much tighter and more accurate bounds on the number of false discoveries. In many scenarios, the standard methods were overly cautious, warning researchers that their findings might be unreliable when they were actually quite solid. The bootstrap method, by learning the specific structure of the noise in the data, allowed researchers to claim with high confidence that a larger portion of their findings were genuine. For example, in simulations involving different levels of smoothness and varying numbers of subjects, the new method consistently kept the error rate at the desired level, whereas the older methods often fell short, either being too loose or too conservative depending on the data's complexity.

To demonstrate the power of this approach in a real-world setting, the team applied it to two distinct datasets. The first came from the Human Connectome Project, a massive collection of brain scans from 386 unrelated individuals who performed a working memory task. The researchers were interested in seeing which parts of the brain were active in relation to a person's sex and their intelligence quotient. Using their new method, they could confidently state that within specific clusters of active brain tissue, a high proportion of the voxels (the tiny 3D pixels of the scan) were truly active. When they compared this to the standard methods, the bootstrap approach identified significantly more active tissue with the same level of certainty. In the second application, they analyzed gene expression data from 135 patients with chronic obstructive pulmonary disease. Here, the goal was to find genes linked to lung function. The standard methods suggested that fewer than half of the genes identified as significant were likely to be true discoveries. In contrast, the new bootstrap method allowed the researchers to conclude with 90 percent confidence that at least 1,354 of the 1,745 genes flagged as significant were indeed active, a much more informative and useful result for medical researchers.

The significance of this work lies in its ability to handle the complex, dependent nature of biological data without making unrealistic assumptions. Traditional methods often assume that data points are independent or follow a specific, simple pattern of correlation, which is rarely true in the brain or the genome. By using the bootstrap to simulate the actual distribution of the data, the authors created a tool that is robust to these complexities. They also introduced a "step-down" version of their method, which iteratively refines the results to squeeze out even more power, though they found that in many real-world cases where true signals are rare, the standard version was already nearly as effective. The authors acknowledge that their method provides guarantees that hold as the amount of data grows, and their simulations showed that with a reasonable number of subjects, the error control is precise. This work offers a practical upgrade for scientists who need to move beyond broad averages and make precise, trustworthy statements about the specific discoveries they make in the noisy, interconnected world of modern biology.

Technical Summary: FDP Control in Multivariate Linear Models Using the Bootstrap

Problem Statement
The paper addresses the challenge of performing post hoc inference on the False Discovery Proportion (FDP) within multivariate linear models, particularly in contexts involving multiple contrasts (e.g., functional Magnetic Resonance Imaging and transcriptomics). While False Discovery Rate (FDR) control is a standard, it controls only the expected value of the FDP, which can lead to unreliable error control in the presence of dependence among test statistics. Furthermore, standard FDR procedures often assume positive regression dependence, an assumption that may not hold when testing multiple contrasts simultaneously at each feature (voxel or gene). Existing post hoc inference methods, such as those based on the Simes inequality (Goeman and Solari, 2011; Rosenblatt et al., 2018), provide probabilistic bounds on the FDP but rely on parametric assumptions regarding the dependence structure of the data. The authors identify a gap in obtaining joint error rate (JER) control for multiple contrasts in general linear models without relying on strong parametric dependence assumptions.

Methodology
The authors propose a non-parametric approach utilizing the residual bootstrap to simulate the joint null distribution of test statistics for multiple contrasts. The methodology proceeds as follows:

Framework: The data is modeled as random fields observed at a finite set of points $V$ (e.g., voxels or genes) under a linear model $Y_n(v) = X_n\beta(v) + E_n(v)$ . The goal is to test null hypotheses $H_{0,l}(v): c_l^T\beta(v) = 0$ for $L$ contrasts.
Bootstrap Procedure: The method employs a residual bootstrap. Residuals $\hat{E}_n$ are calculated from the fitted model. For each bootstrap iteration $b$ , residuals are resampled with replacement to create a new error field $E_n^b$ . New response fields are generated as $Y_n^b = X_n\hat{\beta}_n + E_n^b$ , and bootstrap parameter estimates $\hat{\beta}_n^b$ are computed.
Theoretical Justification: The authors prove the consistency of the multivariate residual bootstrap in the linear model. Unlike previous proofs (e.g., Eck, 2018) that relied on complex metric convergence arguments, this paper utilizes the Lindeberg Central Limit Theorem to demonstrate that the bootstrapped test statistics converge in distribution to the same Gaussian limit as the true test statistics under the global null.
Joint Error Rate (JER) Control: The bootstrap is combined with the post hoc inference framework of Blanchard et al. (2020). By estimating the $\alpha$ -quantile of the bootstrap distribution of a specific functional of the test statistics (related to the minimum of inverse template functions applied to p-values), the method determines a threshold $\lambda$ . This threshold defines a "canonical reference family" of rejection sets.
Step-Down Procedure: To improve power, a step-down algorithm (Algorithm 1) is introduced. It iteratively refines the set of hypotheses by removing those rejected at the current step, recalculating the threshold on the remaining set, and repeating until convergence.
FDP Bounds: Once the JER is controlled, the method provides simultaneous upper bounds on the number of false positives (and thus lower bounds on the True Discovery Proportion, TDP) for any data-driven subset of hypotheses.

Key Contributions

Theoretical Proof: The paper provides a simpler proof of the consistency of the multivariate residual bootstrap in linear models compared to existing literature (specifically Eck, 2018), relying on the Lindeberg CLT.
Asymptotic FDP Control: It establishes that the proposed bootstrap procedure yields asymptotic simultaneous control of the FDP over all subsets of hypotheses in the multivariate linear model setting.
Extension to Multiple Contrasts: The method extends post hoc inference capabilities to scenarios involving multiple contrasts, a setting where traditional permutation methods often fail due to the lack of exchangeability under the alternative hypothesis.
Algorithmic Implementation: The authors provide a step-down bootstrap procedure that maintains error rate control while potentially increasing power.

Results
The authors validate their method through simulations and real-world applications:

Simulations: Numerical simulations were conducted on 2D Gaussian random fields with varying sample sizes ( $n=20$ $n = 20$ to $100$), smoothness levels (FWHM 0, 4, 8), and proportions of true nulls ( $\pi_0$ $π_{0}$ ).
- Error Control: The bootstrap methods (both single-step and step-down) demonstrated that the empirical Joint Error Rate (JER) converges to the nominal level (e.g., 0.1) as the sample size increases. In contrast, parametric methods (Simes and ARI) were found to be conservative (substantially below the nominal level) when data dependence was present (non-zero smoothing).
- Power: The bootstrap-based approaches generally provided higher power (tighter bounds on the TDP) than the parametric alternatives, particularly in settings with dependence. The step-down bootstrap offered marginal improvements over the single-step version, though the difference was small in sparse signal scenarios.
Real Data Applications:
- Neuroimaging (HCP): Applied to 386 subjects performing an m-back working memory task. The bootstrap method yielded higher lower bounds on the proportion of active voxels within clusters for IQ and sex contrasts compared to Simes and ARI.
- Transcriptomics (COPD): Applied to a dataset of 135 subjects and 12,531 genes. The bootstrap approach concluded that at least 1,354 of the 1,745 genes identified by Benjamini-Hochberg were true positives (90% confidence), whereas Simes and ARI provided significantly lower bounds (917 and 966, respectively).

Significance and Claims
The paper claims that the proposed bootstrap method offers a robust alternative to parametric post hoc inference methods. Its primary significance lies in its ability to model the dependence structure within the data without requiring the "positive regression dependence" assumption necessary for Simes-based methods. Consequently, the bootstrap approach provides less conservative (more powerful) bounds in realistic scenarios where data exhibits dependence, such as neuroimaging and genomics.

The authors modestly note that the control of the FDP is asymptotic; while simulations show correct control for reasonable sample sizes ( $n \ge 20$ ), the method can be slightly conservative at very small sample sizes ( $n=10$ ) or low smoothness levels. Additionally, while the step-down procedure theoretically improves power, the authors observe that in real-world sparse data settings (where $\pi_0$ is close to 1), the practical improvement over the single-step method is negligible. The work extends the applicability of post hoc inference to multivariate linear models with multiple contrasts, a domain where exact permutation methods are often inapplicable due to the breakdown of exchangeability.

More like this