📊 statistics

Predictive Power Analysis of Multiple Test Procedures Under Arbitrary Dependence

This paper introduces a novel Bayesian predictive power analysis method for multiple testing procedures under arbitrary dependence, utilizing a joint prior distribution of effect sizes and correlation matrices to facilitate sample size determination and assess significance-chasing biases without assuming independence.

Original authors: George Karabatsos

Published 2026-03-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: George Karabatsos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive case. You have 41 different clues (hypothesis tests) that might point to a culprit (a real effect). However, these clues are messy: some are related to each other (correlated), some are weak, and some might be false leads. Your goal is to figure out which clues are genuine without getting tricked by the noise.

In the world of statistics, this is called Multiple Testing. The problem is that when you look at 41 clues at once, you are very likely to find a "false alarm" just by chance. To fix this, statisticians use rules called Multiple Testing Procedures (MTPs) to filter out the noise.

This paper introduces a new, smarter way to check if your detective work (your study) is strong enough to catch the real clues before you even start looking. Here is the breakdown using simple analogies:

1. The Old Way: The "Rigid Blueprint"

Imagine you are building a house. The old way of planning your study (called Conditional Power Analysis) is like using a rigid blueprint where you assume:

The foundation will be exactly 10 feet wide.
The wind will blow exactly from the North at 10 mph.
The soil is perfectly flat.

If reality turns out to be different (the wind blows from the East, or the soil is rocky), your blueprint fails. In statistics, this means assuming the "effect size" (how strong the clue is) and the "correlation" (how the clues relate to each other) are fixed numbers. But in the real world, we rarely know these numbers exactly. We are guessing.

2. The New Way: The "Weather Forecast"

The author, George Karabatsos, proposes a Predictive Power Analysis. Instead of a rigid blueprint, imagine you are a weather forecaster. You don't say, "It will rain exactly 2 inches at 2 PM." Instead, you say, "There is a 70% chance of rain, but it could be a drizzle or a storm, and the wind could shift."

This new method uses a simulation (a computer game) to run thousands of "what-if" scenarios:

Scenario A: What if the effect is strong, but the clues are very tangled?
Scenario B: What if the effect is weak, but the clues are independent?
Scenario C: What if the clues are moderately related?

By running these thousands of simulations, the method gives you a probability of success rather than a single, fragile guess. It accounts for the fact that we don't know the exact "weather" of the data.

3. The "Arbitrary Dependence" Problem

Usually, statisticians get stuck because they don't know how the 41 clues are related. Are they like 41 people in a room talking to each other (highly correlated)? Or are they 41 people in separate rooms (independent)?

Old Methods: Often force you to assume they are independent or require you to know the exact relationship beforehand. If you guess wrong, your results are garbage.
The New Method: It says, "We don't need to know the exact relationship." It uses a mathematical trick (a Dirichlet Process) that allows the computer to consider every possible way the clues could be connected. It's like having a detective who is prepared for any social dynamic in the room, whether the suspects are whispering secrets or shouting in unison.

4. The "Significance Chasing" Bias

Sometimes, researchers (or the media) get excited about a result just because it looks "significant," even if it's a fluke. This is called "significance chasing" (or p-hacking).

The new method acts like a lie detector. It calculates a "Bias Index" for each clue.

If a clue is truly strong, the method says, "Yes, this is real, and it's not surprising."
If a clue is weak but the researcher is screaming "Look at me!", the method says, "Wait, this result is actually very surprising given what we know. It might be a fluke."

It helps you assign weights to your clues. If a clue is likely to be a fluke, you give it less weight. If it's robust, you give it more weight. This prevents you from building your house on a shaky foundation.

5. The Real-World Test: The Lead Exposure Study

To prove this works, the author tested it on a famous, old study about lead poisoning in children.

The Study: Researchers looked at 41 different ways lead exposure affected children's behavior and IQ.
The Problem: The original study was criticized because they didn't properly account for the fact that they were testing 41 things at once.
The Result: The author ran his new "Weather Forecast" simulation on this data. He found that while some results were still strong, the new method provided a much clearer picture of which results were likely real and which were likely just noise, without needing to know the exact relationships between the 41 tests.

Summary

Think of this paper as a new navigation system for scientific research.

Old GPS: "Turn left in 500 feet." (Fails if you miss the turn).
New GPS (This Paper): "There are many possible routes. Here is the probability of reaching your destination via Route A, Route B, or Route C, even if the traffic (correlations) is unpredictable."

It allows scientists to plan their studies better, know how many participants they need, and spot fake results, all without needing to know the impossible details of how their data is connected. It turns a rigid, fragile guess into a robust, flexible prediction.

1. Problem Statement

The paper addresses a critical gap in statistical methodology regarding Multiple Testing Procedures (MTPs). While MTPs are widely used to control the Family-Wise Error Rate (FWER) or False Discovery Rate (FDR) when testing multiple hypotheses simultaneously, existing methods for power analysis and sample size determination suffer from significant limitations:

Conditional vs. Predictive: Traditional power analysis is "conditional," assuming fixed effect sizes and a fixed correlation matrix for test statistics. This ignores the inherent uncertainty in these parameters, which is particularly problematic for future studies (e.g., replications).
Arbitrary Dependence: Most MTPs (like Bonferroni, Holm, and Benjamini-Yekutieli) are designed to control error rates under arbitrary (unknown) dependencies between p-values. However, standard power analysis methods often require the user to specify a fixed correlation matrix, which is difficult to estimate accurately, especially with heterogeneous test statistics or large numbers of hypotheses ( $m$ ).
Bias and Optimization: Current methods for optimizing p-value weights to maximize power often rely on fixed assumptions and do not adequately account for "significance-chasing" biases (e.g., publication bias, p-hacking) without assuming independence.

The core problem is the lack of a predictive power analysis framework that accounts for uncertainty in both effect sizes and the correlation structure of test statistics, specifically for MTPs that control errors under arbitrary dependence.

2. Methodology

The author proposes a novel Bayesian Predictive Power Analysis method based on simulation. The methodology introduces a joint prior distribution to model the uncertainty of the parameters driving the test statistics.

A. Statistical Framework

Test Statistics: The method models $m$ $m$ test statistics ( $T_1, \dots, T_m$ $T_{1}, \dots, T_{m}$ ) under alternative hypotheses using an m-variate upper non-central t-distribution ( $NT_m$ $N T_{m}$ ). This distribution is defined by:
- A location vector ( $\mu$ ) representing the ratio of effect sizes to standard errors.
- A scale matrix ( $\Sigma$ ), primarily driven by a correlation matrix ( $P$ ).
- A vector of degrees of freedom ( $\nu$ ).
Joint Prior Distribution: Instead of fixing parameters, the method assigns priors:
1. Effect Sizes ( $\mu$ ): Modeled via a multivariate normal distribution centered on anticipated or estimated effect sizes ( $\hat{\theta} / \hat{\sigma}_{\theta}$ ). This allows for shrinkage to correct for "winner's curse" (overestimation in prior studies).
2. Correlation Matrix ( $P$ ): Modeled using a uniform prior distribution (specifically the LKJ distribution with $\eta=1$ ). This is the key innovation, as it supports the entire space of possible correlation matrices, fully accounting for the lack of prior knowledge about dependencies between p-values.

B. Algorithm 1 (Simulation-Based Power Analysis)

The paper presents a simulation algorithm to estimate predictive power:

Sampling: Iterate $S$ $S$ times. In each iteration $s$ $s$ :
- Draw a location vector $\mu^{(s)}$ from the normal prior.
- Draw a correlation matrix $P^{(s)}$ from the uniform LKJ prior (using the efficient Pourahmadi & Wang algorithm).
- Generate test statistics $t^{(s)}$ from the $NT_m(\mu^{(s)}, P^{(s)}, \nu)$ distribution.
- Convert statistics to p-values.
MTP Application: Apply the chosen MTP (Bonferroni, Holm, Benjamini-Yekutieli, or the new Dirichlet Process MTP) to the simulated p-values at level $\alpha$ .
Aggregation: Calculate the proportion of rejections across $S$ $S$ iterations to estimate:
- Marginal Power: Probability of rejecting a specific hypothesis.
- Average Power: Mean marginal power.
- Disjunctive Power: Probability of rejecting at least one false null.
- Conjunctive Power: Probability of rejecting all false nulls.

C. Additional Outputs

Optimal Weights: The method derives p-value weights to maximize power while minimizing the impact of significance-chasing biases.
Bias Assessment: It calculates a Hellinger distance index to compare observed p-values against the predictive power distribution, quantifying the "surprise" of a result and detecting potential biases.

3. Key Contributions

First Predictive MTP Power Analysis for Arbitrary Dependence: This is the first method to provide predictive power analysis for MTPs (FWER/FDR controlling) without assuming a fixed correlation matrix or independence. It treats the correlation matrix as a random variable with a uniform prior.
Integration of Dirichlet Process (DP-MTP): The paper extends the application of the DP-MTP (a Bayesian nonparametric approach) to power analysis, allowing for uncertainty quantification in the selection of the MTP and its cutoff points.
Bias Detection Mechanism: The introduction of a Hellinger distance metric allows researchers to assess "significance-chasing" biases (like p-hacking) by evaluating how surprising a result is relative to its predicted marginal power.
Software Implementation: The method is implemented in the R package bnpMTP, making it accessible for practical application.
Theoretical Convergence: The paper proves (Theorem 5.1) that the simulation estimator converges almost surely to the true expectation via the Strong Law of Large Numbers, with a bounded Monte Carlo variance.

4. Results (Case Study)

The method was illustrated using data from the landmark Needleman et al. (1979) study on childhood lead exposure, which reported 41 hypothesis tests regarding neuropsychologic effects.

Setup: The analysis simulated a future replication study with the same sample sizes ( $N=158$ ) and 41 tests, using the observed test statistics as priors for effect sizes.
Comparisons: The predictive power of the DP-MTP was compared against Bonferroni (B-MTP), Holm (H-MTP), and Benjamini-Yekutieli (BY-MTP).
Findings:
- Power: The DP-MTP achieved an average predictive power of 0.25, slightly outperforming B-MTP (0.21), H-MTP (0.22), and BY-MTP (0.26).
- Bias Detection: The DP-MTP yielded lower "Significance Chasing" (Hellinger distance) indices for most tests compared to the other MTPs. This suggests that the DP-MTP, by accounting for the entire space of MTPs and correlations, is less susceptible to the biases inherent in fixed-cutoff procedures.
- Weighting: The method successfully generated p-value weights that identified significant discoveries with uncertainty quantification, whereas traditional methods provided binary (0/1) decisions.

5. Significance

This paper represents a paradigm shift in how power analysis is conducted for multiple testing scenarios:

Robustness: By removing the need to specify a fixed correlation matrix, the method is robust to the "unknown arbitrary dependence" that characterizes real-world data, particularly in meta-analyses or studies with heterogeneous test statistics.
Planning Future Studies: It provides a more realistic tool for sample size determination in replication studies, accounting for the uncertainty of effect sizes and the "winner's curse."
Scientific Integrity: The ability to quantify and minimize significance-chasing biases offers a new tool for improving the reproducibility and reliability of scientific findings in fields like psychology, education, and biostatistics.
Flexibility: The framework is applicable to any MTP (FWER or FDR controlling) and can be adapted for online testing or continuous hypotheses.

In summary, Karabatsos provides a comprehensive, simulation-based Bayesian framework that resolves the tension between controlling error rates under arbitrary dependence and performing rigorous, uncertainty-aware power analysis.