📊 statistics

Weighted Holm Procedures: Theory, Properties, and Recommendations

This paper systematically compares the weighted Holm procedure (WHP) and the weighted alternative Holm procedure (WAP), demonstrating through theoretical analysis, graphical representations, and simulations that WHP is uniformly more powerful, monotonic, and optimal for controlling the familywise error rate without violating its constraints, leading to a recommendation for its preferential use over WAP.

Original authors: Beibei Li, Wenge Guo

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Beibei Li, Wenge Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge presiding over a courtroom with 100 different cases (hypotheses) to decide on at once. You have a strict rule: you can only make one wrong decision (convicting an innocent person) in the entire batch, or the whole trial is considered a failure. This is what statisticians call controlling the "Familywise Error Rate" (FWER).

However, not all cases are created equal. Some are high-profile murder trials (very important), while others are minor parking tickets (less important). You want to be extra careful with the murder trials, but you also want to give the parking tickets a fair shot.

This paper is about two different ways to organize your courtroom to handle these cases fairly while keeping your "one mistake" rule intact. The authors, Beibei Li and Wenge Guo, compare two methods: WHP and WAP.

Here is the breakdown in simple terms:

The Two Methods: Sorting the Cases

Imagine you have a pile of evidence (p-values) for all 100 cases. You need to decide which ones to reject (convict) and which to keep.

1. The "Weighted" Method (WHP): The Priority Sort

How it works: Before you even look at the evidence, you assign a "priority score" (a weight) to each case. A murder trial gets a high score; a parking ticket gets a low score.
The Strategy: You mix the evidence with the priority score. You create a "weighted score" for every case. Then, you sort the cases from the lowest weighted score to the highest.
The Logic: You tackle the most "important" cases first. If a murder trial has weak evidence, you might still reject it because its priority is so high. You are essentially saying, "Because this case matters so much, I will lower the bar slightly for it to get a closer look."
The Result: This method is smarter and stronger. It finds more guilty parties (true discoveries) without making more mistakes.

2. The "Alternative" Method (WAP): The Raw Sort

How it works: You ignore the priority scores when sorting. You look strictly at the raw evidence (the p-values) and sort the cases from the weakest evidence to the strongest, regardless of whether it's a murder trial or a parking ticket.
The Strategy: You treat the sorting as "objective." You say, "The evidence speaks for itself."
The Catch: Once you sort them, you apply the priority scores to decide how strict the bar is for each one.
The Problem: This creates a weird disconnect. You might sort a low-priority parking ticket first because its evidence looks slightly better than a murder trial's, but then you apply a very strict rule to it because it's "low priority." It's like putting a VIP in the back of the line just because they arrived a few seconds later, then treating them like a VIP anyway. This inconsistency makes the method less powerful.

The Big Discovery: WHP Wins Every Time

The authors ran simulations (like running the courtroom 5,000 times with different scenarios) and found a clear winner:

WHP (The Priority Sort) is uniformly more powerful. This means it catches more true effects (finds more guilty people) than WAP, without ever breaking the "one mistake" rule.
WAP (The Raw Sort) is sometimes okay, but it often misses things that WHP would have caught. It's like using a net with bigger holes; you let some fish escape.

Why Does WHP Work Better? (The Analogy)

Think of the "weights" as magnifying glasses.

WHP puts the magnifying glass on the most important cases before you look at them. It helps you see the details of the important cases immediately.
WAP looks at everything with the naked eye first, decides what to look at, and then tries to use the magnifying glass. By the time you get to the important cases, you might have already missed the subtle clues because you were looking at the wrong things first.

The "Graphical" Picture

The paper also introduces a cool way to visualize this using a flowchart (a graph).

Imagine a bucket of water (your total allowed error rate, $\alpha$ ) being poured into different cups (the hypotheses).
When a cup is "rejected" (a case is solved), the water from that cup flows into the remaining cups, making them easier to solve.
WHP flows the water in a way that always helps the most important cups get filled up faster.
WAP flows the water based on a different rule that sometimes leaves the important cups with less water than they could have had.

The Bottom Line for Real Life

If you are a researcher, a doctor, or a data analyst running a clinical trial (like testing a new drug):

Use WHP (The Weighted Holm Procedure). It is the "Gold Standard." It is mathematically proven to be the best way to handle weighted hypotheses. It gives you the best chance of finding real effects while keeping your error rate safe.
Avoid WAP unless you have a very specific, rare reason to do so. It is an older method that, while "objective" in how it sorts, ends up being less effective at finding the truth.

In short: Don't let the "objectivity" of raw numbers fool you. When you know some things are more important than others, let that importance guide your sorting process. WHP does exactly that, making it the superior choice for modern science.

1. Problem Statement

In clinical trials and statistical applications, hypotheses often possess varying levels of importance. To address this, Weighted Multiple Testing Procedures (wMTPs) are employed to control the Familywise Error Rate (FWER) while prioritizing specific hypotheses. Two prominent step-down procedures are widely used:

Weighted Holm Procedure (WHP): Orders hypotheses based on weighted p-values ( $\tilde{P}_i = P_i / w_i$ ).
Weighted Alternative Holm Procedure (WAP): Orders hypotheses based on raw p-values ( $P_i$ ), regardless of weights, but adjusts rejection thresholds using weights.

While WAP is often perceived as more "objective" because its ordering is independent of weights, it lacks certain desirable statistical properties. The paper aims to systematically compare WHP and WAP to determine which is superior in terms of power, structural properties (monotonicity, consonance), and optimality, providing clear recommendations for practitioners.

2. Methodology

The authors employ a multi-faceted theoretical and empirical approach:

Closed Testing Procedures (CTPs): Both WHP and WAP are reformulated as CTPs. This allows for a rigorous comparison of their local tests and global properties.
- WHP is shown to be equivalent to a CTP using weighted Bonferroni local tests.
- WAP is shown to be equivalent to a CTP using a specific local test based on the minimum p-value in an intersection set relative to the sum of weights.
Graphical Representations: The authors develop graphical algorithms for both procedures using a common initial graph structure (nodes representing hypotheses, edges representing transition coefficients). The key distinction lies in the selection step: WHP selects the node with the smallest weighted p-value, while WAP selects the node with the smallest raw p-value.
Adjusted P-values: Analytical formulas are derived for adjusted weighted p-values (for WHP) and adjusted p-values (for WAP) to facilitate direct comparison with the significance level $\alpha$ .
Optimality Analysis: The paper investigates whether the critical values of these procedures can be increased without violating FWER control. This involves constructing Least Favorable Configurations (LFC) under arbitrary dependence.
Simulation Studies: Extensive simulations were conducted under equicorrelated normal distributions with varying sample sizes, correlation structures ( $\rho$ ), proportions of true nulls ( $\pi_0$ ), and weight scenarios (informative vs. non-informative).
Real Data Analysis: Two clinical trial case studies (ARDS and Type 2 Diabetes) were analyzed to demonstrate practical performance.

3. Key Contributions and Theoretical Results

A. Power Dominance

The paper establishes a definitive theoretical result: WHP is uniformly more powerful than WAP.

Theorem 3.1: If a hypothesis is rejected by WAP, it is necessarily rejected by WHP. The converse is not true; there exist configurations where WHP rejects hypotheses that WAP fails to reject.
Proposition 5.1: The adjusted weighted p-values under WHP are always less than or equal to the adjusted p-values under WAP for any hypothesis.

B. Structural Properties: Monotonicity and Consonance

Monotonicity: WHP satisfies p-value monotonicity (lowering a p-value never decreases the number of rejections) and satisfies the monotonicity condition in its CTP representation.
WAP Deficiency: WAP lacks monotonicity under both definitions. Lowering a p-value can paradoxically reduce the number of rejections in WAP.
Consonance: Despite lacking monotonicity, the CTP representation of WAP is consonant (if an intersection hypothesis is rejected, at least one component hypothesis is rejected). This is a surprising property, as consonance is often linked to monotonicity. WHP is also consonant.

C. Optimality Results

WHP Optimality: WHP is unconditionally optimal. No critical value can be increased without violating FWER control under arbitrary dependence. It is the uniformly most powerful procedure among all weighted p-value step-down procedures controlling FWER.
WAP Optimality: WAP is conditionally optimal. It is optimal only if the ratio of the smallest weight to the largest weight satisfies $\frac{\min w_i}{\max w_i} \geq \alpha$ . If weights are highly disparate (e.g., one weight is much larger than others), WAP loses its optimality.

D. Graphical Framework

The authors provide a unified graphical framework for both procedures. This visualization clarifies that the difference between WHP and WAP is solely the selection criterion (weighted vs. raw p-values) in Step 2 of the algorithm, while the updating mechanism (redistribution of significance levels) remains identical.

4. Simulation and Empirical Results

FWER Control: Both procedures strictly controlled FWER at the nominal level $\alpha = 0.05$ across all scenarios.
Power Comparison:
- WHP consistently outperformed WAP in terms of average power across all correlation levels, sample sizes, and weight settings.
- The power advantage of WHP was most pronounced when weights were informative (i.e., weights for false nulls were significantly larger than for true nulls) and when the proportion of false nulls was high.
- Under non-informative weights (weights drawn from the same distribution for null and non-null hypotheses), the unweighted Holm procedure sometimes outperformed both weighted methods, but WHP still generally maintained better or comparable power to WAP.
Clinical Examples:
- In the ARDS trial, both methods rejected all four hypotheses, but WHP yielded smaller adjusted p-values.
- In the Type 2 Diabetes trial, WHP rejected four hypotheses ( $H_1, H_2, H_3, H_4$ ), whereas WAP only rejected two ( $H_1, H_3$ ). This demonstrated WHP's ability to detect additional treatment effects while maintaining strict error control.

5. Significance and Recommendations

Recommendation: The authors strongly recommend WHP as the method of choice for weighted multiple testing in practice. It offers superior power, unconditional optimality, and desirable monotonicity properties.
Practical Impact: The lack of monotonicity in WAP can lead to counter-intuitive results where improving a p-value leads to fewer rejections. WHP avoids this pitfall.
Communication: The graphical representations provided allow statisticians to explain the procedures to non-statisticians (e.g., clinical teams) more intuitively than complex decision tables.
Future Work: The paper suggests extending these weighted procedures to structured hypotheses (e.g., one- or two-way groupings) to further enhance applicability in complex clinical trial designs.

In conclusion, this paper resolves the ambiguity between WHP and WAP, demonstrating that WHP is theoretically and empirically superior, providing a robust framework for weighted hypothesis testing in clinical research.