📊 statistics

Analysis of Stepped-Wedge Randomised Cluster Trial using a generalized pairwise comparison approach : a simulation study

Through a comprehensive simulation study, this paper identifies a hierarchical mixed-effects model (b4) and a cluster-restricted probabilistic index model (c2) as the most reliable generalized pairwise comparison approaches for analyzing stepped-wedge cluster randomised trials, demonstrating that these methods effectively control Type I error and maintain statistical power across diverse correlation structures and temporal trends.

Original authors: Yohan Bard, Emilie Presles, Marc Buyse, Silvy Laporte, Paul Zufferey, Frederikus A. Klok, Olivier Sanchez, Francis Couturaud, Edouard Ollier

Published 2026-03-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yohan Bard, Emilie Presles, Marc Buyse, Silvy Laporte, Paul Zufferey, Frederikus A. Klok, Olivier Sanchez, Francis Couturaud, Edouard Ollier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a new, fancy recipe for a community potluck actually makes the food taste better. But there's a catch: you can't just ask one person at a time. You have to test the recipe on entire neighborhoods (clusters) because if you tell one neighbor the secret, they might tell the next one, ruining the test. This is called a Cluster Randomized Trial.

Even worse, you can't cook the new recipe for everyone at once because your kitchen is too small. So, you roll it out slowly: Neighborhood A tries it in January, Neighborhood B in February, Neighborhood C in March, and so on. This is called a Stepped-Wedge Design.

Now, you want to know: "Is the new recipe better?" But "better" is complicated. Maybe it saves lives (huge win), maybe it prevents a stomach ache (small win), but maybe it causes a mild rash (a loss). How do you weigh a life saved against a rash?

This paper is about a new way to score these potlucks using a method called Generalized Pairwise Comparisons (GPC). Instead of averaging scores, you take every person from the "New Recipe" group and pair them up with every person from the "Old Recipe" group. You ask: "Who had the better outcome?"

If the New Recipe person had a life saved and the Old Recipe person didn't, that's a Win.
If the New Recipe person got a rash and the Old Recipe person didn't, that's a Loss.
If both had the same result, it's a Tie.

The paper asks: "When we do this pairing game with neighborhoods that switch recipes at different times, what's the best way to count the wins so we don't get fooled?"

The Problem: The "Time" and "Neighbor" Traps

The authors found that if you just count the wins simply, you get tricked by two things:

The Neighbor Effect: People in the same neighborhood tend to be similar (they eat the same food, live in the same climate). If you ignore this, you might think a small improvement is huge because you counted the same "neighborhood vibe" too many times.
The Time Effect: Maybe the weather gets colder in December, or people get sicker in winter, regardless of the recipe. Since Neighborhood A tried the recipe in January and Neighborhood B in June, the "time of year" might look like the recipe's fault.

The Simulation: A Massive Potluck Experiment

To solve this, the authors didn't just guess; they ran a massive computer simulation. They created 500 fake potlucks with 45 neighborhoods, switching recipes over 6 time periods. They tested 8 different ways to count the wins, changing the rules of the game to see which method survived the chaos.

They looked for two things:

Honesty (Type I Error): Does the method falsely claim the recipe is great when it's actually terrible? (Like a judge who always gives a gold medal).
Strength (Power): Can the method actually spot a great recipe when it exists?

The Results: The Two Champions

Out of the 8 methods, most failed. Some were too naive (ignoring the neighborhood effect), and some were too messy (getting confused by the time changes).

Only two methods stood out as the reliable champions:

1. The "Nested Ladder" Method (Model b4)

The Analogy: Imagine a school with 5 grades (sequences), and each grade has 9 classrooms (clusters). This method builds a ladder. It acknowledges that classrooms in the same grade are similar, but also that every single classroom has its own unique personality. It uses a "hierarchical" structure to weigh the wins carefully, ensuring that the "grade" and the "classroom" don't skew the results.
Why it works: It's like having a very strict referee who knows exactly how the groups are organized and adjusts the score accordingly. It's very reliable and works well even when the data is messy.

2. The "Strictly Local" Method (Model c2)

The Analogy: This method says, "Let's only compare people who live in the same neighborhood." It ignores comparisons between Neighborhood A and Neighborhood B entirely. It only looks at the people in Neighborhood A who got the new recipe versus the people in Neighborhood A who got the old one.
Why it works: By keeping the comparisons strictly local, it automatically cancels out the "neighborhood vibe" and the "time of year" because everyone in that specific comparison is experiencing the same context.
The Catch: It's incredibly powerful (it finds the best recipes easily), but it's computationally heavy. It's like trying to count every single handshake in a stadium; if the stadium is huge, your calculator might explode. It takes a lot of computer power to run.

The Real-World Test: The ETHER Trial

The authors tested these two champions on a real upcoming study called ETHER, which is trying to help doctors and patients decide on blood thinners. The study involves 45 hospitals switching to a new decision-making tool over time.

They simulated the ETHER trial using their new methods and found:

Both methods were honest (they didn't lie about the results).
The "Strictly Local" method (c2) was slightly better at spotting the truth, especially when the data was tricky.
However, if the study includes a "Patient Activation Score" (how engaged the patient feels) along with medical outcomes, the difference between the two methods shrinks, and both work great.

The Bottom Line

If you are running a complex study where groups switch treatments over time (like a stepped-wedge trial) and you want to measure complex outcomes (like lives saved vs. side effects):

Don't just count wins simply. You will get fooled by time and group similarities.
Use the "Nested Ladder" (b4) if you want a robust, reliable method that handles the data structure well without needing a supercomputer.
Use the "Strictly Local" (c2) if you have the computer power and want the absolute highest chance of detecting a real effect, especially if your groups are very similar to each other.

The paper essentially gives researchers a "User Manual" on how to play the "Win Counting Game" correctly so they don't accidentally declare a bad recipe a winner, or miss a good one.

1. Problem Statement

Stepped-Wedge Cluster Randomised Trials (SW-CRTs) are increasingly used to evaluate complex interventions, particularly when individual randomization is impractical or unethical. However, a significant methodological gap exists in analyzing composite endpoints within these designs using Generalized Pairwise Comparisons (GPC).

The Challenge: Standard GPC methods (e.g., Win Ratio, Win Odds) assume independent observations. In SW-CRTs, data exhibit two critical dependencies:
1. Clustering: Outcomes within the same cluster are correlated (Intra-class Correlation Coefficient, ICC).
2. Temporal Trends: Clusters transition from control to intervention at different times, creating secular time trends that confound treatment effects. Additionally, cluster effects may persist or vary over time (Cluster Autocorrelation Coefficient, CAC).
The Risk: Applying unadjusted GPC methods to SW-CRTs leads to underestimated variance, inflated Type I error rates, and biased treatment effect estimates due to the failure to account for both clustering and time.

2. Methodology

The authors conducted an extensive simulation study to evaluate the performance of eight analytical approaches for SW-CRTs using GPC.

Simulation Design

Structure: 45 clusters, 5 sequences, 6 time periods, with 10 individuals per cluster-period (mimicking the ETHER trial).
Data Generation: Binary outcomes generated via a logistic model with random effects for cluster ( $\eta_k$ ) and period-within-cluster ( $\zeta_{jk}$ ). Continuous outcomes (PAM-13 score) were generated via a linear mixed model.
Parameters: Varied Intraclass Correlation (ICC: 0–0.3), Cluster Autocorrelation (CAC: 0.5–1.0), secular time trends ( $\beta_t$ ), baseline risk ( $p_0$ ), and treatment effect sizes ( $\delta$ ).
Endpoint: A hierarchical composite endpoint (Mortality > VTE recurrence > Bleeding > Patient Activation).

Evaluated Methods

The study compared eight methods categorized by their adjustment strategies:

Unadjusted/Naive Approaches (Ignoring Time/Clustering):
- a1 (Crude Win Odds): Pools all treatment vs. control pairs.
- a2 (Cluster-Stratified Win Odds): Computes win odds within clusters and aggregates.
Cluster-Level Mixed-Effects Models (Analyzing Cluster-Period Pairs):
- b1: Random intercepts for clusters and period-pairs.
- b2: Adds random slopes for treatment at the cluster level.
- b3: Adds random slopes for treatment at the sequence level.
- b4 (Hierarchical Mixed-Effects): Nested random slopes (sequence-level and cluster-level within sequence). Includes Kenward-Roger correction for small samples.
Individual-Level Probabilistic Index Models (PIMs):
- c1 (Standard PIM): Regresses pairwise comparisons on covariates using all pairs (assumes independence).
- c2 (Cluster-Restricted PIM): Restricts pairwise comparisons to individuals within the same cluster to eliminate between-cluster heterogeneity.

3. Key Contributions

Methodological Extension: This is the first work to explicitly adapt the GPC framework (specifically Win Odds and PIMs) to account for the dual complexities of SW-CRTs: intra-cluster correlation and secular time trends.
Identification of Robust Estimators: The study identifies two specific methods that maintain valid Type I error control in SW-CRTs:
1. Model b4: A hierarchical mixed-effects model with sequence- and cluster-level random slopes.
2. Model c2: A cluster-restricted Probabilistic Index Model.
Practical Guidance: Provides a decision framework for researchers choosing between mixed-effects models and PIMs based on sample size, computational resources, and correlation structures.

4. Results

Type I Error Control

Failed Methods: Unadjusted methods (a1, a2) and simpler mixed models (b1, b2) exhibited severe Type I error inflation (often >20–30%) when ICC > 0 or time trends existed. Standard PIM (c1) failed when ICC > 0.1.
Successful Methods:
- Model b4: Consistently maintained the nominal 5% error rate across all ICC, CAC, and time trend scenarios.
- Model c2: Generally maintained correct error rates. It showed moderate inflation (up to 15%) only in extreme scenarios (high ICC = 0.3 and low CAC = 0.5), but remained superior to all other non-hierarchical methods.

Statistical Power

Comparison: Model c2 generally demonstrated higher statistical power than Model b4, particularly for small-to-intermediate treatment effects.
Drivers of Power:
- c2's advantage increased with higher ICC and lower CAC (strong within-cluster dependence, weak period stability).
- The presence of secular time trends further amplified c2's efficiency.
- For large treatment effects, both methods converged to similar power.
Composite Endpoint Impact: Power increased significantly with the richness of the composite endpoint. Adding the PAM-13 score (patient activation) to mortality, VTE, and bleeding raised power from ~50% to >90% in most scenarios.

Application to the ETHER Trial

Under the specific assumptions of the ETHER trial (low ICC 0–0.01, high CAC 0.75–1.0), both b4 and c2 achieved >90% power.
Model c2 approached 100% power, confirming the trial design is robust for detecting the pre-specified effects using GPC.

5. Significance and Implications

Validity in Complex Designs: The study proves that GPC can be rigorously applied to SW-CRTs, provided specific hierarchical or restricted modeling techniques are used. Ignoring time or clustering renders standard GPC invalid.
Trade-off Analysis:
- Model b4 (Mixed-Effects): Recommended for its robustness, ability to model random effects explicitly, and computational scalability. It is the "safe" choice for large datasets.
- Model c2 (Cluster-Restricted PIM): Recommended when maximizing power is critical and computational resources allow. It offers superior efficiency in high-correlation settings but suffers from high computational cost (quadratic increase in comparisons) and cannot explicitly model random effects (relying instead on within-cluster restriction).
Future Directions: The authors note that while linear time trends were modeled, future work should address non-linear trends (e.g., splines) and unequal cluster sizes. They also highlight that PIMs currently lack a mechanism to incorporate random effects directly, which is a limitation compared to mixed models.

Conclusion: For analyzing composite endpoints in Stepped-Wedge Cluster Randomised Trials, the authors recommend Model b4 (hierarchical mixed-effects) or Model c2 (cluster-restricted PIM) as the only reliable strategies to ensure valid inference and optimal power.