📊 statistics

Who's Winning? Clarifying Estimands Based on Win Statistics in Cluster Randomized Trials

This paper highlights that in cluster randomized trials with informative cluster sizes, individual-pair and cluster-pair win estimands can yield substantially different or even contradictory conclusions, necessitating careful specification of the target estimand and the use of appropriate consistent estimators and variance methods to ensure valid inference.

Original authors: Kenneth M. Lee, Xi Fang, Fan Li, Michael O. Harhay

Published 2026-02-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kenneth M. Lee, Xi Fang, Fan Li, Michael O. Harhay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a cooking competition. Your goal is to decide which team (Team A or Team B) makes better food.

In a standard contest, you taste every single dish from every single chef, one by one. You count how many times a Team A dish beats a Team B dish. This gives you a clear "Win Ratio." If Team A wins 2 out of every 3 comparisons, you declare them the winner. This is how most medical trials work when they compare patients individually.

But what happens when the competition isn't just about individual chefs? What if the teams are entire restaurants?

The Problem: The "Big Restaurant" Trap

In a Cluster Randomized Trial (CRT), researchers don't randomize individual patients; they randomize whole groups (clusters), like hospitals, schools, or entire communities.

The paper by Lee and colleagues points out a massive confusion that happens when these groups are different sizes. Imagine:

Team A has one giant restaurant with 1,000 chefs.
Team B has 99 tiny cafes with only 1 chef each.

If you simply count every single dish (every single patient) to see who wins, the giant restaurant's 1,000 dishes will drown out the 99 tiny cafes. The result will be entirely determined by that one big restaurant.

The authors call this "Informative Cluster Size." It means the size of the group tells you something about the outcome. Maybe the big hospital is actually worse at treating patients, but because it has so many patients, its bad results dominate the math. Or maybe the small clinics are amazing, but because they are small, their success gets ignored.

The Two Ways to Judge (The Two "Win Statistics")

The paper explains that there are two completely different ways to ask the question "Who is winning?" and they can give you opposite answers.

1. The "Individual-Pair" Approach (Counting Every Plate)

The Metaphor: You taste every single plate of food from every single chef, regardless of which restaurant they work for.
The Math: You give equal weight to every single patient.
The Result: If the big restaurant has 1,000 patients and they do well, this method says "Team A is a huge winner!" Even if the other 99 restaurants did better on average, their small size makes them invisible in this count.
The Risk: The paper warns that in small studies, this method can be a bit "jittery" (biased) and might give you a slightly wrong answer just by chance.

2. The "Cluster-Pair" Approach (Counting Every Restaurant)

The Metaphor: You pick one random dish from the Giant Restaurant and one random dish from a Tiny Cafe. You compare them. Then you pick another pair. You treat the Giant Restaurant and the Tiny Cafe as equals. You don't let the Giant Restaurant shout louder just because it has more chefs.
The Math: You give equal weight to every cluster (hospital/school), not every patient.
The Result: This method asks, "If I pick a random hospital, is the treatment better there?" It might say, "Actually, the treatment works better in the small clinics, so Team B is winning."
The Benefit: This method is very stable and doesn't get confused by the size of the groups.

The Shocking Twist: They Can Disagree

The authors ran a simulation (a computer experiment) to show how crazy this can get.

Imagine a scenario where:

Big Hospitals (1,000 patients) get the treatment, but the patients do okay.
Small Clinics (20 patients) get the treatment, and the patients do amazingly.
The "Individual" Judge looks at the numbers: "Wow, 1,000 patients did okay! That's a lot of wins!" -> Verdict: Treatment is Great.
The "Cluster" Judge looks at the groups: "The big hospital did okay, but the small clinics were amazing. On average, the small clinics win." -> Verdict: Treatment is Terrible (or at least, the big hospitals are dragging the average down).

In their example, the "Individual" judge said the treatment was a 2.2x win, while the "Cluster" judge said it was a loss (0.88x). They were looking at the exact same data and coming to opposite conclusions!

Why Does This Matter?

If you are a doctor or a policy maker reading a study, you need to know which question is being answered:

"What happens to a random patient?" (Individual-Pair)
"What happens to a random hospital/community?" (Cluster-Pair)

If the study doesn't say which one they used, and the groups are different sizes, you might think a treatment is a miracle cure when it's actually a failure for the communities that need it most (or vice versa).

The Solution

The paper provides a "rulebook" (mathematical formulas) to help researchers:

Decide upfront: Which question do you want to answer? The "Patient" question or the "Community" question?
Use the right tool: Use the "Individual" math for the first question and the "Cluster" math for the second.
Be careful: If you use the "Individual" math in a world with different-sized groups, be aware that your results might be a little shaky in small studies.

The Bottom Line

In medical trials involving groups (like schools or hospitals), size matters. If you don't account for the fact that some groups are huge and others are tiny, you might be counting the votes of the giants and ignoring the voices of the small. This paper teaches us how to make sure we are asking the right question so we don't accidentally declare the wrong winner.

1. Problem Statement

Win statistics (e.g., Win Ratio, Win Odds, Win Difference) are increasingly used in clinical trials to prioritize outcomes by clinical importance, comparing treated vs. control patients to determine who "wins" based on a hierarchy of endpoints. While well-established in individual-randomized trials, their application in Cluster Randomized Trials (CRTs) is methodologically ambiguous.

The core problem is that in CRTs, researchers can target two fundamentally different estimands:

Individual-Pair Estimand: Treats every individual as the unit of inference, giving equal weight to all participants regardless of cluster size.
Cluster-Pair Estimand: Treats every cluster as the unit of inference, giving equal weight to all clusters regardless of size.

The paper argues that when Informative Cluster Sizes (ICS) exist—where cluster size is correlated with baseline outcomes or treatment effects—these two estimands can diverge significantly, potentially leading to opposite conclusions regarding treatment efficacy (e.g., one suggesting benefit, the other harm). Current literature lacks clear definitions and consistent estimators for these distinct targets in CRTs.

2. Methodology

The authors extend the potential outcomes framework to define and estimate win statistics in CRTs.

Definitions:
- Individual-Pair: The probability that a randomly selected treated individual from a treated cluster has a better outcome than a randomly selected control individual from a control cluster. This estimator weights pairs by the product of cluster sizes ( $N_i N_j$ ).
- Cluster-Pair: The average of win probabilities calculated within each pair of clusters, then averaged across all cluster pairs. This estimator applies inverse cluster-size weights ( $1/N_i N_j$ ) to ensure each cluster pair contributes equally.
- Unified Framework: Both estimands are unified under a general weighted framework where weights ( $\omega_{ij}$ ) determine the target (setting $\omega_{ij} = N_i N_j$ targets individual-pair; $\omega_{ij} = 1$ targets cluster-pair).
Estimation & Inference:
- The authors propose consistent estimators for Win Ratio (WR), Win Odds (WO), and Win Difference (WD) for both targets.
- Variance Estimation: A leave-one-cluster-out jackknife variance estimator is proposed. For inference, a $t$ -distribution with $M-2$ degrees of freedom (where $M$ is the number of clusters) is recommended for robustness, particularly with small numbers of clusters.
Evaluation:
- Hypothetical Examples: The authors constructed a 4-cluster-type example with ordinal outcomes (A, B, C) featuring both Type I ICS (baseline outcomes depend on size) and Type II ICS (treatment effects depend on size).
- Simulation Study: A simulation was conducted with $M=100$ clusters and a 5-level ordinal outcome under scenarios with and without ICS to evaluate bias, consistency, and coverage probabilities of the proposed estimators.

3. Key Contributions

Conceptual Clarification: The paper formally distinguishes between individual-pair and cluster-pair win estimands in CRTs, highlighting that they answer different scientific questions (population of individuals vs. population of clusters).
Demonstration of Non-Collapsibility: The authors show that win estimands are generally non-collapsible. Even difference-based estimands (like Win Difference), which are often assumed to be collapsible in other contexts, can differ between individual and cluster levels when ICS is present.
Methodological Tools: The paper provides consistent estimators and a specific jackknife variance estimator tailored for win statistics in clustered settings.
Bias Analysis: The study identifies a critical nuance: while individual-pair estimators are asymptotically consistent, they exhibit finite-sample bias in the presence of ICS. In contrast, cluster-pair estimators remain unbiased for their respective targets.

4. Results

Divergence in Estimands:
- In the hypothetical example with Type I and II ICS, the Individual-Pair Win Ratio was 2.238 (indicating strong treatment benefit), while the Cluster-Pair Win Ratio was 0.880 (indicating treatment harm).
- This divergence occurred because the individual-pair estimand was dominated by large clusters (which had favorable outcomes), whereas the cluster-pair estimand gave equal weight to smaller clusters (which showed unfavorable outcomes).
Simulation Findings:
- No ICS: Both estimands were equivalent, unbiased, and achieved nominal 95% coverage.
- With ICS:
  - Individual-Pair: Showed finite-sample bias (e.g., relative bias of 3.1% for WR, 5.9% for WD) and reduced coverage probabilities (dropping to ~91-93%).
  - Cluster-Pair: Remained unbiased with nominal coverage probabilities (~94-95%), demonstrating robustness against the influence of large clusters.

5. Significance

Critical for Trial Design: The paper underscores that failing to specify whether the target is an individual-level or cluster-level effect can lead to answering the wrong scientific question. In settings with informative cluster sizes (common in real-world trials like hospitals or schools), the choice of estimand dictates the conclusion.
Guidance on Estimator Selection: Researchers are advised to carefully select the estimand based on the intervention's mechanism. If the intervention targets "cluster behavior" (e.g., hospital prescribing habits), the cluster-pair estimand is appropriate. If the goal is the average patient experience, the individual-pair estimand is relevant, but researchers must account for potential finite-sample bias.
Future Directions: The framework is presented as a foundation for extending win statistics to more complex CRT designs, such as stepped-wedge or crossover trials, where informative sizes may vary over time.

In summary, this paper provides a rigorous statistical framework to prevent misinterpretation of win statistics in cluster randomized trials, emphasizing that cluster size informativeness is a pivotal factor that can invert treatment conclusions depending on the chosen estimand.