📊 statistics

Effect modification and non-collapsibility leads to conflicting treatment decisions: a review of marginal and conditional estimands and recommendations for decision-making

This paper reviews how the combined presence of effect modification and non-collapsibility can cause marginal and conditional estimands to yield conflicting treatment rankings, and it provides practical recommendations for decision-making that leverage multilevel network meta-regression to generate both types of estimates for the target population.

Original authors: David M. Phillippo, Antonio Remiro-Azócar, Anna Heath, Gianluca Baio, Sofia Dias, A. E. Ades, Nicky J. Welton

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David M. Phillippo, Antonio Remiro-Azócar, Anna Heath, Gianluca Baio, Sofia Dias, A. E. Ades, Nicky J. Welton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of healthcare, doctors and policymakers face a constant challenge: deciding which treatment works best for a specific group of people. They rely on clinical trials, which are carefully controlled experiments, to find the answers. However, people are not all the same. Some have severe forms of a disease, others have mild forms, and some carry specific biological markers that change how their bodies react to medicine. When a factor like this changes how well a treatment works, scientists call it "effect modification." It is a well-known fact that if a drug works wonders for one type of patient but barely helps another, the best choice for the whole group depends on who is in that group.

To make these decisions, researchers often combine data from multiple trials to get a clearer picture. But there is a mathematical quirk that complicates things. When measuring the success of a treatment, scientists often use ratios, such as odds ratios or hazard ratios, which compare the likelihood of an event happening in one group versus another. These measures are "non-collapsible," a technical way of saying that the average result for a whole group is not simply a weighted average of the results for the individuals inside it. Because of this, the way you calculate the average matters. You can calculate the average by looking at the effect on each person and then averaging those effects, or you can calculate the average by looking at the overall event rates for the whole group and comparing them. Usually, these two methods give slightly different numbers, but they generally agree on which treatment is better.

A new paper by David Phillippo and his colleagues reveals that this agreement is not guaranteed. The researchers show that when effect modification is present, these two ways of calculating the average can lead to completely opposite conclusions. One method might suggest that Treatment A is the best choice for the entire population, while the other method suggests that Treatment B is superior. This happens because the two methods are actually answering two different questions. One asks, "Which treatment gives the best average benefit to the individual patients?" while the other asks, "Which treatment results in the fewest total bad events for the entire population?" When the effectiveness of a treatment varies significantly across different types of patients, the answer to these two questions can diverge.

The authors illustrate this conflict using a series of simulations. In one scenario, they modeled a population where a specific biomarker made a treatment highly effective for some people but less effective for others. When they calculated the average benefit for the individual, the data pointed to one drug as the clear winner. However, when they calculated the overall event rate for the whole group, a different drug appeared to be the best choice. In this specific example, choosing the drug that minimized the total number of bad events meant that 75 percent of the population would receive a treatment that was actually inferior for them personally. Conversely, choosing the drug that was best for the average individual would result in a higher total number of bad events across the population.

This finding challenges the common assumption that there is always a single "best" treatment for a population. The researchers argue that the conflict arises because the mathematical curves representing the probability of an event for different treatments can cross each other. If a treatment is better for one subgroup but worse for another, and these subgroups are mixed together, the overall average can flip the ranking. This is particularly relevant for time-to-event outcomes, such as survival times, where the presence of any patient characteristic, even one that does not change the treatment effect, can cause the average risk to change over time.

The paper also examines the tools currently used to adjust for these differences between study populations. Methods like matching-adjusted indirect comparison and simulated treatment comparison are widely used to tailor trial results to a specific target population. However, the authors demonstrate that these methods often produce estimates that are specific only to the population of the original study and cannot be easily transported to a new population with different characteristics. They identify a more advanced method, multilevel network meta-regression, as the only current approach capable of producing both types of estimates—those for the individual and those for the group—in any target population.

Ultimately, the authors recommend that decision-makers stop relying on a single number to make choices. Instead, they should look at both the individual-level and the population-level estimates. If both methods agree on the best treatment, the decision is straightforward. If they disagree, it signals that the population is not uniform and that a single decision for everyone may be suboptimal. In such cases, the best course of action might be to split the population into subgroups and assign different treatments to each group. This approach ensures that patients receive the treatment that is best for their specific characteristics, rather than forcing a compromise that might leave the majority worse off. The paper concludes that while identifying these subgroups is difficult and requires careful statistical work, it is the only way to resolve the conflict between maximizing individual benefit and minimizing total harm when effect modification is present.

Technical Summary: Effect Modification and Non-Collapsibility in Treatment Decision-Making

Problem Statement
Healthcare decision-makers frequently rely on network meta-analyses (NMA) and population-adjustment methods to select optimal treatments. A critical challenge arises when two statistical phenomena coexist: effect modification (where covariates alter relative treatment effectiveness) and non-collapsibility (where marginal and conditional effect measures, such as odds ratios or hazard ratios, do not coincide even in the absence of effect modification).

While it is established that effect modification necessitates subgroup-specific recommendations and that non-collapsible measures yield different marginal and conditional estimates, the literature has not sufficiently addressed the consequences of their interaction. Specifically, it remains unclear how these factors jointly influence treatment rankings and whether standard population-adjustment methods (e.g., Matching-Adjusted Indirect Comparison [MAIC], Simulated Treatment Comparison [STC]) can adequately resolve conflicting decision questions. The core problem is that conditional and marginal estimands correspond to distinct decision questions that may diverge when effect modification is present, potentially leading to suboptimal population-level decisions.

Methodology and Framework
The authors define and contrast two primary estimands for non-collapsible effect measures (e.g., log odds ratios, log hazard ratios) within a generative outcome model:

Population-Average Conditional Estimand ( $d_{ab}(P)$ ): The average of individual-level conditional treatment effects over the covariate distribution in the target population. This is calculated by averaging the linear predictors (e.g., log odds) across the population.
Population-Average Marginal Estimand ( $\Delta_{ab}(P)$ ): The treatment effect calculated on the probability scale after averaging the event probabilities across the population. This represents the difference in average event probabilities between treatments.

The paper utilizes a theoretical framework involving:

Generative Models: Binary outcomes modeled via logit links and time-to-event outcomes via proportional hazards models.
Population Adjustment Methods: A review of Multilevel Network Meta-Regression (ML-NMR), MAIC, and STC, analyzing which estimands they target and their limitations regarding transportability to external target populations.
Worked Examples: Numerical simulations demonstrating scenarios where effect modification causes individual treatment effect curves to cross, leading to divergent rankings between conditional and marginal estimands.
Contingency Tables: Illustrative examples with binary covariates to show how weighting differs between the two estimands.

Key Contributions and Results

Divergence of Decision Questions: The paper establishes that conditional and marginal estimands answer different questions. The conditional estimand asks, "Which treatment has the greatest effect for individuals, on average?" while the marginal estimand asks, "Which treatment minimizes (or maximizes) the average event probability in the population?"
Rank-Switching Phenomenon: The authors demonstrate that when effect modification is present, the direction of the treatment effect can differ between conditional and marginal estimates.
- Example: In a simulated scenario, Treatment B had a superior conditional effect (better for the average individual), but Treatment C had a superior marginal effect (fewer total events in the population). Choosing based on the marginal effect would result in 75% of the population receiving an inferior treatment.
- Mechanism: Rank-switching occurs only when individual-level treatment effect curves cross within the population, which requires substantive effect modification (different interaction coefficients for the treatments being compared).
Dependence on Baseline Risk: Marginal rankings are shown to depend on baseline risk and the distribution of prognostic factors, even if these factors do not modify the treatment effect. In contrast, conditional rankings depend solely on the distribution of effect modifiers.
Survival Outcomes: For time-to-event data, the presence of covariates implies that marginal hazard ratios are time-varying, violating the proportional hazards assumption at the marginal level. Effect modification can further cause marginal hazard curves to cross over time, altering treatment rankings dynamically.
Limitations of Current Methods:
- MAIC and STC: These methods typically target the marginal estimand but are restricted to the aggregate study population of an indirect comparison. They generally cannot produce estimates for external target populations unless specific assumptions (e.g., shared effect modifiers with linear models) are made. Crucially, they cannot produce both conditional and marginal estimates simultaneously.
- ML-NMR: Identified as the only current population-adjustment method capable of producing both conditional and marginal estimates in any decision target population. It achieves this by integrating individual-level models over aggregate study populations.

Significance and Recommendations
The paper argues that relying on a single estimand in the presence of effect modification can lead to decisions that are suboptimal for the majority of individuals or result in a higher total number of adverse events.

Pragmatic Decision-Making: The authors recommend that decision-makers explicitly specify the target population and the decision question a priori.
Conflict Resolution: If conditional and marginal rankings agree, a single population-wide decision is justified. If they conflict, the authors propose a pragmatic approach: stratify decisions by subgroups defined by the most influential effect modifiers. This aligns the decision with both the average individual benefit and the overall population event rate.
Cost-Effectiveness: For economic evaluations, net benefit should be averaged over the population using individual or subgroup-specific event probabilities (Equation 8). The paper notes that standard Markov models often fail to account for this heterogeneity. ML-NMR is highlighted as the necessary tool to generate the required inputs (individual or subgroup probabilities) for any target population.
Methodological Shift: The paper concludes that while previous discussions on non-collapsibility focused on scenarios without effect modification, the presence of effect modification fundamentally alters the relationship between estimands, rendering standard power arguments (which favor conditional estimands) invalid in certain contexts.

In summary, the paper provides a rigorous theoretical and practical guide for handling the conflict between marginal and conditional estimands, emphasizing the necessity of using flexible methods like ML-NMR to support nuanced, subgroup-informed healthcare decisions.

Technical Summary: Effect Modification and Non-Collapsibility in Treatment Decision-Making

More like this