📊 statistics

Reframing Population-Adjusted Indirect Comparisons as a Transportability Problem: An Estimand-Based Perspective and Implications for Health Technology Assessment

This paper reframes population-adjusted indirect comparisons as a transportability problem, demonstrating that marginal treatment effects derived from such comparisons are often population-dependent due to non-collapsibility and effect modification, thereby necessitating explicit assumptions and careful alignment of estimands to ensure their validity for health technology assessment decision-making.

Original authors: Conor Chandler, Jack Ishak

Published 2026-08-19

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Conor Chandler, Jack Ishak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medicine, deciding which new treatment works best for a specific group of patients often relies on comparing results from different clinical trials. These trials are the gold standard for proving that a medicine works, but they are rarely perfect mirrors of the real world. A trial might enroll only young, healthy volunteers, while the people who actually need the medicine are older and have other health conditions. When researchers want to compare two new drugs that have never been tested head-to-head, they must stitch together evidence from separate studies. This process, known as an indirect comparison, becomes complicated when the patient groups in those studies look different. To fix this, scientists use statistical tools to adjust the data, essentially reweighting the results so that the groups look more alike. This adjustment is meant to answer a crucial question for health agencies: if we give this drug to the people who actually need it, how much better will they do compared to the alternative?

A new paper by Conor Chandler and Jack Ishak challenges a long-held belief about how these adjustments work. For years, health technology agencies have operated under the assumption that if researchers correctly identify the factors that change how a drug works—such as age or disease severity—and adjust for them, the resulting comparison is valid for any population. The authors argue that this is not always true. They show that even when the adjustment is done perfectly, the way the final result is calculated can make it specific to the original study group and useless for the broader population. Their work reveals that the mathematical nature of the measurement itself—whether it is a simple difference in averages or a more complex ratio—determines whether the results can be safely moved from one group of people to another.

The researchers approached this problem by reframing the entire process as a question of "transportability." Imagine you have a map drawn for a specific city, and you want to use that map to navigate a different city that looks similar. If the streets are laid out in the exact same way, the map works. But if the layout is different, even slightly, the map might lead you astray. In the context of drug trials, the "map" is the statistical estimate of how well a drug works, and the "cities" are the different patient populations. The authors demonstrated that for many common ways of measuring drug effectiveness, the map is not universal. It is tied to the specific layout of the original city.

The paper focuses on two main ways of measuring treatment effects. The first is a conditional effect, which looks at how a drug works for a patient with a specific set of characteristics, like a 60-year-old with mild disease. The second is a marginal effect, which asks what the average outcome would be if everyone in a whole population received the drug. Health agencies usually care about the marginal effect because they need to know the impact on the entire population they serve. The authors found that while conditional effects can often be transported from one population to another if the right adjustments are made, marginal effects are much trickier.

Through a series of detailed simulations, the researchers tested five different types of measurements used in medical research. They created virtual populations with varying characteristics and tried to transport the results from one group to another. They found that for simple measurements, like the average difference in blood pressure or a simple risk difference, the results held up. If the adjustment was done correctly, the result was the same regardless of the population's makeup. However, for more complex measurements that are standard in the field, such as hazard ratios (used for survival data) and odds ratios (used for binary outcomes like success or failure), the results broke down. Even when the researchers assumed that the factors influencing the drug's effectiveness were identical across all groups, the marginal results still changed depending on the population.

This finding directly contradicts the current guidance used by major health assessment bodies. These agencies typically invoke the "shared effect modifier" assumption (SEMA) to justify transferring results from one study population to another. This idea suggests that if the same factors affect how different drugs work, then the comparison between those drugs should be the same for everyone. However, the authors show that this assumption is not sufficient on its own. While SEMA may allow for the transport of conditional effects, it fails to guarantee that the average, population-wide results will remain stable when moved to a new group. The problem lies in the mathematics of the measurement itself. Non-collapsible measures, which include the widely used odds and hazard ratios, do not behave like simple averages. They are influenced by the entire distribution of the population, not just the average characteristics.

The researchers illustrated this with specific examples. When they looked at a simple difference in means, the results were stable across all simulated populations. But when they switched to a log odds ratio, a common metric for binary outcomes, the results shifted as the population changed, even though the underlying drug performance and the adjustment factors remained constant. Similarly, for survival data, they examined the restricted mean survival time, a measure that calculates the average time a patient survives. Even though this measure is mathematically "collapsible" in a simple sense, it failed to transport correctly because the way the effect was modeled did not align with the way the final result was calculated. The mismatch between the scale used to model the data and the scale used to report the result created a hidden dependency on the population.

The implications of this work are significant for how medical evidence is used to make decisions about funding and access to new treatments. The authors argue that researchers and health agencies must stop assuming that a successful adjustment automatically makes a result transportable. Instead, they must carefully consider what kind of effect they are trying to measure and whether the mathematical properties of that measure allow it to be moved across populations. If the goal is to estimate a population-wide effect, using a non-collapsible measure like an odds ratio may lead to biased conclusions if the results are simply applied to a new group without further adjustment.

The paper does not suggest that indirect comparisons are useless. Rather, it calls for a more precise and honest approach to how these comparisons are interpreted. It suggests that when a study produces a result that is inherently tied to the specific population it was measured in, that result should be treated as population-specific. It should not be blindly applied to a different group of patients, even if the groups look similar on paper. For health technology assessments, this means that the choice of statistical method and the type of effect measure are not just technical details; they are fundamental to whether the evidence can be trusted for real-world decision-making.

Ultimately, the work provides a clear framework for determining when a treatment effect can be safely transported and when it cannot. It shows that direct transportability is not a given; it is a property that depends on the alignment of the statistical model, the type of effect being measured, and the assumptions about how the population is structured. By distinguishing between conditional and marginal effects and understanding the role of collapsibility, researchers can avoid the trap of assuming that a corrected result is universally applicable. The path forward involves recognizing that some results are local to the study that generated them, and that applying them elsewhere requires additional, often unspoken, assumptions that may not hold true. This clarity allows for more transparent and principled use of indirect evidence, ensuring that decisions about patient care are based on a solid understanding of what the data can and cannot tell us.

Technical Summary: Reframing Population-Adjusted Indirect Comparisons as a Transportability Problem

1. Problem Statement

Health Technology Assessment (HTA) agencies frequently rely on indirect treatment comparisons (ITCs) when head-to-head randomized controlled trials (RCTs) are unavailable. While Randomized Controlled Trials (RCTs) possess strong internal validity, their external validity is often limited because trial populations differ from the broader, heterogeneous populations relevant to reimbursement decisions. To address this, Population-Adjusted Indirect Comparisons (PAICs)—such as Matching-Adjusted Indirect Comparisons (MAIC) and Simulated Treatment Comparisons (STC)—have been developed to adjust for observed differences in effect modifiers between studies.

However, a critical methodological gap exists in how PAIC results are applied in HTA. Current practice often involves a two-step process:

Step 1: Adjusting treatment effects from an index trial to the population of a comparator trial (conditional transport).
Step 2: Applying the resulting relative effect (e.g., Treatment B vs. Treatment C) directly to a decision-relevant target population (often the index population or a real-world population).

The validity of Step 2 is frequently assumed based on the Shared Effect Modifier Assumption (SEMA), which posits that the same covariates modify treatment effects in the same way across all active treatments. Current HTA guidance (e.g., NICE DSU TSD 18) suggests that if SEMA holds, relative effects are directly transportable to any population. This paper argues that this guidance is insufficient. It posits that SEMA alone does not guarantee direct transportability, particularly for marginal estimands using non-collapsible effect measures (e.g., odds ratios, hazard ratios). The paper identifies that direct transportability is a joint property of the estimand type, the collapsibility of the effect measure, the alignment between the scale of effect modification and the effect measure, and SEMA.

2. Methodology and Framework

The authors reframe PAICs as a transportability problem using a causal inference framework based on the Rubin Causal Model. The analysis distinguishes between two primary types of estimands:

Conditional Estimands: Effects defined conditional on covariates ( $X$ ), often evaluated at the mean covariate values or averaged over the covariate distribution after transformation.
Marginal Estimands: Effects defined as the average potential outcomes over the entire population, where averaging occurs before the transformation (e.g., calculating the average risk difference or log-odds ratio).

The paper formalizes the conditions under which a treatment effect estimated in a source population ( $P_1$ ) is invariant to the covariate distribution of a target population ( $P_2$ ). This invariance is defined as direct transportability.

The analysis relies on three core conceptual pillars:

Collapsibility: Whether a marginal effect can be expressed as a weighted average of conditional effects. Non-collapsible measures (e.g., odds ratios, hazard ratios) generally depend on the full joint distribution of covariates, not just their means.
Scale Alignment: The relationship between the model scale (linear predictor defined by link function $g(\cdot)$ ) and the effect measure scale (defined by transformation $h(\cdot)$ ). Direct transportability is facilitated when effect modification and the effect measure operate on the same linear predictor scale (i.e., $h \circ g^{-1} = I$ ).
Shared Effect Modifier Assumption (SEMA): The assumption that effect modifiers are shared across active treatments and that their impact is identical.

The authors derive theoretical conditions for direct transportability and validate them through illustrative numerical examples. These examples simulate 21 populations with varying covariate distributions to test the transportability of five specific effect measures:

Mean difference (Linear regression)
Log odds ratio (Logistic regression)
Log risk ratio (Log-binomial regression)
Difference in Restricted Mean Survival Time (RMST) (Weibull regression)
Log ratio of RMST (Weibull regression)

3. Key Results

Theoretical Findings

The paper demonstrates that direct transportability of the active-to-active contrast ( $\Delta_{BC}$ ) requires a specific conjunction of conditions:

For Conditional Estimands: Direct transportability is achieved if SEMA holds and effect modification is modeled on the linear predictor scale. In this case, effect-modification terms cancel out in the active-to-active contrast, rendering the effect invariant to covariate distribution shifts.
For Marginal Estimands: Direct transportability is significantly more restrictive. It requires:
1. SEMA to hold.
2. Effect modification and the effect measure to operate on the same linear predictor scale.
3. The effect measure to be collapsible.

If these conditions are not met, marginal effects remain population-dependent even if SEMA is satisfied.

Empirical Illustrations

The numerical examples confirm the theoretical distinctions:

Mean Differences: Being directly collapsible and operating on the linear scale, they are directly transportable under SEMA.
Log Odds Ratios: These are non-collapsible. Even when SEMA holds, marginal log odds ratios exhibit population dependence (bias) when transported to populations with different covariate distributions. Only conditional log odds ratios are directly transportable under SEMA.
Log Risk Ratios: These are collapsible. When effect modification is defined on the log-risk scale (matching the effect measure), marginal log risk ratios are directly transportable under SEMA.
RMST Measures: Despite RMST differences being directly collapsible, they are not directly transportable under standard PAIC assumptions. This is because standard survival models (e.g., Weibull) define effect modification on the log-hazard scale, while RMST is derived from the survival curve (a non-linear transformation). This scale misalignment ( $h \circ g^{-1} \neq I$ ) prevents the cancellation of effect-modification terms, causing the marginal RMST effect to depend on the full covariate distribution, even under SEMA.

Summary of Transportability Requirements (Table 1)

The paper synthesizes that for the most common non-collapsible measures used in practice (Odds Ratios and Hazard Ratios), marginal effects derived from PAICs are generally not directly transportable. They are valid only for the specific population in which they were estimated (the comparator population). Applying them to a different target population without further adjustment introduces bias.

4. Key Contributions

Reframing PAICs: The paper formally reframes PAICs not just as adjustment methods, but as transportability problems involving a dual-step process, highlighting that the second step (application to decision-relevant populations) is often implicitly assumed rather than rigorously justified.
Clarifying SEMA's Limits: It challenges the prevailing HTA guidance that SEMA alone is sufficient for direct transportability. The authors demonstrate that SEMA is necessary but not sufficient, particularly for marginal estimands using non-collapsible measures.
Distinguishing Estimands: The work provides a rigorous distinction between conditional and marginal estimands in the context of indirect comparisons, showing that while conditional effects may be transportable under SEMA, marginal effects often are not.
Scale Alignment Insight: The paper introduces the concept of scale alignment between the model link and the effect measure as a critical determinant of transportability, explaining why collapsible measures like RMST may still fail to be transportable if the underlying effect modification is defined on a different scale (e.g., log-hazard).

5. Significance and Implications

The authors conclude that the current practice of applying PAIC-derived marginal effects (especially odds ratios and hazard ratios) directly to decision-relevant populations in HTA cost-effectiveness models is often unjustified. The paper argues that:

Interpretation of Estimates: PAIC estimates for non-collapsible measures should be interpreted as population-specific (valid for the comparator trial population) rather than decision-relevant for the target population.
Methodological Rigor: HTA decision-making requires explicit acknowledgment of the transport step. If direct transportability cannot be guaranteed (due to non-collapsibility or scale misalignment), additional assumptions or modeling steps are required to transport effects to the target population.
Guidance Revision: The findings suggest that current HTA guidance (e.g., NICE DSU TSD 18) regarding the sufficiency of SEMA for direct transportability needs refinement to account for the properties of the estimand and the effect measure.

The paper advocates for a more principled approach where the choice of estimand (conditional vs. marginal) and effect measure is guided by the research question and the plausibility of the underlying structural assumptions regarding transportability, rather than defaulting to standard non-collapsible measures without verifying their transportability properties.