Transportability of model-based estimands in evidence synthesis

Technical Summary: Transportability of Model-Based Estimands in Evidence Synthesis

Problem Statement
In evidence synthesis, particularly for health technology assessment (HTA), the goal is often to estimate marginal treatment effects for population-level decision-making. However, standard practices frequently rely on conditional effect measures (e.g., odds ratios, hazard ratios) derived from individual-level models. A critical issue arises because many common summary measures are non-collapsible or not directly collapsible. For these measures, the marginal effect is not simply a weighted average of conditional effects. Consequently, purely prognostic covariates—variables that predict the outcome but do not interact with treatment at the individual level—can act as marginal effect measure modifiers. This means that even in the absence of individual-level treatment effect heterogeneity (i.e., no treatment-covariate interactions), differences in the distribution of purely prognostic covariates between studies can induce bias in unadjusted indirect comparisons. Furthermore, current guidance often suggests that covariate adjustment is only necessary when there are imbalances in conditional effect measure modifiers or when individual-level heterogeneity exists. This paper argues that such guidance is insufficient for non-collapsible measures, as marginal effects depend on the full joint distribution of covariates, not just their marginal moments.

Methodology
The paper employs a simulation study based on the ADEMP (Aims, Data-generating mechanisms, Estimands, Methods, Performance measures) framework to empirically investigate these phenomena. The study simulates an anchored indirect treatment comparison scenario involving two randomized controlled trials (RCTs) with a common comparator.

• Data Generation: Three distinct scenarios were constructed: (1) Treatment effect homogeneity with imbalanced marginal covariate means; (2) Treatment effect heterogeneity with balanced marginal means but different correlation structures; and (3) Treatment effect heterogeneity with both imbalanced means and different correlation structures.
• Outcome Models: Three outcome types were modeled: continuous (linear model, mean difference), count (log-linear/Poisson model, log risk ratio), and binary (logistic model, log odds ratio).
• Methods Compared: The performance of three indirect comparison methods was evaluated:
  1. Bucher Method: Standard unadjusted anchored indirect comparison.
  2. Matching-Adjusted Indirect Comparison (MAIC): A weighting-based method balancing marginal covariate moments.
  3. Parametric G-computation: An outcome modeling-based method using standardization to project inferences to the target population.
• Performance Metrics: Bias, Mean Square Error (MSE), and empirical coverage rates of 95% confidence intervals were assessed across 500 simulation replicates.

Key Contributions
The paper provides a theoretical and empirical clarification of how collapsibility properties influence the transportability of treatment effects:

1. Non-Collapsibility and Prognostic Variables: It demonstrates that for non-collapsible measures (e.g., log odds ratio), purely prognostic covariates modify marginal effects even when individual-level treatment effects are homogeneous. Therefore, cross-study imbalances in these variables bias unadjusted comparisons.
2. Direct Collapsibility and Joint Distributions: For measures that are collapsible but not directly collapsible (e.g., log risk ratio) in the presence of individual-level heterogeneity, the marginal effect depends on the full joint distribution of covariates (including correlations), not just marginal moments.
3. Limitations of Current Guidance: The study challenges current guidance which suggests unadjusted comparisons are valid if conditional effect modifiers are balanced or absent. The results show that for non-directly collapsible measures, unadjusted methods remain biased even when marginal covariate moments are balanced, if correlation structures differ.
4. Adjustment Limitations: It highlights that standard covariate adjustment methods (like MAIC) which only balance marginal moments (means/SDs) are inherently limited in removing bias when correlation structures differ between studies, particularly for non-directly collapsible measures.

Results

• Mean Difference (Directly Collapsible): Unadjusted comparisons (Bucher) performed well when treatment effects were homogeneous or when covariate means were balanced. Bias only appeared when means were imbalanced in the presence of heterogeneity. Covariate adjustment successfully removed bias.
• Log Risk Ratio (Collapsible but not Directly Collapsible): In the presence of individual-level heterogeneity, unadjusted comparisons were biased even when marginal covariate means were balanced, due to differences in correlation structures. Covariate adjustment methods (MAIC, G-computation) reduced bias compared to the unadjusted approach but remained biased because they could not account for the differing joint distributions (correlations) of the target population.
• Log Odds Ratio (Non-Collapsible): Unadjusted comparisons exhibited substantial bias even in the absence of individual-level treatment effect heterogeneity, solely due to imbalances in purely prognostic covariates. Covariate adjustment methods improved performance but still struggled to fully eliminate bias when correlation structures differed, as the marginal odds ratio depends on the full joint distribution.
• Precision vs. Bias: Parametric G-computation often showed lower bias but suffered from over-precision (under-coverage) in settings with poor covariate overlap or model extrapolation. MAIC provided more "honest" uncertainty quantification (better coverage) but with higher variance and reduced effective sample size.

Significance and Claims
The paper claims that the choice of summary effect measure fundamentally alters the requirements for external validity and transportability in evidence synthesis.

• Re-evaluating Effect Modifiers: The set of covariates required to ensure the constancy of relative effects depends on whether the measure is collapsible and whether it is directly collapsible. For non-collapsible measures, purely prognostic variables must be considered potential marginal effect modifiers.
• Implications for Practice: For non-directly collapsible measures, relying solely on marginal covariate summaries (means and standard deviations) for adjustment is insufficient to guarantee unbiased transportability if joint distributions (correlations) differ. The paper suggests that directly collapsible measures (like mean differences or risk differences) are preferable for transportability as they reduce dependence on complex model-based adjustments and joint distribution assumptions.
• Guidance Revision: The findings imply that current guidance, which often permits unadjusted indirect comparisons in the absence of treatment-covariate interactions, may be flawed for non-collapsible measures. Furthermore, the assumption that balancing marginal moments is sufficient for covariate adjustment is challenged; accounting for joint distributions is necessary but often practically infeasible with aggregate data, highlighting a significant limitation in current evidence synthesis capabilities.