Hidden in Plain Sight: How Non-Collapsibility Biases Treatment Effects in (Network) Meta-Analysis

1. Problem Statement

The paper addresses a critical, often overlooked source of bias in Network Meta-Analysis (NMA) and Pairwise Meta-Analysis (MA) when using non-collapsible effect measures, specifically the Odds Ratio (OR) and Hazard Ratio (HR).

• The Core Issue: Non-collapsibility occurs when the marginal (population-average) association in a mixed population does not equal a weighted average of the subgroup-specific associations, even in the absence of confounding or effect modification.
• The Flaw in Standard Practice: Standard contrast-based NMA models assume that if the distribution of effect modifiers is constant across studies, a common treatment effect exists regardless of differences in baseline risk (prognostic factors).
• The Consequence: When studies enroll heterogeneous populations (mixtures of distinct risk groups) with varying baseline risks, the marginal ORs estimated by standard models are biased toward the null (attenuated). This happens even if the treatment effect is identical across all subgroups.
  • Example: If Study A enrolls only low-risk patients and Study B enrolls only high-risk patients, but Study C enrolls a 50/50 mix, the marginal OR for Study C will be closer to 1.0 (null) than the true conditional OR, simply due to the mixing of probabilities, not because the treatment is less effective.
• Current Limitations: Standard random-effects models do not solve this; they merely absorb the systematic attenuation into the between-study variance ($\tau^2$), leading to inflated heterogeneity and a pooled estimate that remains biased.

2. Methodology

The authors propose a "Bookend" approach to correct for this bias. The methodology is demonstrated first in a simplified pairwise MA setting and is theoretically extendable to full NMA.

A. Theoretical Demonstration

The authors use a logistic regression framework to show that for a binary outcome:

• The standard model estimates a conditional log-OR ($d$) that applies to specific study populations.
• When a study population is a mixture of two distinct risk groups (e.g., smokers and non-smokers), the marginal log-OR is a non-linear function of the mixture proportions.
• Mathematically, $OR_{mix} \neq \exp(d)$. The magnitude of the bias depends on the heterogeneity of baseline risks ($\mu_1 - \mu_2$) and the mixing proportion.

B. The "Bookend" Model

To recover an unbiased estimate of the conditional treatment effect ($d$), the authors propose modeling mixed-population studies as weighted combinations of two homogeneous "bookend" populations.

Steps of the Approach:

1. Estimate Baseline Risks: Fit a standard model to obtain study-specific baseline log-odds ($\hat{\mu}_j$).
2. Identify Bookends: Select the two studies with the most extreme baseline risks (minimum and maximum $\hat{\mu}$).
   • Assumption 1: These "bookend" studies represent homogeneous populations (e.g., 100% low-risk and 100% high-risk).
3. Model Mixing: Treat all other studies as mixtures of these two extreme populations.
   • Assumption 2: There is a binary latent structure (one dominant prognostic factor with two levels) driving the baseline risk heterogeneity.
4. Likelihood Construction:
   • For bookend studies: Use the standard contrast model: $\text{logit}(p_{jk}) = \mu_j + d \cdot \mathbb{1}(k=2)$.
   • For mixed studies: Model the event probability as a weighted average at the probability scale (not the log-odds scale):
     $$p_{mix, k} = w \cdot p_{1, k} + (1-w) \cdot p_{2, k}$$
     Where $w$ is the mixing proportion to be estimated.

C. Implementation

• The model is implemented using Bayesian inference (JAGS code provided in the appendix).
• It estimates four parameters: $\mu_1$ (low-risk baseline), $\mu_2$ (high-risk baseline), $d$ (true conditional log-OR), and $w$ (mixing proportion).

3. Key Contributions

1. Clarification of Bias Source: The paper rigorously demonstrates that bias in NMA can arise purely from non-collapsibility when baseline risks vary, even if there is zero effect modification and no confounding. This challenges the standard assumption that baseline risk differences do not distort treatment effect estimates.
2. The Bookend Solution: Introduces a novel, explicit modeling strategy that accounts for within-study heterogeneity by treating mixed studies as probabilistic mixtures of extreme homogeneous populations.
3. Distinction from Existing Methods:
   • Unlike Random-Effects Models, which treat attenuation as random variance, the bookend model treats it as a systematic structural bias.
   • Unlike Population-Adjusted Indirect Comparisons (MAIC/STC), which require individual patient data (IPD) or aggregate covariate data, the bookend approach works with standard aggregate data by inferring the latent structure from the extremes of the evidence base.
   • Unlike Wheaton et al. (2024), which assumes subgroup effects are additive on the log-OR scale, the bookend model correctly applies mixing on the probability scale.
4. Practical Guidance: Provides a heuristic for practitioners to detect potential bias:
   • Examine the spread of $\hat{\mu}_j$ across studies.
   • If the spread is large (>1-2 log-odds units) and studies with mixed populations exist, perform a sensitivity analysis using the bookend model.

4. Results

The authors illustrate the method using a hypothetical lung disease example:

• Scenario:
  • Study 1: 100% Smokers (High Risk, $\mu_1 = 0$).
  • Study 2: 100% Non-smokers (Low Risk, $\mu_2 = -2$).
  • Study 3: 50% Smokers / 50% Non-smokers (Mixed).
  • True Treatment Effect: $d = -0.5$ (OR = 0.61) for both groups.
• Findings:
  • Standard Model: The pooled estimate was $\hat{d} = -0.458$ (95% CrI: -0.58, -0.34). This represents a 15% attenuation toward the null compared to the true value.
  • Bookend Model: The estimate was $\hat{d} = -0.492$ (95% CrI: -0.62, -0.37), recovering the true effect much more accurately.
• Observation: While the bias in this specific example was modest, the authors note that bias increases with greater heterogeneity in baseline risks and a higher proportion of mixed-population studies.

5. Significance and Implications

• Validity of NMA: The paper warns that standard NMA results for non-collapsible measures (OR, HR) may be systematically biased toward the null in the presence of baseline risk heterogeneity. This could lead to underestimating the efficacy of interventions in health technology assessments and clinical guidelines.
• Model Selection: It suggests that simply switching to a random-effects model is insufficient. The choice of effect measure (collapsible vs. non-collapsible) and the modeling of baseline risk are critical.
• Future Directions:
  • The approach is applicable to Hazard Ratios (though more complex due to survival data structures).
  • It highlights the need for sensitivity analyses in standard NMA workflows.
  • It challenges the "portability" of the Odds Ratio; while often preferred for its mathematical properties, its non-collapsibility creates specific pitfalls in heterogeneous evidence bases that must be managed explicitly.

In conclusion, Campbell and Jansen provide a necessary correction to the standard interpretation of NMA results, offering a practical "bookend" framework to mitigate bias caused by the mathematical properties of non-collapsible effect measures in heterogeneous populations.