📊 statistics

Adjusting for Outcome Reporting Bias in Meta-analysis: A Multiple Imputation Approach

This paper proposes a multiple imputation framework to adjust for outcome reporting bias in both univariate and multivariate meta-analyses, demonstrating through simulations and real-world data that this approach effectively corrects biased treatment effect estimates by accounting for selective non-reporting mechanisms.

Original authors: Cora Burgwinkel, Saverio Fontana, Leonhard Held

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cora Burgwinkel, Saverio Fontana, Leonhard Held

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Silent" Studies

Imagine you are trying to judge how good a new medicine is. You ask 12 doctors for their reports. However, 6 of those doctors only send you a note saying, "The medicine worked great!" They don't send the full report. The other 6 doctors send you the full report, which shows the medicine worked, but not as amazingly as the first group claimed.

This is Outcome Reporting Bias (ORB). It happens when researchers decide to hide or ignore their results if they aren't exciting or positive. It's like a student only showing their teacher the test questions they got right and hiding the ones they got wrong. If you only look at the "happy" reports, you will think the medicine is a miracle cure, when in reality, it might just be okay.

The Solution: Filling in the Blanks with "Multiple Imputation"

The authors of this paper propose a way to fix this problem without needing the missing data to magically appear. They use a statistical trick called Multiple Imputation.

Think of it like a detective trying to solve a crime where some witnesses are missing.

The Guess: The detective doesn't just guess once. Instead, they create 1,000 different "what-if" scenarios. In each scenario, they fill in the missing witness statements with plausible guesses based on what the other witnesses said.
The Correlation: Sometimes, the missing information is related to information we do have. For example, if a patient's seizure count went down, their quality of life usually went up. The authors' method uses these connections (correlations) to make smarter guesses. If we know how the medicine affected seizures in the missing studies, we can use that to guess how it affected quality of life, even if that data is missing.
The Weighting: Not all guesses are treated equally. The method asks: "If the researchers were hiding bad news, how likely is it that they would have reported this specific result?" If a result looks too good to be true, the method gives it less weight (like a judge discounting a witness who seems too eager to please). If the result looks realistic, it gets more weight.

By averaging these 1,000 scenarios, the method creates a "bias-adjusted" estimate that is much closer to the truth than just looking at the reports we happened to receive.

The Two Approaches: One vs. Many

The paper tests this method in two ways:

The Solo Approach (Univariate): This looks at one outcome at a time (e.g., just seizure reduction). It's like asking a single expert for advice.
The Team Approach (Multivariate): This looks at multiple outcomes at once (e.g., seizure reduction and seizure freedom). It's like asking a team of experts who talk to each other. Because the experts share information, the team can make better guesses about the missing data. If one expert is missing, the others can help fill in the gaps based on how their topics are related.

What They Found

The authors tested this method in two ways:

1. Real-World Test (Epilepsy Data):
They applied their method to a real study about epilepsy drugs.

The Result: When they ignored the missing data (the "naive" way), the drug looked very effective. When they used their new method to account for the missing data, the drug's effectiveness dropped significantly. It didn't disappear, but it wasn't the "miracle cure" the initial reports suggested. This proves that ignoring missing data can make treatments look much better than they actually are.

2. The Simulation Lab:
They created thousands of fake studies on a computer to see how their method performed under different conditions.

The Result: Their method successfully reduced the "lie" caused by hiding bad news.
The Catch: The method works best when the studies are similar to each other. If the studies are all over the place (high "heterogeneity"), or if there are very few studies to begin with, the method struggles a bit more, but it still performs better than doing nothing.
The Team Advantage: The "Team Approach" (Multivariate) generally did a better job than the "Solo Approach" because it could borrow strength from the related outcomes to fill in the blanks.

The Bottom Line

This paper argues that we cannot simply ignore studies that don't report their results. If we do, we are likely to be fooled into thinking treatments work better than they do.

The authors propose a flexible, "detective-style" statistical tool that fills in the missing pieces of the puzzle using logic and probability. While it requires some assumptions about why data is missing, it provides a much more honest picture of reality than simply looking at the reports that were sent in. It turns a biased, one-sided view into a balanced, evidence-based conclusion.

Technical Summary: Adjusting for Outcome Reporting Bias in Meta-analysis via Multiple Imputation

Problem Statement
Outcome Reporting Bias (ORB) occurs when study outcomes are selectively reported based on the nature of their results, typically favoring statistically significant or positive findings. This selective reporting creates a missing data problem where unreported outcomes introduce bias into meta-analyses, distorting overall treatment effect estimates and undermining the validity of clinical evidence. While publication bias (non-publication of entire studies) is well-studied, ORB remains under-addressed. Standard practice often ignores missing outcomes, conducting univariate meta-analyses on complete cases, which leads to biased inferences. Existing adjustment methods, such as the Copas selection model, often rely on categorical risk classifications (e.g., ORBIT) or handle outcomes separately, failing to leverage the correlation between multiple outcomes within studies.

Methodology
The authors propose a flexible approach to adjust for ORB in both univariate and multivariate random-effects meta-analyses using Multiple Imputation (MI) combined with Importance Sampling based on selection models.

Imputation Framework:
- Missing Standard Errors: For studies where sample sizes are known but standard errors are missing, the authors impute standard errors using a weighted average of observed variances and sample sizes.
- Univariate Imputation: Unreported study outcomes are imputed from a conditional multivariate normal distribution derived from the reported outcomes. This assumes a Missing At Random (MAR) mechanism initially.
- Multivariate Imputation: In the bivariate (or multivariate) setting, the method jointly imputes unreported outcomes for correlated endpoints. This allows the model to "borrow strength" from reported outcomes to inform the imputation of missing ones, accounting for within-study and between-study correlations.
Selection and Reweighting:
- To correct for the non-random nature of the missingness (Missing Not At Random, MNAR), the imputed datasets are reweighted using Importance Sampling.
- The authors employ a logistic selection model where the probability of an outcome being reported is a function of the effect estimate ( $\hat{\theta}$ ) or the z-score ( $z$ ).
- A selection weight parameter, $\delta$ , controls the strength of the selection mechanism. $\delta=0$ corresponds to MAR (no selection bias), while $\delta > 0$ implies that outcomes with larger effects (or higher z-scores) are more likely to be reported.
- The final adjusted estimate is a weighted average of the meta-analytic estimates from $M$ imputed datasets, where weights are determined by the selection model.
Implementation:
- The method is implemented for both univariate and multivariate models.
- Uncertainty in the within-study correlation (often unreported) is addressed through sensitivity analyses, varying the correlation parameter across plausible ranges.

Key Contributions

Integration of MI and Selection Models: The paper integrates multiple imputation with probabilistic selection via importance sampling, extending previous work (e.g., Carpenter et al.) to specifically address ORB in meta-analysis.
Multivariate Extension: Unlike many existing methods that adjust outcomes separately, this approach extends ORB adjustment to multivariate meta-analysis. It jointly models correlated outcomes, allowing information to be borrowed across endpoints to improve the imputation of missing data.
Dual Selection Mechanisms: The framework supports selection based on both the raw effect estimate and the z-score (statistical significance), acknowledging that reporting decisions may depend on either metric.
Sensitivity Analysis: The method is designed as a sensitivity analysis tool, requiring the user to specify a range of selection weights ( $\delta$ ) to assess how conclusions change under different assumptions about the reporting mechanism.

Results

Application to Clinical Data: The method was applied to a Cochrane review of Topiramate for drug-resistant epilepsy.
- For the outcome "seizure freedom," where 50% of studies were unreported, the ORB-adjusted estimates were systematically shifted toward the null (less extreme) compared to naive analyses.
- The magnitude of the shift increased with higher assumed selection weights ( $\delta$ ).
- Selection based on the z-score generally resulted in stronger adjustments than selection based on the effect estimate.
- The multivariate approach yielded stronger adjustments than the univariate approach when selection was modeled jointly, as the correlation between outcomes amplified the impact of the selection mechanism.
Simulation Study: A comprehensive simulation study evaluated the method under varying heterogeneity ( $I^2$ $I^{2}$ ), study sizes, and selection strengths.
- Bias Reduction: The naive estimates exhibited substantial bias, particularly under high heterogeneity and strong selection. The proposed ORB-adjusted methods significantly reduced this bias.
- Multivariate Performance: The bivariate approach generally outperformed the univariate approach by borrowing strength from the fully reported outcome, resulting in lower bias and better coverage, provided the correlation between outcomes was non-zero.
- Limitations: Under conditions of very high heterogeneity ( $I^2 = 90\%$ ) and small sample sizes ( $K=6$ ), residual bias persisted, and numerical instability (non-convergence) was observed. The method performed best when the assumed selection mechanism in the analysis matched the true mechanism in the data generation.

Significance and Claims
The authors claim that imputing unreported study outcomes provides a promising and practical approach to address ORB in meta-analyses. They position the method not as a definitive correction (since the true selection mechanism is unknown) but as a sensitivity analysis to evaluate the robustness of meta-analytic conclusions.

The paper emphasizes that:

ORB can substantially distort treatment effect estimates, often leading to overestimation of benefits.
The proposed multivariate approach extends ORB adjustment to jointly model correlated outcomes, offering a more comprehensive framework than separate univariate adjustments.
While the method reduces bias and improves coverage compared to naive analyses, its effectiveness depends on the assumed selection mechanism, the degree of heterogeneity, and the strength of the correlation between outcomes.
The approach is particularly valuable when no external classification of bias risk (like ORBIT) is available to guide the adjustment, as it relies on statistical modeling of the missingness mechanism.

The authors conclude that their approach offers a flexible tool for researchers to assess how sensitive their meta-analytic conclusions are to potential outcome reporting bias.