📊 statistics

Prediction intervals for random-effects meta-analysis based on confidence distributions and Edgington's method

This paper proposes a novel method for constructing prediction intervals in random-effects meta-analysis by combining Edgington's $p$ -value approach with confidence distributions to effectively account for heterogeneity uncertainty and skewness, thereby achieving nominal coverage where traditional formulations fail.

Original authors: David Kronthaler, Leonhard Held

Published 2026-06-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: David Kronthaler, Leonhard Held

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Predicting the Future of Medical Studies

Imagine you are a doctor trying to figure out if a new medicine works. You don't just look at one study; you look at a collection of studies (a meta-analysis) to get the big picture.

Usually, statisticians calculate an "average effect." They say, "On average, this medicine lowers blood pressure by 10 points." But this is like saying, "The average temperature in a city is 70°F." That doesn't tell you if it's going to be a freezing 30°F tomorrow or a scorching 100°F. In medicine, knowing the average isn't enough; you need to know the range of possibilities for the next patient or the next study.

This paper introduces a new, more accurate way to predict that range, especially when the studies you are looking at are very different from each other (a problem statisticians call heterogeneity).

The Problem: The "Symmetric" Trap

Most current methods for predicting future results act like a perfectly balanced seesaw. They assume that if the average effect is in the middle, the future results will be equally likely to be slightly higher or slightly lower. They draw a symmetric "prediction interval" (a range of likely outcomes) around the average.

However, real-world data is often lopsided (skewed).

The Analogy: Imagine you are predicting the height of the next person walking into a room. If you only have data on professional basketball players and one toddler, the "average" might be 6 feet. But a symmetric prediction interval would suggest it's equally likely the next person is 2 feet tall or 10 feet tall. In reality, the next person is almost certainly a basketball player (tall), and it's very unlikely they are a giant. The data is skewed toward the tall side.

The paper argues that old methods force a symmetric shape onto skewed data, which leads to wrong predictions. They also often ignore the fact that we aren't 100% sure about how "different" the studies are from one another.

The Solution: A New "Confidence Map"

The authors propose a new method based on two main ideas: Edgington's Method and Confidence Distributions.

1. Edgington's Method: The "Voting System"

Instead of just averaging numbers, this method treats each study like a voter.

The Analogy: Imagine you are trying to guess the winner of a race. Instead of averaging the finish times, you ask each expert to vote on who might win. Some experts are very confident; others are unsure.
How it works: The authors combine the "votes" (p-values) from all studies. Crucially, this method is smart enough to handle skewed data. It doesn't force the result to be a perfect bell curve. If the studies lean heavily in one direction, the prediction leans that way too. It preserves the "lopsidedness" of the real world.

2. Confidence Distributions: The "Uncertainty Cloud"

Traditional methods often treat the "difference between studies" (heterogeneity) as a single, fixed number.

The Analogy: Imagine you are trying to guess the size of a hidden box. Old methods might say, "The box is definitely 10 inches wide."
The New Way: The authors say, "We aren't sure. The box could be 10 inches, but it might be 5, or 15, or even 20." They create a cloud of possibilities (a confidence distribution) for how different the studies are.
The Result: When they make their final prediction, they don't just use one number for the "difference." They run thousands of simulations, picking a different "difference" size from their cloud every time. This creates a prediction that accounts for all the uncertainty, not just the average.

How They Tested It: The Simulation Lab

To prove their method works, the authors built a virtual laboratory.

The Setup: They created thousands of fake meta-analyses with different numbers of studies (from 3 to 50) and different levels of "messiness" (heterogeneity).
The Test: They compared their new method against the old, standard methods.
The Findings:
- Accuracy: When there were more than three studies, their new method hit the "target" (correct coverage) almost exactly 95% of the time, which is the gold standard.
- Skewness: Their method correctly predicted when the future results would be lopsided. The old methods often failed here, giving symmetric answers to skewed problems.
- The "Ignore Uncertainty" Mistake: They showed that if you ignore the uncertainty about how different the studies are (like the old "fixed" methods do), your predictions become dangerously narrow and often wrong.

A Real-World Example: Corticosteroids and COVID-19

The authors tested their method on a real set of seven studies about steroids and COVID-19 mortality.

The Old View: Standard methods gave a narrow range, suggesting steroids were likely beneficial, but the range was tight and symmetric.
The New View: Their method produced a wider, more realistic range. It showed that while steroids were likely beneficial, there was a much higher chance of a "zero effect" or even a negative effect than the old methods suggested.
Why? Because the new method acknowledged that the studies were quite different from each other and that we weren't perfectly sure about that difference. It didn't pretend to know more than it did.

The Takeaway

This paper is about humility in statistics.

When we look at a group of different studies, we shouldn't pretend the future is a perfect, symmetric mirror of the past. We should acknowledge that:

The data might be lopsided (skewed).
We are never 100% sure about how different the studies are.

By using Edgington's method to handle the shape of the data and Confidence Distributions to handle the uncertainty, this new approach gives us a "prediction interval" that is more honest, more accurate, and better suited for making decisions about future medical events.

In short: It's a better way to say, "Based on what we know, here is the realistic range of what might happen next, including the weird, lopsided possibilities."

Technical Summary: Prediction Intervals for Random-Effects Meta-Analysis Based on Confidence Distributions and Edgington's Method

Problem Statement
In random-effects meta-analysis, statistical inference regarding the average effect is often deemed insufficient when substantial between-study heterogeneity exists. While traditional measures like the estimated between-study variance ( $\tau^2$ ) and Higgins' $I^2$ quantify heterogeneity, they lack interpretability on the effect scale and do not convey absolute effect variation. Predictive distributions are recognized as a more clinically relevant tool for quantifying heterogeneity, as they describe the distribution of true effects and inform about future events.

However, existing frequentist methods for constructing predictive distributions face significant limitations:

Symmetry Constraints: Most methods constrain predictive distributions to be symmetric around the estimated average effect, failing to capture the skewness often present in effect estimates. This leads to poor calibration and prediction intervals that frequently fail to achieve nominal coverage, particularly in scenarios with few studies or skewed data.
Handling Uncertainty: Standard approaches often address uncertainty in the heterogeneity estimate ( $\tau^2$ ) by applying a $t$ -distribution with arbitrary degrees of freedom (e.g., $k-2$ ). Simulation studies suggest these post-hoc adjustments are often poorly calibrated, especially when the number of studies is small or heterogeneity is considerable.
Skewness Neglect: Methods that do not account for the skewness of study estimates may produce misleading inferences, as evidenced by simulation results showing under-coverage in skewed scenarios.

Methodology
The authors propose a novel framework for constructing predictive distributions that explicitly accounts for uncertainty in both the average effect ( $\mu$ ) and the heterogeneity parameter ( $\tau^2$ ) using confidence distributions and Edgington's p-value combination method.

Confidence Distributions (CDs): Instead of relying on sampling distributions of estimators, the method constructs CDs, which are frequentist probability distributions over the parameter space derived from p-value functions without invoking prior distributions.
Edgington's Method: The authors utilize Edgington's method (1972) to combine one-sided p-value functions across studies. Unlike the classical weighted average approach, Edgington's method:
- Is invariant to the orientation of the alternative hypothesis.
- Yields nearly unbiased estimates.
- Produces confidence intervals that adapt to the skewness of effect estimates, preserving asymmetry rather than imposing symmetric summaries.
Heterogeneity Uncertainty: The method incorporates uncertainty in $\tau^2$ by utilizing the confidence distribution implied by the generalized heterogeneity statistic ( $Q(\tau^2)$ ). This statistic, based on the exact distribution of Cochran's $Q$ , varies with $\tau^2$ and serves as a pivot.
Predictive Distribution Construction: Three versions of the predictive distribution are proposed:
- PCD-fixed: Integrates the normal effect distribution over Edgington's CD for $\mu$ , using a plug-in estimate for $\tau^2$ .
- PCD-simplified: Integrates over the CD for $\mu$ (using a plug-in $\tau^2$ ) and the CD for $\tau^2$ .
- PCD-full: Fully integrates over the joint conditional distribution of $\mu$ and $\tau^2$ .
These integrals are evaluated via a Monte Carlo algorithm (Algorithm 1) that samples $\tau^2$ from its CD (solving $Q(\tau^2) = W$ where $W \sim \chi^2_{k-1}$ ), samples $\mu$ from Edgington's CD (using the probability integral transform), and finally samples the future effect $\theta_{new}$ from $N(\mu, \tau^2)$ .

Key Contributions

Skewness-Aware Prediction: The proposed method generates predictive distributions that naturally inherit the skewness of the underlying study estimates, addressing a critical gap in existing symmetric methods.
Rigorous Uncertainty Propagation: By integrating over the confidence distributions of both $\mu$ and $\tau^2$ , the method avoids arbitrary adjustments (like fixed $t$ -distribution degrees of freedom) and provides a more principled handling of estimation uncertainty.
Implementation: The authors provide the edgemeta R package, making these methods accessible for calculating equi-tailed and Highest Confidence Density (HCD) prediction intervals, as well as confidence probabilities for specific events (e.g., the probability a future effect is beneficial).

Results
A simulation study (following the ADEMP framework) compared the proposed PCD methods against the Higgins-Thompson-Spiegelhalter (HTS) method and a parametric bootstrap approach (Nagashima et al., 2018).

Coverage:
- PCD-full and PCD-simplified typically achieved nominal 95% coverage when more than three studies were available.
- These methods effectively reflected skewness in scenarios with 20 or fewer studies.
- In contrast, PCD-fixed (ignoring heterogeneity uncertainty) consistently undercovered, often falling below 80% with three studies, and only approached nominal coverage with very large numbers of studies.
- The HTS method showed a U-shaped coverage pattern: over-covering with three studies and under-covering with 5–50 studies, particularly under moderate heterogeneity.
Skewness Correlation: The PCD-full intervals demonstrated moderate to strong Pearson correlations between interval skewness and effect estimate skewness for fewer than 50 studies, whereas HTS intervals (being symmetric) had zero skewness.
Sharpness: While PCD-full and PCD-simplified intervals were generally wider than HTS intervals (reflecting proper uncertainty propagation), they often outperformed HTS in terms of Continuous Ranked Probability Score (CRPS) when the number of studies was small or heterogeneity was low.
Case Study: In a meta-analysis of corticosteroids and mortality in COVID-19 patients, the CD-Edgington estimator yielded a wider, right-skewed confidence interval compared to the symmetric Hartung-Knapp-Sidik-Jonkman (HKSJ) interval. The resulting predictive distributions (PCD-full/simplified) were substantially wider than HTS intervals, reflecting the substantial uncertainty in the heterogeneity estimate even when the point estimate of $\tau^2$ was near zero.

Significance and Claims
The paper claims that the proposed approach offers a viable alternative or complement to classical random-effects meta-analysis. The authors emphasize that:

Ignoring uncertainty in heterogeneity estimation leads to incorrect coverage, underscoring the necessity of the proposed adjustment.
The method successfully adapts to skewness in effect estimates, providing more accurate predictive inferences than symmetric alternatives.
While the method shows performance comparable to the parametric bootstrap in calibration and sharpness, it provides a distinct frequentist framework based on confidence distributions.
The authors acknowledge that further independent evaluation is needed, particularly across different effect scales and distributions, and note that the method is currently restricted to unimodal, normal random-effects models. They suggest future extensions could incorporate more flexible effect distributions (e.g., skew-normal) and study-specific weights to downweight biased studies.