📊 statistics

Edgington's Method for Random-Effects Meta-Analysis Part I: Estimation

This paper extends Edgington's $p$ -value-based meta-analysis framework to random-effects models by integrating heterogeneity uncertainty via a confidence distribution approach, yielding point estimates with low bias and confidence intervals that generally achieve nominal coverage while often remaining narrower than the Hartung-Knapp-Sidik-Jonkman interval.

Original authors: David Kronthaler, Leonhard Held

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Kronthaler, Leonhard Held

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out the "true" average height of a specific type of tree. You can't measure every single tree in the forest, so you ask ten different foresters to measure a few trees each and send you their results.

In the world of statistics, this is called a meta-analysis. It's the art of combining many small studies to get one big, reliable answer.

This paper introduces a new, smarter way to do this math, specifically when the trees (or studies) aren't all exactly the same. Here is the breakdown of their method using everyday analogies.

The Problem: The "Wobbly" Average

Usually, when scientists combine these studies, they assume the differences between the foresters' results are just random noise (like a shaky hand). But often, the differences are real: maybe one forester measures pine trees and another measures oaks. This is called heterogeneity (or "between-study differences").

The old way of handling this is like asking a forester, "How much do the trees vary?" and then taking their answer as a fixed fact. The problem is, if you only have three foresters, their guess about the variation is a guess! It's shaky. If you treat a shaky guess as a solid fact, your final answer might look too precise when it's actually quite uncertain.

The Old Solution vs. The New Solution

The Old Way (Classical Methods): Imagine you ask the foresters for their measurements, calculate an average, and draw a "confidence interval" (a range where the true height likely sits). The old methods often draw this range as a perfect, symmetrical bell curve. But real data is often lopsided (skewed). If the data is lopsided, a symmetrical range is like trying to fit a square peg in a round hole—it doesn't capture the truth well.
The New Way (Edgington's Method): The authors use a technique called Edgington's method. Think of this as a special recipe for mixing the foresters' reports. Instead of just averaging numbers, it mixes their "confidence levels" (p-values).
- The Benefit: This recipe naturally handles lopsided data. If the results are skewed to the left, the final range of uncertainty stretches out to the left, just like the data does. It's more honest about the shape of the truth.

The Big Innovation: Accounting for the "Guesswork"

The paper's main contribution is fixing a specific flaw in Edgington's method.

The Flaw: Edgington's method is great, but it still treats the "variation between studies" (heterogeneity) as if it were a known, fixed number. It ignores the fact that estimating that variation is itself a guess, especially when you have few studies.

The Fix: The authors propose a two-step dance:

Step 1: They create a "confidence distribution" for the variation. Instead of saying "The variation is exactly 5," they say, "The variation is probably 5, but it could be 3 or 7." It's like a weather forecast saying "70% chance of rain" rather than "It will rain."
Step 2: They take Edgington's method and run it thousands of times, each time using a different possible variation value from that forecast.
The Result: They blend all those thousands of results together. This creates a final answer that accounts for the fact that we didn't know the variation perfectly to begin with.

What Did They Find? (The Simulation)

The authors tested this new method by running a massive computer simulation (like a video game where they created thousands of fake forests with different rules).

When it works best: If you have more than three studies and there is some real variation between them, this new method is excellent. It gives you a "confidence interval" that hits the true answer about 95% of the time (which is the gold standard).
When it's a bit too cautious: If you only have three studies, or if there is absolutely no variation between studies, the method is a little too conservative. It draws a slightly wider safety net than necessary (overcovering). However, even then, it is often narrower (more precise) than the current standard method (called Hartung-Knapp-Sidik-Jonkman), which is a big win.
Handling Skew: The new method is great at handling "lopsided" data. If the studies are clustered on one side, the new method's confidence interval stretches to match that shape, whereas old methods force it into a symmetrical box.

The Bottom Line

Think of this new method as upgrading from a rigid ruler to a flexible, smart tape measure.

Old Ruler: Assumes everything is symmetrical and ignores the fact that we aren't sure about the "variation" between studies.
New Smart Tape Measure: Bends to fit lopsided data and explicitly accounts for the fact that our estimate of "variation" is a bit fuzzy.

The authors conclude that this approach is a practical, better alternative to the old ways, especially when you have a moderate number of studies and the data isn't perfectly symmetrical. It makes the final answer more honest about what we know—and what we don't know.

Technical Summary: Edgington's Method for Random-Effects Meta-Analysis Part I: Estimation

Problem Statement
Classical random-effects meta-analysis typically relies on inverse-variance weighting (IVW) to estimate a mean effect, often producing symmetric confidence intervals that fail to capture data skewness. While methods like Hartung–Knapp–Sidik–Jonkman (HKSJ) improve coverage by accounting for heterogeneity estimation uncertainty via a $t$ -distribution, they retain symmetric interval structures. Conversely, $p$ -value combination methods, such as Edgington's approach (based on the sum of $p$ -values), offer the advantage of generating asymmetric confidence intervals that reflect the skewness of effect estimates. However, a critical limitation of the standard Edgington method is that it treats the between-study heterogeneity variance ( $\tau^2$ ) as a fixed plug-in estimate, thereby ignoring the uncertainty inherent in estimating $\tau^2$ , particularly in meta-analyses with a small number of studies.

Methodology
The authors propose an extension of Edgington's method that incorporates uncertainty in heterogeneity estimation through a confidence distribution (CD) framework. The core methodology involves the following steps:

Confidence Distribution of the Mean Effect: Edgington's method combines study-specific one-sided $p$ -value functions using the sum of $p$ -values, yielding a combined $p$ -value function $p_E(\mu)$ . This function is interpreted as a confidence distribution for the mean effect $\mu$ , conditional on a specific value of $\tau^2$ .
Confidence Distribution of Heterogeneity: To account for uncertainty in $\tau^2$ , the authors utilize the generalized heterogeneity statistic $Q(\tau^2)$ (Viechtbauer, 2006). Under the random-effects model, $Q(\tau^2)$ follows a $\chi^2_{k-1}$ distribution, allowing it to serve as a pivotal statistic and define a confidence distribution for $\tau^2$ .
Marginalization: The proposed estimator marginalizes the conditional confidence distribution of $\mu$ over the confidence distribution of $\tau^2$ . This is achieved by integrating the conditional density $c(\mu|\tau^2)$ with the marginal density $c(\tau^2)$ :
$c(\mu) = \int c(\mu|\tau^2) c(\tau^2) d\tau^2$
Computation: Two computational approaches are evaluated for this integration:
- Monte Carlo (MC) Sampling: Involves drawing $\tau^{2*}$ from the $\chi^2$ -based confidence distribution of $\tau^2$ and then drawing $\mu^*$ from the conditional confidence distribution of $\mu$ . The empirical distribution of $\mu^*$ estimates the marginalized CD.
- Global Adaptive Quadrature (GAQ): A deterministic integration method. Pilot simulations indicated that while GAQ is computationally efficient, it may produce slightly overly wide intervals in scenarios with few studies (e.g., $k=3$ or $5$). Consequently, the authors recommend the Monte Carlo algorithm for practical application, particularly when the number of studies is small.

Key Contributions

Extension of Edgington's Method: The paper extends Edgington's $p$ -value combination approach to the random-effects framework by explicitly integrating heterogeneity estimation uncertainty.
Asymmetric Intervals: The method produces confidence intervals that are not constrained to symmetry, allowing them to reflect the skewness of the underlying effect estimates, a property lacking in classical IVW and HKSJ intervals.
Algorithmic Implementation: The authors provide a Monte Carlo algorithm to perform the necessary marginalization and implement the method in the R package edgemeta.

Simulation Results
A simulation study was conducted across scenarios varying the number of studies ( $k \in \{3, 5, 10, 20, 50\}$ ), heterogeneity levels ( $I^2 \in \{0\%, 30\%, 60\%, 90\%\}$ ), and the presence of large studies.

Coverage: The CD-Edgington 95% confidence intervals approach nominal coverage in most scenarios involving more than three studies and heterogeneity. Under conditions of no heterogeneity or with only three studies, the intervals typically overcover (conservative), though they are often narrower than HKSJ intervals in these specific conservative cases.
Comparison to HKSJ: The CD-Edgington intervals generally achieve better coverage than the standard random-effects interval and often outperform Edgington's method with additive heterogeneity adjustment (which ignores $\tau^2$ uncertainty). Compared to HKSJ, CD-Edgington intervals are typically narrower, especially under large heterogeneity, while maintaining comparable coverage.
Skewness: The method effectively captures the skewness of effect estimates, showing high correlation between the skewness of the confidence intervals and the observed effect estimates.
Bias: The point estimator exhibits small bias under model misspecification (skew-normal true effects) and moderate to large heterogeneity, though it remains approximately unbiased under correct model assumptions.

Significance and Claims
The authors position this work as a practical alternative to classical random-effects meta-analysis approaches. They claim that by adjusting for the uncertainty in heterogeneity estimation, the proposed method improves confidence interval coverage, particularly in scenarios where heterogeneity is present and the number of studies is not large. The method addresses the limitation of symmetric summaries by providing inference that aligns with the skewness of the data.

The paper explicitly notes limitations: the method is not applicable to fixed-effect models, and the Monte Carlo approach requires a substantial number of samples for stability. The authors also acknowledge that while the method performs well under normal and skew-normal distributions, further assessment is needed for other effect measures (e.g., odds ratios). Finally, the authors state that this paper focuses solely on estimation, with the extension to prediction intervals for future study effects addressed in a subsequent part of the series.