📊 statistics

Z-Curve Plot: A Visual Diagnostic for Publication Bias in Meta-Analysis

This paper introduces the z-plot, a novel visual diagnostic tool implemented in the RoBMA R package that overlays model-implied and observed z-statistic distributions to detect publication bias, assess model fit, and compare competing meta-analytic models.

Original authors: František Bartoš, Ulrich Schimmack

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: František Bartoš, Ulrich Schimmack

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine science as a giant library where researchers write books about how the world works. To understand the big picture, other scientists act as librarians who gather all these books, read them, and combine their findings into one "super-summary" called a meta-analysis. This is how we get our best answers about things like "Does this medicine work?" or "Does this teaching method help?" But there's a sneaky problem: not every book gets published. If a study finds a boring result or a negative result, the author might hide it in a drawer, while the exciting, positive results get printed and put on the shelves. This is called "publication bias." It's like judging a movie only by the reviews from people who loved it, while ignoring the thousands of people who hated it. This makes the library look like every movie is a masterpiece, even if they aren't. Scientists have tried to fix this with tools like the "funnel plot," which is a chart meant to spot missing books, but it's often blurry and hard to read, like trying to find a specific needle in a haystack while wearing foggy glasses.

This paper introduces a new, sharper tool called the "z-plot" (or z-curve plot) to help librarians see exactly what's missing. Instead of looking at a messy 2D chart, the z-plot lines up all the scientific results in a single row based on how "surprising" they are. Think of it like a line of students waiting for a prize. If the rules say only students with a score above 90 get a prize, you'd expect a smooth line of students. But if the school is omitting all the papers with scores below 90, you'd see a sudden, jagged cliff in the line where the low scores used to be. The z-plot draws a smooth line showing what the results should look like if no data was omitted, and then overlays the actual results on top. If the actual results have a jagged cliff right at the "prize line," the plot screams, "Hey, someone is hiding the bad news!"

The authors, František Bartoš and Ulrich Schimmack, tested this idea using computer simulations. They created two fake worlds: one where scientists were honest and published everything, and another where they only published the "good" stuff. In the honest world, the z-plot showed a smooth, gentle curve, and the computer models agreed that everything looked normal. But in the dishonest world, the z-plot showed a sharp, suspicious drop-off right at the point where results became "statistically significant" (the magic number of 1.96). The old models, which ignored the omission, tried to draw a smooth line through the jagged cliff and failed miserably. The new models, which accounted for the omission, drew a line that perfectly followed the jagged edge. This proved that the z-plot can visually show when a model is wrong because it's ignoring hidden data.

To show how this works in the real world, the team applied the z-plot to a real study about "social comparison" (like looking at a leaderboard to see if you're doing better than your friends). The original study looked at 37 trials and said, "Everything looks fine, no cheating here!" But when the authors drew the z-plot, it told a different story. The plot showed a massive cliff where negative results should have been, meaning the study was likely hiding all the times the technique didn't work. The old models couldn't explain this cliff, but the new, bias-aware models could. In fact, the z-plot revealed that the original conclusion was likely too optimistic; when you fix for the missing data, the best estimate suggests the technique might not have any real effect at all, with the data showing moderate evidence that the effect is absent.

The paper concludes that the z-plot is a powerful way to "see" the truth behind the numbers. It doesn't just give a yes-or-no answer; it lets researchers look at a graph and instantly understand why a model is failing. If the line is smooth, you can trust the results. If there's a jagged cliff, you know you need a better model to account for the missing pieces. While the authors admit this tool works best when there are many studies to look at (like a big crowd of students), it offers a much clearer view than the old, foggy funnel plots. It's a way to make sure our scientific library tells the whole story, not just the happy ending.

Technical Summary: Z-Curve Plot as a Visual Diagnostic for Publication Bias

Problem Statement
Publication bias—the suppression of statistically non-significant or non-conforming findings—undermines the validity of meta-analytic inferences by inflating effect sizes and evidence against the null hypothesis. While numerous statistical methods exist to adjust for publication bias (e.g., selection models, regression-based approaches, and Bayesian model averaging), there is a notable lack of publication bias-specific visual diagnostics. Existing tools, such as funnel plots and their extensions (contour and power enhancement), are criticized for unreliability in detecting bias, vulnerability to "small-study effects," and an inability to facilitate the visual assessment of already estimated models. Consequently, researchers lack a straightforward method to visually determine if a specific meta-analytic model fits the observed data well or if it fails to account for systematic discontinuities caused by selection processes.

Methodology
The authors propose the z-plot, a visual diagnostic tool that overlays the model-implied distribution of $z$ -statistics onto the observed distribution of $z$ -statistics. The methodology proceeds as follows:

Data Projection: Instead of using two-dimensional funnel plots, the method projects data into a one-dimensional histogram of $z$ -statistics (observed effect sizes divided by their standard errors). This preserves high resolution in the dense area of small $p$ -values and maps directly to $p$ -values.
Detection of Discontinuities: In the absence of publication bias, the sampling distribution of $z$ $z$ -statistics is a smooth mixture of approximately normal distributions. Publication bias manifests as sharp discontinuities in the observed histogram at specific thresholds:
- Statistical significance thresholds ( $|z| = 1.96$ for $\alpha=0.05$ , $|z|=1.64$ for $\alpha=0.10$ ).
- The direction of the effect ( $z=0$ ).
Model Implied Distribution: For Bayesian meta-analytic models, the authors compute the posterior predictive distribution of $z$ -statistics ( $z_{pred}$ ). This involves obtaining the posterior predictive distribution for effect sizes ( $y_{pred}$ ) and performing a change of variables ( $z_{pred} = y_{pred}/se$ ).
Visual Comparison: The z-plot overlays these model-implied curves on the observed histogram.
- Good Fit: The model-implied distribution closely tracks the observed histogram, including any discontinuities.
- Poor Fit: Systematic discrepancies appear, such as models overestimating non-significant results or failing to capture the "cliff" at the significance threshold.

The methodology is implemented in the RoBMA R package, allowing for the simultaneous visualization of competing models, including Bayesian random-effects models, Bayesian PET (Precision-Effect Test) models, Bayesian three-parameter selection models (3PSM), and Robust Bayesian Meta-Analysis (RoBMA) which uses Bayesian model averaging.

Key Contributions

Visual Absolute Fit Diagnostic: The z-plot provides a visual assessment of absolute model fit, answering whether a model describes the data well, distinct from relative model comparison tests (e.g., Bayes factors, AIC).
Simultaneous Model Comparison: It allows researchers to compare the fit of multiple competing meta-analytic models within a single figure, revealing which models successfully account for selection mechanisms.
Differentiation from Small-Study Effects: Unlike funnel plot asymmetry, which can be caused by heterogeneity or small-study effects, discontinuities at significance thresholds in the z-plot are presented as direct evidence of publication bias.
Implementation: The method is integrated into the RoBMA package, making it accessible for standard Bayesian meta-analytic workflows.

Results
The paper demonstrates the utility of the z-plot through simulated datasets and a real-world meta-analysis:

Simulated Unbiased Data: In datasets generated without publication bias, all models (random-effects, PET, 3PSM, RoBMA) closely approximate the observed $z$ -distribution. The z-plot confirms that no model exhibits systematic misfit, validating the use of simpler models in the absence of bias.
Simulated Biased Data: In datasets with simulated publication bias (retaining only 50% of non-significant/negative studies), the random-effects and PET models fail to capture the discontinuity at $z=1.96$ , overestimating non-significant results. In contrast, the 3PSM and RoBMA models track the discontinuity accurately. The z-plot visually corroborates statistical findings that bias-adjusted models provide a better fit and more accurate effect size estimates.
Real-World Example (Social Comparison): In a meta-analysis of social comparison as a behavior change technique, traditional funnel plots and Egger's test suggested no significant bias. However, the z-plot revealed a clear discontinuity at $z=0$ (selection for positive results). Simple models (random-effects, PET, 3PSM) failed to account for this, systematically overestimating negative estimates. RoBMA, which includes selection models for positive direction, provided a better fit, resulting in a model-averaged effect size near zero ( $g \approx -0.008$ ) with moderate evidence for the absence of an effect, contrasting with the original positive finding.

Significance and Claims
The authors position the z-plot as a complementary tool to standard statistical tests. Its primary significance lies in its ability to:

Validate Statistical Tests: It allows researchers to visually validate the results of model comparison tests (e.g., Bayes factors) by "seeing" the misfit that statistical metrics quantify.
Guide Model Selection: It helps practitioners identify when simple, unadjusted models are inappropriate and when complex, bias-adjusted models are necessary.
Identify Model Limitations: If no considered model fits the observed distribution well (e.g., if a model fails to account for $p$ -hacking in marginal significance regions), the z-plot can signal the need for more complex modeling or suggest abstaining from interpretation.

The paper modestly notes that the method is most informative for moderately sized or large meta-analyses and that the visualization relies on $z$ -statistics which may introduce slight distortions if original studies used non-normal test statistics (e.g., $F$ -tests). Furthermore, while the z-plot can detect patterns indicative of questionable research practices (QRPs), it cannot always distinguish them from standard publication bias. The method is currently implemented within a Bayesian framework, though the authors suggest similar visualizations could be developed for classical frameworks.

Technical Summary: Z-Curve Plot as a Visual Diagnostic for Publication Bias

More like this