📊 statistics

The dangers of using three-number summaries to estimate unknown standard deviations: sensitivity analyses and some possible improvements incorporating shape

This paper demonstrates that three-number summaries are insufficient for reliably estimating standard deviations in meta-analyses, leading to potentially invalid inferences, and proposes a sensitivity analysis framework alongside a new scaled Beta distribution-based estimator that incorporates data shape information to improve accuracy.

Original authors: Udara Kumaranathunga, Alysha De Livera, Luke A. Prendergast

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Udara Kumaranathunga, Alysha De Livera, Luke A. Prendergast

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery about a group of people's heights. You don't have a list of every single person's height. Instead, you only have a tiny, three-line note left by a witness. This note tells you three things:

The height of the shortest person.
The height of the person right in the middle (the median).
The height of the tallest person.

Or, perhaps the note only gives you the middle person and the "middle 50%" (the range between the 25th and 75th percentiles).

The Problem: The "Normal" Trap
For years, researchers have been trying to guess the average height and the spread (standard deviation) of the whole group using just these three numbers. Most of them have been using a "magic formula" that assumes everyone's height follows a perfect, symmetrical bell curve (like a normal distribution).

This paper argues that this is like trying to guess the shape of a cloud just by looking at its shadow. If the cloud is actually a flat pancake, a jagged mountain, or a U-shaped valley, the "bell curve" formula will give you a completely wrong answer.

The authors show that even if the three numbers look perfectly symmetrical, the actual data behind them could be shaped in dozens of different ways.

The Uniform Trap: Imagine a classroom where every single student is exactly the same height. The "spread" is zero. But if you use the standard formula on the min/max/median, it might tell you the students vary wildly.
The U-Shape Trap: Imagine a room full of very short people and very tall people, but almost no one in the middle. The standard formula might think everyone is average, missing the extreme spread entirely.

The Danger: Why It Matters
Why does getting the "spread" wrong matter? Think of it like aiming a cannon.

If you guess the spread is smaller than it really is, your cannonball (your statistical test) will fly too far. You might think you've discovered a "miracle cure" or a "huge difference" when it's just random noise. You might claim a result is significant when it's not.
If you guess the spread is larger than it really is, your cannonball falls short. You might miss a real discovery because you think the data is too messy to trust.

The paper shows that by using these three-number summaries without checking the shape, researchers are often firing blind, leading to false conclusions.

The Solution: Sensitivity Analysis (The "What If?" Game)
Instead of picking one formula and hoping for the best, the authors suggest playing a game of "What If?" called a Sensitivity Analysis.

Imagine you are a chef tasting a soup. Instead of just guessing the salt level, you taste it assuming it's a tomato soup, then a chicken soup, then a vegetable soup.

If the soup tastes salty no matter what you assume, you can be confident.
If the salt level changes wildly depending on whether you assume it's tomato or chicken, you know you don't have enough information.

The authors provide a new tool (a web app) that lets researchers run this "What If?" game. They test the data against many different shapes (bell curves, U-shapes, skewed hills) to see which ones are even possible given the three numbers. This helps researchers say, "We can't be sure, but the answer is likely between X and Y," rather than giving a single, potentially wrong number.

The New Trick: Using the "Fence" (Population Bounds)
Sometimes, researchers know the "fences" of the data.

Example: If you are studying the age of adults in a study, you know the minimum is 18 and the maximum is 100. No one is 5 or 150.
Example: If you are studying a test score from 0 to 100, you know the limits are 0 and 100.

The paper introduces a new method using a Scaled Beta Distribution. Think of this as a flexible rubber band. If you know the fences (the minimum and maximum), you can stretch this rubber band to fit the three numbers you have. Because the rubber band knows it can't go past the fences, it can guess the "spread" much more accurately than the old methods, which pretend the data could go on forever.

The Bottom Line
The paper concludes with three simple rules for researchers:

Don't guess the spread if you only have three numbers. If you can, just analyze the medians directly instead of trying to convert them to averages.
Play the "What If?" game. Before you publish, check if your answer changes drastically if you assume the data looks like a hill, a valley, or a flat line. If it does, admit the uncertainty.
Use the fences. If you know the absolute minimum and maximum possible values (like age limits or test score limits), use them! The new method using these limits is much better at guessing the spread.

In short: Three numbers are not enough to tell the whole story. Trying to force them into a single answer is dangerous. Instead, explore the possibilities, check the shape, and be honest about the uncertainty.

Technical Summary: The Dangers of Using Three-Number Summaries to Estimate Unknown Standard Deviations

Problem Statement
In meta-analysis, researchers frequently encounter studies reporting continuous data via quantile summaries (medians, quartiles, ranges) rather than means and standard deviations (SDs). While methods exist to convert five-number summaries (minimum, $Q_1$ , median, $Q_3$ , maximum) to means and SDs, the conversion of three-number summaries (e.g., median and interquartile range, or median and range) remains problematic. The authors argue that three-number summaries lack sufficient information to reliably estimate the unknown SD because they do not constrain the underlying data shape. Consequently, assuming a specific distribution (typically normality) to derive an SD can lead to severe estimation errors, potentially invalidating subsequent statistical inferences such as t-tests or Cohen's $d$ calculations.

Methodology
The paper employs a combination of theoretical analysis, simulation studies, and real-data case studies to evaluate existing and proposed methods.

Review of Existing Methods: The authors analyze popular conversion techniques, including:
- Wan et al. (2014) and Luo et al. (2018): Closed-form estimators assuming underlying normality and symmetry.
- Quantile Estimation (QE) (McGrath et al., 2020a): A method fitting parametric distributions (Normal, Lognormal, Gamma, Weibull, Beta) by minimizing the sum of squared errors (SSE) between observed and theoretical quantiles.
- Box-Cox (BC) Transformation: A method attempting to symmetrize data via power transformation before applying normal-based estimators.
Sensitivity Analysis Framework: Motivated by the limitations of single-method selection, the authors propose a goodness-of-fit type sensitivity analysis. This involves:
- Fitting multiple candidate distributions to the three-number summary.
- Calculating SSEs and performing a parametric bootstrap to generate p-values ( $\hat{p}_B$ ).
- Using these metrics to rule out implausible distributions and report a range of plausible SD estimates rather than a single point estimate.
Proposed Improvement (QE-beta): To address scenarios where population bounds (minimum $L$ and/or maximum $U$ ) are known (e.g., age limits, questionnaire scores), the authors introduce a new estimator based on the Scaled Beta (SB) distribution.
- The SB distribution is defined on $[L, U]$ and offers high flexibility in shape (uniform, skewed, U-shaped, J-shaped).
- The method estimates Beta shape parameters ( $\alpha, \beta$ ) by matching the three-number summary (or four-number if only $L$ is known) to the theoretical quantiles of the SB distribution.
Simulation Design:
- Generated Data: Simulations tested various SB distributions (uniform, U-shaped, skewed, J-shaped) with known $L$ and $U$ against existing methods that incorrectly treat population bounds as sample extremes.
- Real Data: Simulations were conducted using the PHQ-9 depression dataset (bounded 0–27) to evaluate performance under realistic conditions.

Key Results

Sensitivity to Shape: Even for symmetric distributions (e.g., Uniform, Beta), assuming normality to estimate SD from three-number summaries results in large relative errors (ranging from -35% to +50%). For skewed distributions, errors are often more severe.
Failure of Symmetrization: The Box-Cox transformation can achieve symmetry but does not guarantee normality. For example, transforming Pareto data yields a uniform distribution, leading to significant SD estimation errors if normality is assumed post-transformation.
Impact on Inference: The paper demonstrates that under-estimating the SD (a common outcome of assuming normality for non-normal data) artificially reduces standard errors, leading to inflated statistical significance (Type I errors) and misleading confidence intervals.
Performance of QE-beta:
- When population bounds ( $L, U$ ) are known, the QE-beta method consistently outperforms existing five-number summary methods (Wan, Shi, QE, BC) that treat these bounds as sample extremes.
- Existing methods often overestimate SD for skewed distributions when population bounds are used as sample extremes, whereas QE-beta remains stable.
- Even when only a lower bound ( $L$ ) is known (four-number summary), QE-beta provides more accurate estimates than standard three-number methods.

Significance and Recommendations
The paper concludes that estimating SDs from three-number summaries is inherently risky due to the inability to verify distributional assumptions. The authors do not claim that a single "best" estimator exists for all cases but rather advocate for a shift in practice:

Avoid SD Conversion if Possible: For meta-analyses, the authors recommend meta-analyzing medians directly using existing methods rather than converting to means/SDs.
Mandatory Sensitivity Analysis: If SD estimation is required, researchers must perform sensitivity analyses across multiple distributional shapes. They should report the range of plausible SDs and discuss how this variation impacts downstream inferences.
Utilize Population Bounds: When known population limits exist (e.g., age, test scores), researchers should use the proposed QE-beta approach, which leverages these bounds to constrain the shape of the distribution, significantly improving estimation accuracy.
Tool Availability: A Shiny web application is provided to facilitate these sensitivity analyses, allowing researchers to input three-number summaries and explore the resulting distribution of SD estimates across various models.

The authors emphasize that their work highlights the dangers of current practices and provides a framework for greater transparency and rigor, rather than offering a definitive solution that eliminates all estimation uncertainty.

Technical Summary: The Dangers of Using Three-Number Summaries to Estimate Unknown Standard Deviations

More like this