📊 statistics

The Smallest Standard Error Is the Wrong One: Selection Among Multiple Imputations Invalidates Inference

This paper demonstrates that selecting the multiply imputed analysis with the smallest standard error systematically invalidates statistical inference by discarding essential between-imputation variance and artificially depressing within-imputation variance, leading to severely under-covered confidence intervals and biased estimates, whereas Rubin's pooling rules correctly preserve calibration.

Original authors: Nyiko Muhluri Khoza¹, Bhekisipho Twala², Jean-Claude Malela-Majika³

Published 2026-10-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nyiko Muhluri Khoza¹, Bhekisipho Twala², Jean-Claude Malela-Majika³

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data analysis, missing information is a constant companion. Whether tracking patient recovery in a hospital, measuring crop yields in a field, or studying economic trends, researchers rarely have a perfect set of numbers. Some entries are blank, lost, or never recorded. To make sense of these gaps, statisticians use a technique called multiple imputation. Instead of guessing a single value to fill a blank spot, they create several different, plausible versions of the missing data. They then run their analysis on each version separately. This approach acknowledges that the missing numbers are unknown and that different guesses could lead to slightly different conclusions. By combining the results from all these versions, analysts can produce a final answer that honestly reflects both the data they have and the uncertainty about what they are missing. The goal is to ensure that the reported confidence in a result matches the reality of the situation.

A team of researchers from South Africa recently discovered that a seemingly logical shortcut in this process actually breaks the math. They found that when analysts generate multiple versions of their data and then pick the single version that looks the most precise—specifically, the one with the smallest margin of error—they are tricking themselves. This choice, which looks like a way to sharpen the focus of a study, systematically hides the true amount of uncertainty. The researchers showed that by selecting the "best-looking" result from a group of candidates, analysts discard the very information that makes the method reliable. They proved that this practice leads to confidence intervals that are too narrow and point estimates that are subtly biased, making the findings appear more certain than they truly are.

The study began by examining a natural temptation in statistical work. When a researcher has twenty different completed datasets, each with its own calculated estimate and a measure of how much that estimate might vary, it is tempting to choose the one with the smallest variation. A smaller variation suggests a tighter, more precise result, which is usually what scientists want to report. The researchers, who had previously defended this very approach, decided to test it rigorously. They set up a controlled simulation where they knew the true answer in advance. They generated thousands of datasets with missing values, created twenty different imputed versions for each, and then applied the selection rule: pick the version with the smallest estimated error.

The results were stark. In their simulations, where the true answer was known, the method of picking the smallest error produced confidence intervals that covered the true answer only about 84.5% of the time, when they should have covered it 95% of the time. This means the method was failing to capture the truth more often than it should. The researchers identified three distinct reasons for this failure. First, by picking just one dataset and treating it as if it were the only real data, the method threw away the variation between the different guesses. This variation is crucial because it represents the uncertainty about the missing numbers. Second, the act of choosing the smallest number from a group of twenty naturally biases the result downward. Just as the shortest person in a room of twenty is likely shorter than the average person in the general population, the smallest error estimate from a group of twenty is likely smaller than the true average error. Third, because the estimate of the value and the estimate of its error are often linked, picking the smallest error inadvertently picks a specific kind of value, shifting the final answer away from the truth.

To illustrate the severity of the problem, the researchers compared this selection method against the standard, accepted way of handling multiple imputations, known as Rubin's rules. Instead of picking one winner, Rubin's rules combine all twenty guesses into a single, averaged result. When the researchers applied this standard method to the exact same set of twenty guesses, the results were perfectly calibrated. The confidence intervals covered the true answer 94.6% of the time, right on target. The selection method, by contrast, looked more precise on paper, reporting a narrower range of uncertainty, but that precision was an illusion bought at the cost of accuracy.

The study also explored what happens when researchers increase the number of guesses they generate. Standard advice suggests that using more imputations leads to better, more stable results. However, for the selection method, the opposite occurred. As the researchers increased the number of candidate datasets from two to one hundred, the performance of the selection method got worse. The confidence intervals became even less reliable, dropping to cover the true answer only 81.7% of the time, while the bias in the final answer grew larger. In contrast, the standard method of combining all guesses remained stable and accurate regardless of how many guesses were used. This revealed a dangerous trap: an analyst following the advice to use more data to be more careful would, by using the selection method, actually be making their inference less reliable without any warning signs in their output.

The researchers concluded that the standard error cannot serve as both the goal of an analysis and the judge of its success. You cannot optimize for the smallest error and then trust that error as a honest measure of uncertainty. They proposed a new protocol for evaluating statistical methods: before comparing how precise different methods are, researchers must first check if their reported uncertainty matches the actual variability of the results. They called this a "calibration-first" approach. If a method reports an error that is too small compared to the truth, it should be rejected immediately, no matter how narrow its confidence intervals look. This rule would prevent the selection method from ever appearing to be a winner in a fair comparison.

This work serves as a correction to a specific, attractive idea in data science. The researchers admitted that they had previously proposed the selection method themselves, believing it to be a valid refinement. By re-examining their own work with a rigorous simulation that knew the truth, they demonstrated that the method was fundamentally flawed. They emphasized that this failure is not a minor technicality but a structural issue that invalidates the statistical inference. The solution they offer is simple: do not pick a winner. Instead, combine all the candidates. By pooling the results, the method preserves the necessary uncertainty and produces valid, trustworthy conclusions. The study stands as a reminder that in the pursuit of precision, the most attractive shortcut can sometimes be the path that leads away from the truth.

Technical Summary: "The Smallest Standard Error Is the Wrong One: Selection Among Multiple Imputations Invalidates Inference"

1. The Problem

The paper addresses a specific methodological flaw in the analysis of missing data: the practice of generating multiple imputed datasets, analyzing each, and then selecting the single candidate with the smallest estimated standard error (SE) for reporting. While this approach appears to maximize precision, the authors demonstrate that it systematically invalidates statistical inference.

The core issue is that selecting the minimum SE from a set of $M$ candidates introduces three distinct, compounding mechanisms of error:

Discarding Between-Imputation Variance: By treating the selected dataset as if it were observed, the procedure ignores the variability between imputations (the $B$ term in Rubin's rules), which is essential for capturing uncertainty due to missing data.
Order-Statistic Bias: Selecting the minimum of $M$ estimated variances introduces a downward bias purely due to the act of minimization. As $M$ increases, the expected value of the minimum decreases, further understating the true within-imputation variance.
Selection-Induced Bias: Since point estimates and their standard errors are often dependent (e.g., in skewed distributions or sample means), selecting based on the SE constitutes a non-random selection on the point estimate itself. This introduces bias into the final estimate that is undetectable from the output of a single analysis.

The authors argue that using the standard error as both an optimization criterion and a performance metric creates a circular validation where a method "wins" simply because it optimizes the metric by which it is judged, rendering the comparison uninformative.

2. Methodology

The paper employs a rigorous Monte Carlo simulation framework to isolate and quantify these effects.

Data Generation: The study uses a linear data-generating process with covariates drawn from a multivariate normal distribution. Missingness is imposed under three regimes: Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR).
Imputation Mechanism: A "proper" Bayesian linear regression imputation model is used to generate $M$ completed datasets. This ensures that the candidate draws themselves are statistically valid; the failure is isolated strictly to the combination step.
Procedures Compared:
- RDOI (Random-Draw Observations Imputation): The flawed procedure where the candidate with the minimum SE is selected.
- RDOI-R: The corrected procedure where the same $M$ candidates are pooled using Rubin's rules.
- RDOI-SC: A selection-retaining variant that attempts to correct the variance estimate by adding back the between-imputation component and a penalty for the displacement of the point estimate.
- Comparators: Complete-case analysis, mean imputation, deterministic/stochastic regression, and k-nearest neighbors.
Evaluation Protocol: The authors propose a "calibration-first" protocol. Instead of ranking methods by reported SE, they introduce the calibration ratio ( $\rho$ $ρ$ ), defined as the mean reported standard error divided by the empirical standard deviation of the estimator across replications.
- $\rho \approx 1.0$ : Honest uncertainty reporting.
- $\rho < 1.0$ : Overconfidence (under-reported variance).
- Admissibility is determined by whether $\rho$ falls within a tolerance (e.g., [0.9, 1.1]) before efficiency (interval width) is considered.

3. Key Results

The simulation results, conducted with $n=400$ , 30% missingness, and $M=20$ (with sweeps up to $M=100$ ), yield the following findings:

Invalidity of Selection: Under MAR, the RDOI (minimum-SE selection) procedure produced a calibration ratio of 0.77 and a confidence interval coverage of 84.5% for a nominal 95% interval. In contrast, Rubin pooling (RDOI-R) of the identical candidate draws achieved a calibration ratio of 1.00 and 94.6% coverage.
The $M$ Sweep: As the number of imputations ( $M$ ) increased from 2 to 100, the performance of the selection procedure degraded monotonically. Coverage dropped from 86.7% to 81.7%, and the magnitude of point-estimate bias increased from 0.021 to 0.079. Conversely, Rubin pooling remained stable and well-calibrated.
Bias Mechanism: The selection procedure introduced a downward bias in the point estimate (-0.063) even though the underlying imputation model was correctly specified and unbiased. This confirms that the selection rule itself distorts the center of the inference.
Comparison with Other Methods: Mean imputation achieved the smallest reported SE (0.104) but had the worst coverage (23.3%), illustrating that minimizing reported SE is a poor proxy for accuracy.
Sensitivity: The failure persisted across sample sizes ( $n=100$ to $1000$) and missing rates (10% to 60%). Under MNAR, while variance calibration remained near 1.0 for pooled methods, coverage collapsed due to model bias, highlighting that calibration is necessary but not sufficient for validity.

4. Key Contributions

Formal Identification of Failure Modes: The paper formally delineates the three mechanisms (discarded between-variance, order-statistic bias, and selection-induced point estimate bias) that cause minimum-SE selection to fail.
The Calibration-First Protocol: The authors propose a new evaluation standard where methods must first pass a calibration gate (honest reporting of uncertainty) before their efficiency is ranked. This prevents the "optimization artifact" where a method appears superior simply because it minimizes the criterion used to judge it.
Corrected Procedures:
- RDOI-R: Recommends abandoning selection entirely in favor of Rubin pooling.
- RDOI-SC: Offers a fallback for workflows requiring a single completed dataset, using an inflated variance estimator to restore coverage, though at the cost of wider intervals.
Reproducibility: The authors provide a fully frozen, reproducible Monte Carlo implementation (including seeds and data generation scripts) that regenerates all reported results, correcting numerical discrepancies from their own earlier, unaccepted work.

5. Significance and Claims

The paper claims that the practice of selecting the "best" imputed dataset based on standard error is a systematic error that renders inference invalid. The authors explicitly withdraw their earlier defense of this procedure (RDOI/RDCOI), acknowledging that their previous reliance on standard-error comparisons was flawed.

The significance of the work lies in its critique of benchmark design in missing data literature. The authors argue that:

A quantity that a procedure directly optimizes (like SE) cannot simultaneously serve as an unbiased performance metric.
Benchmarks that rank methods solely on reported standard errors or value-recovery accuracy are fundamentally flawed because they cannot detect under-coverage or overconfidence.
The "calibration ratio" is a low-cost, high-value diagnostic that should be mandatory in simulation studies to distinguish between honest uncertainty and artificial precision.

The paper concludes that the failure is not a matter of degree but of principle: an uncertainty estimate cannot serve as both an optimization criterion and retain its usual inferential interpretation. The authors urge the community to adopt a protocol where validity (calibration) is established before efficiency is assessed.