Predicting missing values: A good idea?

Technical Summary: Predicting Missing Values. A good idea?

Problem Statement
The paper addresses a critical disconnect in the handling of missing data within machine learning and statistical analysis. While minimizing Mean Squared Error (MSE) is the dominant criterion for evaluating imputation methods—prioritizing the accuracy of point estimates—this approach systematically introduces biases in downstream analyses. The core issue is that MSE-optimized imputations are essentially conditional means (averages). By replacing missing values with these deterministic averages, the natural variability of the data is artificially reduced. This reduction distorts key statistical parameters, including variance, prevalence, correlation, regression slopes, and explained variance ($R^2$), leading to invalid inferences and overconfident estimates in subsequent modeling.

Methodology
The author employs a theoretical framework and simulation studies to compare two distinct imputation strategies within a multivariate normal setting:

1. Predictive Imputation (Predict): This method minimizes MSE by imputing the expected value ($\hat{y}$) based on observed predictors. It is deterministic and corresponds to regression imputation or conditional mean imputation.
2. Stochastic Imputation (Draw): This method adds random noise ($\epsilon$) to the predicted value, such that $\dot{y} = \hat{y} + \epsilon$. The noise is drawn from the residual distribution of the imputation model. This approach acknowledges the inherent uncertainty of missing data.

The study utilizes a simulation setup with synthetic populations representing "high-signal" ($R^2=0.8$) and "low-signal" ($R^2=0.2$) data. Missingness is introduced via two mechanisms: Missing Completely at Random (MCAR) and Missing at Random (MAR). The performance of these methods is evaluated not just by MSE, but by their impact on downstream parameters ($\mu, \sigma, P90, \rho, \gamma, R^2$) in regression models where the imputed variable serves as both a predictor and an outcome.

Furthermore, the paper evaluates three popular software tools—missForest, softImpute, and mice—to assess whether their default behaviors align with predictive or stochastic paradigms.

Key Results

• Bias in Predictive Methods: The "predict" method consistently reduces the standard deviation ($\sigma$) and the proportion of cases exceeding the 90th percentile ($P90$). Conversely, it inflates correlations ($\rho$), regression slopes ($\delta$), and the proportion of explained variance ($R^2$). These biases are most severe in low-signal scenarios where the relationship between variables is weak.
• Validity of Stochastic Methods: The "draw" method, which incorporates noise, produces estimates for variance, correlation, and slopes that closely align with the true population parameters, effectively eliminating the systematic biases introduced by predictive methods.
• The MSE Paradox: While the "draw" method yields unbiased downstream estimates, it results in an MSE approximately twice as large as the "predict" method ($MSE_{draw} \approx 2 \times MSE_{predict}$). Relying solely on MSE would incorrectly identify the biased predictive method as superior.
• Software Evaluation:
  • missForest behaves similarly to the predictive method, introducing significant bias.
  • softImpute shows mixed results; it performs well on variance in high-signal settings but exhibits excessive spread and high MSE in low-signal settings, while still overestimating explained variance.
  • mice (using its default Bayesian framework) produces the most accurate downstream estimates, preserving data variability. The paper notes a minor issue with $P90$ in mice when using predictive mean matching, which can be resolved by switching to normal imputation.

Key Contributions

1. Demonstration of Systematic Bias: The paper provides empirical evidence that minimizing MSE for imputation systematically distorts downstream statistical properties, specifically by compressing the data distribution toward the conditional mean.
2. Noise as a Solution: It proposes and validates the addition of noise (stochastic imputation) as a simple, effective mechanism to restore the natural variability of the data and ensure unbiased downstream analysis.
3. Critique of Evaluation Metrics: The work challenges the machine learning community's reliance on MSE as the primary metric for imputation quality, arguing that it prioritizes point accuracy over the statistical validity of the data distribution.
4. Practical Tool Assessment: It offers a comparative analysis of widely used imputation tools, highlighting that many popular "black box" methods (like missForest) inadvertently perpetuate the biases of deterministic imputation.

Significance and Claims
The paper argues that the goal of imputation should not be to perfectly recreate the true missing values (which is impossible), but to facilitate valid and informative downstream analyses. The author asserts that "predicting missing values" is only a "good idea" if the prediction process embraces the addition of noise to preserve data variability.

The significance lies in shifting the paradigm from accuracy (minimizing MSE) to validity (preserving variability). The paper concludes that incorporating noise is essential to prevent biases in variance, correlation, and slope estimates. While acknowledging that adding noise increases the MSE, the author posits that this is a necessary trade-off to account for the irreducible error inherent in missing data. The work bridges statistical rigor with practical utility, suggesting that stochastic approaches are superior for handling incomplete data in scenarios requiring reliable inference.