Impact of Missing Data Imputation on The Structure of Real-World Clinical Datasets: A Comparison of Median, K-Nearest Neighbours, and MICE Approaches
This study demonstrates that k-nearest neighbours (KNN) imputation outperforms both median substitution and multiple imputation by chained equations (MICE) in preserving the distributional fidelity and multivariate structural integrity of real-world clinical datasets across varying levels of missingness.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of medical research, data is the lifeblood of discovery. Doctors and scientists rely on records of patient history, lab results, and treatment outcomes to understand diseases and improve care. However, real-world medical records are rarely perfect. Patients miss appointments, machines fail, or doctors simply forget to write down a specific measurement. These gaps, known as missing data, create a problem: if a researcher tries to analyze a list of numbers with holes in it, the results can be misleading or impossible to calculate. To fix this, researchers often fill in the blanks using a process called imputation. The most common, and simplest, way to do this is to look at all the numbers they have for a specific measurement, find the middle value, and paste that same number into every empty spot. While this is easy to do, it has a known flaw: it flattens the natural variety of the data, making a diverse group of patients look more identical than they really are. A more sophisticated approach involves using complex computer models to guess what the missing numbers might be based on how they relate to other information about the patient. The question facing the scientific community is whether the simple method is good enough, or if the complex method is necessary to keep the data honest.
A team of researchers from Valencia, Spain, set out to answer this question by testing three different ways of filling in the gaps in real medical records. They did not create fake data to test their theories; instead, they used two large, actual collections of patient information. The first group consisted of 408 patients who had undergone lung transplants, a complex procedure where doctors track dozens of measurements before, during, and after surgery. The second group included 1,700 patients who had suffered a heart attack, with records detailing their vital signs and medical history. In both groups, many numbers were missing. In the lung transplant group, some measurements were missing from as few as 0.2 percent of patients to as many as 77.9 percent. In the heart attack group, missing data ranged from 0.24 percent to over 63 percent. The researchers wanted to see which method of filling in these blanks best preserved the true shape and relationships of the original data.
The team compared three specific strategies. The first was the standard, simple approach: replacing every missing number with the median, or middle value, of the numbers that were actually recorded. The second was a method called k-nearest neighbours, which works by finding the patients in the database who are most similar to the one with the missing data and using their values to make a guess. The third was a highly complex statistical technique known as multiple imputation, which runs a series of sophisticated models to generate several different possible versions of the missing data and then averages them. The researchers then measured how closely the filled-in data matched the original, complete data using a statistical ruler that checks if the distribution of numbers looks the same. They also checked if the relationships between different variables, such as how age might relate to blood pressure, remained intact.
The results were surprising to many who expect the most complex method to always be the best. The simple method of using the middle value did indeed distort the data, compressing the variety of numbers and making the dataset look less diverse than it truly was. However, the complex, multi-step statistical model performed even worse in this specific test. It introduced the largest errors and altered the relationships between variables more than any other method. The method that performed the best was the k-nearest neighbours approach. It preserved the natural spread of the data and kept the relationships between different medical measurements more accurate than either the simple middle-value method or the complex model. In the lung transplant group, the complex model distorted the data for nearly 39 percent of the variables, while the k-nearest neighbours method only distorted about 11 percent. In the heart attack group, the complex model distorted 39 percent of the variables, while the k-nearest neighbours method distorted only 11 percent.
The researchers found that as the amount of missing data increased, all methods made more mistakes, but the k-nearest neighbours method made mistakes much more slowly. When a variable had a high percentage of missing numbers, the simple middle-value method and the complex model both struggled to keep the data looking real, but the k-nearest neighbours method held up better. This is likely because the complex model tries to build a perfect mathematical rule for every single variable, which becomes difficult when there are many variables and not enough patients to learn from. The k-nearest neighbours method, by contrast, simply looks at the most similar patients and copies their pattern, which works well even when the data is messy or incomplete.
This study does not suggest that the complex statistical model is useless. That model is designed for a different purpose: to provide a range of possible answers and to account for the uncertainty of not knowing the true numbers, which is crucial for certain types of deep statistical analysis. However, for many researchers who simply need a clean, complete dataset to describe a group of patients or to explore relationships between variables, the complex model may be overkill and even counterproductive. The findings suggest that for creating a single, reliable version of a medical dataset, the k-nearest neighbours method offers a powerful balance. It is far more accurate than the simple middle-value method, which is still widely used, and it is easier to use than the complex model, requiring fewer technical decisions from the researcher.
The study highlights a practical lesson for medical science: sometimes, the most sophisticated tool is not the best one for the job. In the messy reality of hospital records, where data is often incomplete and patient profiles are unique, a method that looks at similar patients to fill in the gaps can preserve the true story of the data better than a rigid mathematical formula. By choosing the right tool, researchers can ensure that the patterns they discover in their data reflect the reality of the patients they are studying, rather than the artifacts of how they tried to fix the missing numbers. This work provides a clear path forward for scientists who want to keep their data honest without getting bogged down in unnecessary complexity.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.