Exact and near-duplicate symptom profiles inflate the reported performance of machine learning classifiers for diabetes: a leakage- aware re-evaluation of the UCI Early-Stage Diabetes dataset

In the world of medical technology, machine learning has promised a future where computers can spot diseases like diabetes simply by reading a list of symptoms. The idea is powerful: a patient describes their feelings—thirst, fatigue, blurred vision—and an algorithm instantly calculates their risk, potentially catching the disease before it causes irreversible harm. To build these digital doctors, scientists rely on massive collections of data, where thousands of past patient records are used to teach the computer what a sick person looks like versus a healthy one. The standard for testing these systems is a method called cross-validation. Imagine a teacher giving a student a practice exam, then a slightly different version of the same exam, to see if the student truly understands the material or has just memorized the answers. In the scientific community, this process is supposed to reveal how well a model will perform on new, unseen patients.

However, a recent re-examination of one of the most famous datasets used for this purpose has uncovered a quiet flaw in how these tests were conducted. The dataset in question, gathered from a hospital in Bangladesh, contains records of people's symptoms and whether they had diabetes. For years, researchers have used this data to train computer models, reporting success rates that seemed almost too good to be true, often claiming accuracy levels between 97 and 99 percent. These numbers suggested that the models were nearly perfect. But a team of researchers from Peru has now shown that these high scores were not a sign of genius, but a result of a simple mistake in the testing process. They found that the data contained many identical copies of the same patient profile, and the way the tests were run allowed the computer to obtain the answers before the test even began.

The researchers, led by William Marin-Rodriguez and his colleagues, took a fresh look at the UCI Early-Stage Diabetes dataset. This collection holds 520 records, each describing a person's age and fifteen different symptoms, such as frequent urination or sudden weight loss. When they counted the unique combinations of these symptoms, they found something startling: out of the 520 records, there were only 251 truly unique profiles. This means that 269 of the records were exact duplicates of others, and many more shared the same symptoms but differed only in age. In the original studies, when the data was split into training and testing groups, these identical copies were often scattered randomly. A record might end up in the training group, teaching the computer the answer, while its exact twin ended up in the testing group. When the computer was tested on that twin, it didn't need to guess or diagnose; it simply recognized the pattern it had already memorized.

To see how much this inflated the results, the team ran the same computer models again, but this time they changed the rules of the game. Instead of splitting the data randomly, they kept all records with the same symptom profile together in the same group. This ensured that if a specific set of symptoms appeared in the training data, that exact set could never appear in the testing data. The computer was forced to learn the patterns of illness rather than just memorizing specific patient lists. The difference in the results was dramatic. Under the old, random method, the models achieved accuracy scores between 97 and 98 percent, matching the glowing reports from previous years. But when the researchers enforced the stricter, leakage-free rules, those accuracy numbers dropped to a range of 90 to 93 percent. While still impressive, this drop revealed that the previous claims of near-perfect performance were largely an illusion created by the data structure.

The study went further to measure not just whether the models were right or wrong, but how confident they were in their predictions. They looked at a specific metric called the Matthews correlation coefficient, which gives a more honest picture of performance when the number of sick and healthy people is uneven. Under the random testing method, this score was artificially high, overstated by as much as 0.16 points for some models. When the duplicates were removed from the test set, the scores fell significantly. More importantly, the ranking of the models changed. The model that had been crowned the best under the old rules, a complex system called a stacking ensemble, fell from first place to eighth place under the new rules. The model that actually performed best when tested fairly was a different type of tree-based algorithm known as Extra Trees. This shift proved that the choice of the "best" model was not a solid fact, but a result of the flawed testing method.

The researchers also investigated what happens when they account for "near-duplicates." In the dataset, many people shared the exact same symptoms but had different ages. The team tested a scenario where even these near-identical profiles were kept together, preventing the computer from learning from a 40-year-old's symptoms to predict the risk for a 45-year-old with the same symptoms. This stricter test lowered the performance of every single model even further. Interestingly, a very simple model that works by finding the single most similar past case performed almost as well as the complex ones under the first strict test, but its performance collapsed when the near-duplicates were removed. This suggested that the simple model was also relying on finding copies of itself in the training data rather than truly understanding the disease.

The implications of this work extend beyond just one dataset. The researchers argue that the high accuracy numbers reported for diabetes prediction in the past were likely inflated by this same issue of duplicate records. They found that the difference between the old and new results was large enough to swallow up the tiny improvements that new studies often claim to have discovered. If a new algorithm claims to be one or two percent better than the old ones, that improvement might just be noise caused by the way the data was split, rather than a real breakthrough. The team also noted that many of the models were poorly calibrated, meaning that while they might be right often, their probability scores did not reflect the true likelihood of the disease. For instance, some models were overly confident, while others were too hesitant, which could be dangerous if a doctor relied on those numbers to make a decision.

This re-evaluation does not mean that machine learning cannot help diagnose diabetes, nor does it suggest that the models are useless. It simply means that the benchmarks used to judge them have been flawed. The researchers emphasize that their findings are internal estimates based on a specific dataset and do not prove that these models work in real-world clinics. They point out that the data comes from a single hospital and lacks details on how the diabetes was confirmed, so the results cannot be directly applied to general screening without further testing. However, the study offers a clear path forward for scientists. They recommend that future studies must check for duplicate records and group them together during testing, report how uncertain their results are, and avoid declaring a single "best" model unless the difference is statistically significant. By cleaning up the testing process, the scientific community can ensure that the digital doctors of the future are judged on their true ability to learn, rather than their ability to obtain the answers.