💻 computer science

Data leakage inflates reported accuracy of deep learning for kidney stone detection on CT: a leakage-free, calibrated and uncertainty-aware reassessment across three centers

This study demonstrates that image-level data splitting in deep learning models for kidney stone detection on CT artificially inflates reported accuracy by approximately nine percentage points due to data leakage, and advocates for patient-wise, calibrated, and uncertainty-aware evaluation protocols to ensure reliable clinical performance.

Original authors: Okoi Obeten, Abas Aliu, Omowumi Aliu, Ahmed Jimoh, Zibril Aliyu, Christiana Ezeanya

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Okoi Obeten, Abas Aliu, Omowumi Aliu, Ahmed Jimoh, Zibril Aliyu, Christiana Ezeanya

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medical imaging, computers are increasingly asked to perform tasks that once required a human eye. One such task is spotting kidney stones, small and painful mineral deposits that can block the flow of urine. To find these stones, doctors rely on computed tomography, or CT scans, which take hundreds of cross-sectional images of the body. A radiologist must look through these slices to find the tiny, bright spots that indicate a stone, a process that is time-consuming and prone to human error. Because of this, researchers have spent years teaching artificial intelligence to do the spotting automatically. The goal is to create a digital assistant that can flag stones instantly, helping doctors work faster and more accurately. For this technology to be trusted, however, the computer must prove it can recognize a stone it has never seen before, not just memorize a picture it has already studied.

A new study from researchers in Nigeria has uncovered a critical flaw in how many of these computer programs have been tested. For years, reports have claimed that artificial intelligence models can detect kidney stones with near-perfect accuracy, often citing success rates above 98 percent. These numbers suggested that the technology was nearly ready for the clinic. However, the researchers found that these high scores were largely an illusion created by a mistake in the testing process. When the same computer models were tested again using a stricter, more honest method, their performance dropped significantly. The study reveals that the previous high scores were not a sign of superior intelligence, but rather a result of the computer utilizing information from the test set during the training phase.

The problem lies in how the data is prepared for training. To teach a computer to recognize kidney stones, researchers take a limited number of real CT scans and create many slightly altered copies of them. They might flip an image sideways, rotate it slightly, or adjust the brightness. These altered copies, known as augmented images, help the computer learn the features of a stone without needing thousands of unique patients. The error occurs when these original images and their altered copies are mixed together and then split into training and testing groups. If the computer sees an original image during its training phase, and then sees a nearly identical altered copy of that same image during the testing phase, it does not need to learn what a stone looks like. It simply recognizes the pattern it has already memorized. It is like a student who memorizes the answers to a practice quiz and then takes the final exam with the same questions slightly reworded; they will get a perfect score, but they have not actually learned the subject.

The researchers in this study took a dataset containing images from 201 patients across three different hospitals. They first recreated the testing method used in previous studies, which mixed the original and altered images together before splitting them. Under these conditions, the computer model achieved an accuracy of 99.1 percent and a score of 1.000 on a scale where 1.000 represents perfect performance. This result matched the glowing reports found in earlier scientific papers. However, the researchers then changed the rules. They ensured that the computer was never allowed to see any image from a patient during the testing phase if it had already seen any image from that same patient during training. This meant that every single image in the test set came from a patient the computer had never encountered before.

When the model was forced to face this stricter challenge, the results changed dramatically. The accuracy fell to 90.5 percent, and the performance score dropped to 0.946. While this is still a strong result, it is far from the near-perfect numbers previously celebrated. The difference of nearly nine percentage points represents a massive gap between what was reported and what is real. The study showed that the previous high scores were not a reflection of the computer's ability to diagnose disease, but a reflection of a testing protocol that allowed the computer to utilize information from the test set during training. This finding suggests that many of the impressive statistics seen in the field of medical artificial intelligence may be inflated by the same mistake.

The researchers also investigated whether more complex computer architectures were necessary to solve this problem. They compared their main model, which combined two different types of advanced computing structures, against a simpler, standard model. They found that the complex model offered no real advantage over the simpler one when tested fairly. In fact, a model that relied only on the more complex structure without the standard foundation performed almost no better than random guessing. This indicates that the key to success was not the novelty of the design, but the honesty of the testing method. The study also looked at how the computer handled uncertainty. They found that the model was often overconfident, claiming to be sure when it was actually wrong. By teaching the model to admit when it was unsure and to defer difficult cases to a human doctor, the accuracy of the cases it did answer rose to 95.2 percent. This suggests that the most useful role for such a tool is not to replace the doctor, but to act as a second pair of eyes that knows when to ask for help.

The implications of this work extend beyond kidney stones. The study serves as a warning to the entire field of medical artificial intelligence. It demonstrates that a model can appear brilliant in a research paper while failing to generalize to real-world patients if the testing is flawed. The researchers emphasized that for these tools to be safe and effective in hospitals, they must be tested on data that truly represents new, unseen patients. They also noted that the dataset they used had limitations, such as a lack of detailed patient information and the fact that the images were processed in a way that removed some of the raw data doctors use to make decisions. Despite these limitations, the core message is clear: the path to reliable medical AI requires rigorous, leakage-free testing. Until the field adopts these stricter standards, the reported success rates of these technologies will remain an overestimate, potentially leading to false confidence in tools that are not yet ready for the clinic.

Technical Summary: Data Leakage in Kidney Stone Detection on CT

Problem Statement
Deep learning models for kidney stone detection on non-contrast CT frequently report accuracies exceeding 98%. However, this literature often relies on evaluation protocols that introduce data leakage. Specifically, many studies augment datasets and then partition them at the image (slice) level. Because augmented images are near-duplicates of their source slices, this practice allows training data to appear in the test set, artificially inflating performance metrics. Furthermore, reported accuracies often lack calibration, uncertainty quantification, and external validation across diverse centers, raising questions about whether these numbers reflect diagnostic capability or merely the conditions of the evaluation protocol.

Methodology
The study reassessed a deep learning detector using a multi-center CT dataset comprising 3,364 original axial images from 201 patients across three centers in the Kurdistan Region of Iraq. The dataset was expanded via twelve augmentation strategies to 35,457 augmented images.

Leakage-Free Protocol: The primary evaluation used a patient-wise partition. Patients were stratified into training (143), validation (29), and test (29) sets. Crucially, all images belonging to a patient (including augmented derivatives) were confined to a single split. The validation and test sets contained only original, unaugmented images.
Leakage Experiment: To quantify the inflation caused by leakage, the identical model architecture and training budget were retrained under a random image-level split of all 38,821 images (original + augmented), replicating protocols used in prior literature.
Model Architecture: A hybrid CNN-Transformer detector was employed, utilizing a pretrained ResNet-18 backbone to extract local features, projected into a token grid, and processed by two Transformer encoder blocks. This was compared against matched comparators: a plain ResNet-18, a custom CNN trained from scratch, an EfficientNet-B0, and a Transformer trained without a convolutional stem.
Robustness and Calibration: The study employed patient-grouped five-fold cross-validation, leave-one-hospital-out testing, and multi-seed repetition. Post-hoc calibration (temperature scaling), split conformal prediction, and selective classification (deferral) were applied to assess uncertainty and reliability. Saliency was examined using Grad-CAM and attention maps.

Key Results

Impact of Data Leakage: The leakage-free, patient-wise protocol yielded an AUC of 0.946 (95% CI 0.925–0.967) and an accuracy of 0.905. In contrast, the leaky image-level split inflated accuracy to 0.991 and AUC to 0.999 (effectively 1.000). Across three random seeds, the image-level split consistently produced an accuracy approximately 8.8 percentage points higher than the patient-wise split ( $P = .021$ ). The authors note that the inflated 0.991 accuracy closely matches the 98.8% accuracy previously reported for a different model on this dataset, suggesting prior high scores were artifacts of the evaluation protocol.
Architectural Comparisons: Under the leakage-free protocol, a plain fine-tuned ResNet-18 performed statistically indistinguishably from the hybrid CNN-Transformer model (AUC 0.968 vs. 0.965; McNemar $P = .64$ ). A Transformer trained without a convolutional stem performed near chance (AUC 0.596), indicating that the pretrained convolutional stem, rather than the attention mechanism, drives performance at this cohort size.
Generalization: Performance held reasonably well across held-out hospitals (AUCs ranging from 0.901 to 0.973), though a drop was observed on the smallest, most distinct center, quantifying a real generalization penalty.
Calibration and Uncertainty: The uncalibrated model was overconfident (Expected Calibration Error 0.061), which was reduced to 0.045 via temperature scaling. Selective classification proved effective: deferring the least confident 20% of slices raised the accuracy of the retained cases to 0.952. Conversely, conformal prediction at a 95% coverage level degenerated to returning both labels for every case, offering no useful signal.
Subgroup Analysis: An initial analysis on a single test partition suggested a significant sex disparity in sensitivity. However, a pooled analysis across all 201 patients using patient-level clustering revealed no significant sex disparity (difference in sensitivity: -0.013, 95% CI -0.044 to +0.018). The apparent disparity was an artifact of the partition and inappropriate unit of analysis.
Saliency: Grad-CAM and attention maps did not localize specifically to renal calculi but were distributed over central abdominal structures, indicating that high slice-level accuracy does not necessarily equate to radiologically aligned reasoning.

Significance and Claims
The paper claims that image-level splitting of augmented data substantially overstates real-world performance in kidney stone detection, inflating accuracy by roughly nine percentage points and driving AUC to unity. The authors argue that reported accuracies of ~99% on this dataset are properties of the evaluation protocol rather than the models themselves.

The study establishes that:

Evaluation Protocols Matter: Patient-wise partitioning is essential for honest assessment; image-level splitting on augmented data produces irreproducible, overoptimistic findings.
Architectural Novelty vs. Baselines: On cohorts of this size, standard pretrained convolutional backbones (ResNet-18) perform as well as more complex hybrid architectures, and attention-only models fail without convolutional inductive bias.
Clinical Utility via Deferral: A single headline accuracy figure is insufficient. The model's utility is better described by a policy of calibrated probabilities and selective classification, where the model abstains on low-confidence cases, thereby improving reliability for the cases it does report.
Methodological Rigor: Subgroup analyses must be pooled over held-out folds and clustered by patient to avoid misleading conclusions about disparities.

The authors conclude that for deep learning in stone detection to be clinically viable, studies must explicitly report partitioning units, evaluate on unaugmented images from held-out patients, and accompany accuracy metrics with calibration and abstention behavior. They emphasize that the model, while promising as a triage or second-reader tool, currently requires domain adaptation for new scanners and cannot rely solely on threshold shifting to improve sensitivity without collapsing specificity.

Technical Summary: Data Leakage in Kidney Stone Detection on CT

More like this