Outcome labels and patient reuse in clinical validation: reanalyses of the DIPK and TIL-30 breast-cancer evaluations
This paper reanalyzes breast-cancer clinical validations of the DIPK and TIL-30 models to demonstrate how misidentified outcome labels and patient data reuse can artificially inflate reported performance metrics, urging researchers to verify outcome definitions and patient independence before citing validation results.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
In the fight against cancer, doctors increasingly rely on computer models that read a patient's genetic code to predict how a tumor will respond to treatment. These models act like a crystal ball, analyzing thousands of genes to guess whether a drug will shrink a tumor completely or leave it behind. For these predictions to be trusted, they must be tested against real-world outcomes: did the patient actually get better? The gold standard for this check is a look at the tissue under a microscope after chemotherapy, known as a pathological exam. If a model claims to predict a complete cure, scientists must verify that claim by comparing the computer's guess to the actual microscope results. However, the data used to build and test these models often comes from public archives, vast digital libraries where researchers store genetic information and clinical notes. The integrity of these archives is paramount; if the notes describing a patient's outcome are mistaken, or if the same patient appears twice under different names, the entire test of the model's accuracy can be compromised.
A recent investigation by a researcher at the University of Edinburgh took a close look at two such models, both designed to predict how breast cancer patients would respond to chemotherapy. The study was not a new experiment but a careful re-examination of existing data, asking a simple but critical question: did the researchers who originally published these models test them against the right answers? The first model, called DIPK, was a deep-learning system trained on cancer cells grown in a lab and then tested on real patients. The second, TIL-30, was a simpler tool based on thirty specific genes. The researcher traced the digital footprints of the data used in their published success stories and found that in both cases, the tests had been run against the wrong targets.
In the case of DIPK, the researchers had tested the model's predictions against a dataset of 310 patients. The problem was that the "correct answers" in that dataset were not the actual microscope results from the patients' surgeries. Instead, the data contained predictions made by a different, older computer program called DLDA30. It is like grading a student's essay not by checking the facts, but by seeing if their essay agrees with another student's essay. Because both computer programs were looking at the same genetic patterns, they naturally agreed with each other, making the DIPK model look much more accurate than it truly was. When the researcher replaced those computer-generated answers with the actual microscope results from the patients, the model's performance score dropped significantly. It did not disappear entirely; the model still showed some ability to distinguish between patients who responded well and those who did not, but its apparent success was largely an illusion created by comparing it to a similar computer program rather than to reality.
The second model, TIL-30, faced a different kind of error. This model was designed to predict a complete cure after a full course of treatment. However, the data used to test it in one specific group of patients recorded the outcome after only the first stage of chemotherapy. It is similar to judging a marathon runner's final time based on their speed after the first mile. The model was being tested against a snapshot of the patient's condition that was too early to tell the final story. When the researcher corrected the data to reflect the actual final outcome after all treatment was finished, the model's ability to predict the result dropped, though the statistical evidence for this decline remained uncertain.
Beyond the wrong answers, the study also uncovered a problem with who was being tested. In scientific studies, every patient counted must be a unique individual to ensure the results are robust. The researcher found that in the DIPK study, a large number of patients from one group were actually the same people as those in another group, just listed under different file names. This meant the model was being tested on the same people twice, inflating the sense of confidence in its results. Even more concerning, for the TIL-30 model, many of the patients used to test the model in one group were the exact same people who had been used to teach the model how to work in the first place. This is like a student taking a test that includes questions they were just taught in class; it does not prove they understand the material, only that they remember the answers. While this overlap clearly compromised the separation between training and testing, the specific effect of this reuse on the model's performance score was not estimated in the study.
The investigation did not stop at finding errors; it also checked whether these models offered any real value beyond what doctors already know. Doctors currently use a set of seven standard clinical factors, such as age and tumor type, to estimate a patient's chances of recovery. The researcher built new statistical models to see if adding the computer scores to these seven factors improved the prediction. The results were inconclusive. While the computer scores did not hurt the prediction, the data was not strong enough to prove that they added any meaningful new information. In some tests, the computer scores seemed to help, but the margin of error was so wide that the improvement could have been a fluke. Furthermore, when the researcher tried to apply the models trained on one group of patients to a different group, the predictions became wildly inaccurate, often guessing that a quarter of patients would be cured when only a fifth actually were, or vice versa.
The study concludes that these two specific cases serve as a warning for the entire field of cancer research. The models themselves were not necessarily useless; the DIPK model did retain a connection to real patient outcomes even after the errors were fixed. However, the published claims of their success were built on shaky ground. The "success" relied on comparing predictions to other predictions, testing on incomplete treatment stages, and reusing the same patients in ways that hid the true difficulty of the task. The researcher emphasizes that for any new medical tool to be trusted, scientists must verify three things: that the outcome being predicted is the actual final result, that the patients in the test are truly distinct individuals, and that the tool adds something new to what doctors already know. Without these checks, a model might look like a breakthrough on paper while failing to deliver in the clinic.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.