Cethraian-X: A Leakage-Clean, Multi-Seed Benchmark of Chest X-Ray Classification Under Weak Labels

Imagine you are trying to teach a computer to be a doctor by showing it millions of pictures of human chests. This is the world of medical artificial intelligence, a field where machines learn to spot diseases like pneumonia or broken bones just by looking at X-rays. But here's the tricky part: computers are like super-observant students who sometimes cheat. If you show them a picture of a patient's arm in a specific hospital's X-ray room, the computer might learn to say, "Oh, that hospital always has pneumonia!" instead of actually looking for the disease. This is called "data leakage," and it's like letting a student peek at the answer key before the test. To get a fair grade, scientists have to make sure the computer has never seen the test questions before, and that it isn't just memorizing tiny, invisible patterns in the pixels that happen to repeat. This paper dives into that exact problem, asking: if we clean up the data really, really well, does the computer's score actually change, or was it just lucky?

Enter Cethraian-X, a new "clean room" experiment designed to test a computer's ability to read chest X-rays without any cheating. The researchers took a massive, famous collection of 112,120 chest X-rays and put them through a rigorous audit. Think of it like a detective cleaning up a crime scene: they removed any duplicate photos, checked that no patient appeared in both the "practice" group and the "final exam" group, and even scanned the digital code of the images to make sure no two pictures were secretly identical twins. They wanted to see if the computer's performance would drop once these sneaky shortcuts were blocked.

The computer they tested was a standard, well-known AI model called DenseNet-121, which is like a very smart but basic student. The researchers ran this student through the exam three times, using three different "random seeds" (think of these as different starting points for the student's brain) to see if the results were consistent. They kept everything else exactly the same: the same pictures, the same rules, and the same way of grading.

Here is what they found. After scrubbing the data clean of all those duplicate and overlapping images, the computer's overall score barely budged. It achieved a macro ROC-AUC of 0.80976 ± 0.00027. To put that in perspective, the original, slightly "dirty" version of the test had a score of 0.81030. The difference was tiny—just -0.00054. This suggests that for this specific computer and this specific dataset, the "cheating" wasn't actually boosting the score very much. The computer was stable; it got the same result whether the data was messy or perfectly clean.

However, the paper has a very important "but." While the computer was good at ranking diseases (saying "this patient is more likely to have a nodule than that one"), it was terrible at giving real probabilities. When the computer said there was a "70% chance" of a disease, it wasn't actually 70%. The researchers tried to fix this with a technique called "temperature scaling," which is like adjusting the thermostat on the computer's confidence, but even after the fix, the numbers were still off. The "Brier skill" score, which measures how good a prediction is compared to just guessing the average, was deeply negative (-1.32902). This means the computer's confidence numbers were worse than useless for making real medical decisions.

The study also looked at specific diseases. For something common like Effusion (fluid in the lungs), the computer was decent. But for rare things like Hernia, the computer seemed to do amazingly well with a score of 0.91302, but the researchers warn this is misleading because there were only 86 positive cases in the test group—a tiny number that makes the score unstable and unreliable. Similarly, for Pneumonia, the computer's ability to find the disease (Average Precision) was very low, hovering around 0.05258, which is barely better than random guessing given how rare the positive cases were in the test set.

The authors are very clear about what this paper is not. They explicitly state that this is not a medical breakthrough. They did not prove that this computer can replace a doctor, nor did they show it works on patients in a real hospital. In fact, they list ten specific limitations, including the fact that the "labels" (the disease names) came from radiology reports written by humans, not from expert doctors double-checking every single image. They also admit they didn't test the computer on any other dataset, so we don't know if it would work elsewhere.

In the end, Cethraian-X is a "reality check" for the field of AI. It shows that while we can make our data perfectly clean and our experiments perfectly reproducible, the underlying computer model still struggles to give us trustworthy numbers for real-life medicine. The computer can sort the pictures, but it cannot yet tell a doctor, "I am 90% sure this patient is sick." For now, this work stands as a solid, clean benchmark for researchers to test their future ideas, but it draws a hard line in the sand: we are not ready to let these machines make clinical decisions yet.