Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact

Imagine a world where scientists build digital detectives to read X-rays and tell doctors what's wrong. These detectives are a special kind of computer program called a "Vision-Language Model." Think of them as super-smart robots that can look at a picture of a chest and then write a report, just like a human doctor might. To see if these robots are any good, researchers create "benchmarks." A benchmark is like a standardized test: you show the robot 30 X-rays, ask it specific questions, and then check its answers against a "gold standard" to see how many it got right. Everyone assumes that if the test is written down in a paper, the computer actually did exactly what the paper says. But what if the computer took a shortcut? What if the test questions were swapped out, or the X-rays were flipped upside down, and nobody noticed until years later? This is the scary gap between what we say we did and what the computer actually did.

This paper is a forensic detective story about one of those digital tests. The author, Mateusz Kozłowski, decided to play the role of a digital crime scene investigator. He didn't run the test again; instead, he looked at the "crime scene" (the computer files and code) left behind by a previous study. He wanted to see if the story told in the final report matched the reality of the files on the hard drive. He found that the story and the reality were very different. The study claimed to compare two different ways of asking the robot questions, but by tracing the path of how the instructions were imported and called within the code, he discovered the robot was actually being asked the exact same question twice. It claimed to use 30 different patients, but the files showed only 28. It claimed to use perfect X-ray images, but four of them were flipped the wrong way, like looking at a mirror image of a chest. Because of these mix-ups, the final scores and rankings in the original report were wrong. The author concludes that the original test results cannot be trusted to tell us how well the robots actually work, and he proposes a new set of "machine rules" to make sure future tests don't have these hidden glitches.

The Digital Detective's Report

The Setup: A Test That Wasn't a Test
Imagine you are a teacher giving a math quiz. You tell the class, "I'm going to give you two different versions of the quiz, Version A and Version B, to see which one is harder." You hand out the papers, collect them, and grade them. But when you look at your answer key later, you realize you accidentally gave every single student the exact same Version C paper. You thought you were testing the difference between A and B, but you were just testing how students did on C twice.

That is exactly what happened in this study. The researchers wanted to see if asking a medical AI two different questions (Prompt A and Prompt B) would change the answers. They ran the test on 30 chest X-rays. However, when the author dug into the computer code, he found a "glitch" in the delivery system. By tracing the path of how the instructions were imported and called, he found the code was supposed to switch between Prompt A and Prompt B, but because of a programming error, both groups of questions were actually sent as Prompt C. So, the 60 times the computer was asked to answer, it was actually answering the same question 60 times. The study claimed to find a difference between A and B, but since A and B never actually happened, that difference was a ghost.

The X-Ray Mix-Up
Next, imagine you are looking at a black-and-white photo. If the photo is printed "negative" (where black is white and white is black), you might think a dark spot is actually a light spot. In the world of medical imaging, there are two types of black-and-white settings: one where dark means "bone" and another where dark means "air." The study claimed to use standard X-rays. But the author found that four of the images were in the "air" setting, and the computer code forgot to flip them to the "bone" setting before showing them to the AI.

It's like serving a customer a burger with the bun on the bottom and the patty on top, then claiming you served a "standard burger." The AI saw a flipped image, not the real one. This meant the AI was making decisions based on a distorted reality, not the actual X-ray.

The Patient Count and the Missing Files
The study said it used 30 different patients. But when the author counted the files, he found that two patients had actually shown up twice, meaning there were only 28 unique people. It's like a class roll call where two students signed in twice, making it look like there were 30 kids when there were only 28. This matters because if the same person is in the test twice, it skews the results.

Furthermore, the study planned for 300 answers (30 X-rays × 10 different AI models). But when the author checked the files, three of them were missing or empty. Two files were completely blank (zero bytes), and one was just gone. The original report pretended these were successful answers, but they were actually failures. It's like a teacher grading a quiz and giving a student a score even though the student left the paper blank.

The "Magic" Math
Because of these errors, the math in the original paper was wrong. The authors had calculated a score called the "Cochran statistic" to see if the models were different. The original paper said the score was 154.73. When the author fixed the math by using the correct, complete data (369 blocks of information instead of the messy, incomplete version), the score jumped to 182.29.

The original paper also claimed that 26 out of 45 comparisons showed a "significant" difference. When the author recalculated the math using the correct, complete data, he found that 27 out of 45 comparisons were significant when using the unadjusted method, and 20 out of 45 were significant when using the more strict Holm-adjusted method. However, the original result of 28 out of 45 was actually reproduced when the author performed a separate analysis where each comparison used its own specific denominator. While these numbers look similar, the meaning is totally different because the data they were based on was flawed. The original claim that "Model A is better than Model B" was based on a test that never actually happened.

The "Stale" Release
Here is the most frustrating part. When the researchers realized their mistake and fixed the numbers in their written paper, they forgot to update the digital files they shared with the world. They uploaded a "revised" paper, but the zip file they attached still had the old, wrong numbers and the old, wrong names for the models. It's like a chef correcting a recipe in a cookbook but forgetting to change the ingredients list on the website. Anyone who downloaded the file got the old, broken version.

The Solution: A New Rulebook
The author doesn't just point out the mistakes; he builds a new rulebook to stop them from happening again. He calls this the "Benchmark Contract." Imagine a contract that forces a computer to prove it did the right thing before it's allowed to publish a score.

This contract has eight rules:

1. Cohort Contract: Prove you have the right number of unique patients.
2. Pixel Contract: Prove the X-ray images weren't flipped or broken.
3. Prompt Contract: Prove the AI actually got the specific question you asked, not a different one.
4. Model Contract: Prove you used the exact version of the AI you said you did.
5. Output Contract: Prove the answer file isn't empty.
6. Annotation Contract: Prove the labels (like "sick" or "healthy") were assigned correctly.
7. Analysis Contract: Prove the math was done on the right groups of data.
8. Release Contract: Prove the files you share match the paper you wrote.

The author built a set of 36 automated tests (like a robot inspector) that check these rules. If a test fails, the system stops and says, "Nope, you can't publish this yet."

The Bottom Line
The big takeaway is that the original study's claims about how well these AI models work are withdrawn. The author is very clear: the original results are invalid. You cannot trust the rankings or the "better" model because the test was broken. The only thing that can be saved is the story of how it broke.

The author suggests that in the future, we need to use these machine-checkable contracts. We can't just trust that the computer did what we told it to do; we need the computer to prove it. This paper doesn't give us a new, better AI model; instead, it gives us a better way to make sure the tests we use to judge AI are actually fair and real. It's a reminder that in the digital age, if you don't check the code, you might be grading a test that never happened.