Benchmarking the Domain Gap: Model Selection Instability Under Domain Shift in Video Capsule Endoscopy
This paper demonstrates that model selection for video capsule endoscopy is highly unstable under domain shift, as in-domain performance rankings fail to consistently predict cross-target results across different datasets, necessitating a shift toward evaluating cross-target ranking stability rather than peak single-dataset performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot to recognize different types of fruit. You show it thousands of pictures of apples, oranges, and bananas taken in your own kitchen, under your specific lights, with your specific camera. The robot gets really good at this! It becomes a fruit-whiz in your kitchen. But then, you take that same robot to a friend's house. Their kitchen has different lighting, their camera is a different model, and maybe they even label "ripe" differently than you do. Suddenly, the robot that was the champion in your kitchen might stumble over a banana that looks slightly different, while a robot you thought was mediocre suddenly shines. This is the heart of a problem called "domain shift" in computer science. It's the gap between how well a computer model works in the lab where it was trained versus how well it works in the messy, real world where it's actually used. In the medical world, this matters a lot. If a doctor relies on an AI to spot problems inside a patient's body, that AI needs to be smart not just for one specific hospital, but for any hospital, with any camera, and any patient.
This paper dives deep into that exact problem, but instead of fruit, the robots are looking at video capsules—tiny, pill-sized cameras that patients swallow to take pictures of their insides. The researchers wanted to know: If we pick the "best" AI model based on how it performs on one specific dataset (a collection of images), will that same model still be the best when we test it on a completely different set of images from a different hospital? They didn't just guess; they ran a massive experiment. They took 11 different "brain" architectures (the core engines that power the AI) and trained them on a standard set of capsule videos. Then, they tested these same trained brains on two other totally different video sets.
Here is the twist they found: The "best" model depends entirely on who you ask. When they ranked the models based on the first dataset, the top performer was a model called Swin-B. It was the clear winner. But when they took that same Swin-B model and tested it on a second dataset, it dropped to fifth place. Meanwhile, a model called ConvNeXt-Base, which was only seventh in the first test, became the number one champion in the second test. It's as if you held a race in the morning, and the winner was a sprinter, but when you held the race in the afternoon on a different track, a marathon runner won. The paper shows that there is no single "magic bullet" model that wins everywhere. If you pick a model just because it has the highest score on one specific test, you might be picking the wrong one for the next hospital you visit.
The researchers also tried a second experiment. They trained a new batch of models on the second dataset and tested them on a third. The result was the same: the rankings kept shifting. A model that was great at the second test was mediocre at the third. This suggests that the "leaderboard" you see in a research paper is often just a snapshot of how well a model fits that specific puzzle, not a guarantee that it will solve the next one. The authors conclude that we need to stop looking for the single highest score on one dataset. Instead, we should look for models that stay consistent across many different tests. If a model is truly robust, it shouldn't care if the lights change or the camera changes; it should keep its rank steady. Until we find those steady performers, picking a medical AI based on a single test score is like picking a travel guide based only on how well they know your own neighborhood—they might get lost the moment you leave town.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.