MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts
The paper introduces MissingBench-Verified, a benchmark demonstrating that leading Vision-Language Models consistently fail to detect missing object parts due to inherent biases and training data scarcity, a limitation that persists despite the use of external tools, extended reasoning, or fine-tuning, thereby revealing a fundamental architectural flaw in current VLMs.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a robot to see the world. You show it a million pictures of cats, dogs, and airplanes, and it learns the rules: "Cats have tails," "Airplanes have wings," and "iPhones have home buttons." This is the world of Vision-Language Models (VLMs). These are super-smart computer programs that can look at a picture and describe it in words, or answer questions about what they see. They are like a librarian who has read every book in the universe and can instantly tell you the plot of a story just by looking at the cover.
But here is the tricky part: sometimes, this librarian gets too confident in what they know. If you show them a picture of a cat that has lost its tail, their brain might scream, "That's impossible! All cats have tails!" and they might insist the tail is there, even though their eyes clearly see it's gone. This is called a hallucination. It's not that the robot is lying; it's that its internal knowledge is so strong it overrides what its eyes are actually seeing. Scientists care about this because if we want these robots to help us inspect bridges, check medical scans, or monitor factories, they need to be able to say, "Hey, something is missing," instead of making things up.
The Great "Missing Part" Mystery
Enter MissingBench-Verified, a new experiment designed to test exactly this stubbornness. The researchers behind this study wanted to know: If we take a picture of an airplane and surgically remove its engines, will the robot admit the engines are gone, or will it hallucinate that they are still there?
To find out, the team created a special set of 118 images. They took real photos of everyday objects—like a phone with its home button erased, or a plane without its tail—and showed them to ten of the smartest AI models on the market. They asked a simple question: "Is this part visible?" The models had to choose between "Clearly visible," "Hard to see," or "Not visible."
The results were a bit of a shock. Even the most advanced AI models, the ones that usually ace every test, failed miserably. When the engines were gone, the models insisted they were still there. In fact, nearly half of the models got it wrong more than 50% of the time. The worst performer, Claude Sonnet 4.6, only correctly identified that the part was missing 44.1% of the time. It was as if the robots were wearing blinders made of their own memories, refusing to believe their own eyes.
The "Magic Tools" That Didn't Work
The researchers didn't just stop at asking the robots to look harder. They tried to help them out with a whole toolbox of tricks, hoping to snap them out of their daydreams. They tried three main strategies:
- The "Detective's Report" (External Tools): They gave the models a fake report from a super-accurate object detector that said, "I looked, and I definitely did not find the engine." They hoped the model would trust the report. Instead, most models ignored the report and stuck to their guns. Even when the "perfect detector" said the object was gone, the models still hallucinated it was there.
- The "Zoom Lens" (Image Processing): They let the models use tools to crop the image, brighten it, or sharpen it, hoping that a closer look would reveal the truth. One model even used a cropping tool to zoom in on a missing airplane tail, saw the empty space, and then still claimed the tail was there, blaming the missing part on a "cropping error" rather than admitting it was gone.
- The "Deep Think" (More Time): They forced the models to spend more time thinking and reasoning before answering, hoping that more brainpower would fix the mistake. Surprisingly, making them think longer didn't help; in some cases, it just made them more confident in their wrong answers.
The Verdict: It's Not a Bug, It's a Feature (That's Broken)
The paper suggests that this isn't just a simple mistake the models can be "taught" out of with a quick prompt or a little extra thinking. The problem is deep. The models have learned a "prior"—a strong belief that airplanes must have engines to fly. When the visual evidence (the empty space) clashes with this deep-seated belief, the belief wins. The models are essentially saying, "I know what an airplane looks like, and this picture is wrong, so I will describe the airplane as I expect it to be, not as it is."
The researchers tried to fix this by giving the models access to better tools and more time, but the paper concludes that these current methods provide negligible improvement. The failure rates stayed stubbornly high, with most models still unable to correctly identify the missing parts even with help.
This study doesn't claim to have solved the problem. Instead, it sounds an alarm. It shows that for tasks like checking if a machine part is missing or if a safety feature is intact, our current best AI models are dangerously unreliable. They are so convinced of their internal knowledge that they can't override it, even when the evidence is right in front of their "eyes." The paper suggests that fixing this won't happen with better prompts or more thinking time; it will likely require a complete overhaul of how these models are built or trained, so they can learn to trust what they see over what they think they know.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.