Grounding Agentic VLMs with Dedicated Segmentation for Fine-Grained Vehicle Damage Assessment

Imagine you are trying to teach a super-smart robot how to look at a picture and tell a story about it. This field of science is called "Vision-Language," where computers learn to see images and speak about them. For a long time, these robots were great at describing the big picture—like saying, "That's a red car." But they often struggled with the tiny details, like pointing exactly where a scratch is on the bumper. It's a bit like having a friend who can tell you a story about a movie but can't point out which character is wearing a blue hat. This matters because in the real world, if a robot is supposed to check a car for damage to help with insurance, it can't just guess; it needs to know exactly where the problem is, or it might invent damage that isn't there.

The researchers in this paper tackled this problem by building a new system called TinyDamage to help a robot named Qwen-VL become a better detective. They found that while the robot was very good at knowing what a scratch or crack looked like (getting 87.3% right on naming them), it was terrible at pointing to them on the photo. When asked to find the damage, the robot would often "hallucinate," meaning it would see scratches in shiny reflections or miss long, thin cracks entirely. It was like a detective who knows what a fingerprint looks like but keeps pointing at the wrong person in the lineup.

To fix this, the team didn't try to force the robot to get better at pointing. Instead, they gave it a partner. They built a specialized "spotter" (the TinyDamage model) whose only job is to find the tiny, tricky damage and draw a mask around it. Then, they let the main robot use that mask as a guide to write its report. Think of it like a tour guide who knows all the history but gets lost easily, paired with a local guide who knows exactly where every street is. The local guide points the way, and the tour guide tells the story.

The results were surprising and very specific. The team discovered that the way they taught the "spotter" to learn was more important than the design of the spotter itself. They found that using a common teaching method called "focal loss" actually made the spotter completely blind to tiny scratches, causing it to miss them 100% of the time. However, by switching to a different method called "supervised contrastive learning," the spotter became much better at separating the damage from the background.

When they tested the whole team together, the difference was huge. If they asked the robot to write a report based only on the picture, it made up fake damage 78% of the time. If they asked it based only on a text description, it made up damage 92% of the time. But when they let the robot use the "spotter's" map to guide its writing, the rate of fake damage dropped to just 31%. The paper suggests that for robots to be reliable in the real world, we shouldn't just ask them to "try harder" at seeing small things; instead, we should give them a dedicated tool to find the small things, and then let the robot use that tool to tell the truth.