🤖 AI

Auditing Frontier Vision-Language Models for Trustworthy Medical VQA: Grounding Failures, Format Collapse, and Domain Adaptation

This paper audits five frontier vision-language models on medical VQA, revealing that poor anatomical grounding and format-compliance failures in self-grounding pipelines severely undermine clinical trustworthiness, while supervised fine-tuning demonstrates that VQA-level performance gaps can be effectively addressed through domain adaptation.

Original authors: Xupeng Chen, Binbin Shi, Chenqian Le, Qifu Yin, Lang Lin, Haowei Ni, Ran Gong, Panfeng Li

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xupeng Chen, Binbin Shi, Chenqian Le, Qifu Yin, Lang Lin, Haowei Ni, Ran Gong, Panfeng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of highly intelligent, super-smart assistants to help doctors read medical X-rays and CT scans. These assistants are "Vision-Language Models" (VLMs)—AI systems that can see pictures and talk about them. The big question this paper asks is: Can we trust these assistants to point out exactly where a problem is (like a tumor) before they try to diagnose it?

The researchers treated these AI assistants like new employees and put them through a strict "audit" (a performance review) to see where they fail. Here is what they found, explained simply:

1. The "Pointing" Problem (Perception)

Imagine asking a student to point at a specific tiny freckle on a giant map of the world. Even the smartest students in the class got it wrong.

The Finding: When the AI models tried to draw a box around specific body parts (like a liver) or diseases (like lung cancer) on medical images, they were terrible at it.
The Analogy: It's like asking someone to find a specific grain of sand on a beach, and they just draw a giant box around the whole beach.
The Numbers: The best model in the test only managed to get the box right about 19% of the time. Even worse, they frequently confused "left" and "right" (e.g., pointing to the patient's left lung when the problem was on the right). In medicine, mixing up left and right is a dangerous mistake.

2. The "Two-Step" Disaster (Pipeline Collapse)

The researchers tested a specific workflow: First, ask the AI to find the problem and draw a box. Second, ask the AI to look only inside that box and give a diagnosis.

The Finding: This two-step process made the AI worse, not better.
The Analogy: Imagine asking a chef to first find a specific ingredient in a pantry, then cook a meal using only that ingredient. If the chef grabs the wrong ingredient (or drops the bag), the meal is ruined.
What Happened: Because the AI was bad at step one (finding the spot), step two (diagnosing) became a disaster. For some models, the accuracy dropped to nearly zero.
The "Format" Glitch: Some models got so confused by the complex instructions of the two-step process that they stopped answering correctly entirely. They failed to follow the rules of how to write down the box coordinates, causing the whole system to crash.

3. The "Magic Glasses" Test (Oracle Grounding)

To figure out if the AI was just bad at thinking or just bad at seeing, the researchers did a special test. They gave the AI the perfect box (drawn by a human expert) and asked it to diagnose the image based on that perfect box.

The Finding: When the AI was given the correct location, its diagnosis skills skyrocketed.
The Analogy: It's like giving the confused chef the exact right ingredient. Suddenly, they can cook a perfect meal.
The Conclusion: The AI's "brain" (reasoning) is actually okay. The problem is its "eyes" (perception). If we can fix the part that finds the spot, the diagnosis gets much better.

4. The "Training" Fix (Fine-Tuning)

Finally, the researchers tried to fix the AI by giving it extra homework. They took one of the models and trained it specifically on thousands of medical questions and answers.

The Finding: This "specialized training" made the AI much better at answering medical questions. It became one of the best at giving correct answers.
The Catch: While the AI got better at answering, the researchers didn't test if it got better at pointing (the perception problem). So, we know training helps the answers, but we don't know yet if it fixes the dangerous "left/right" confusion or the bad box-drawing.

Summary

The paper concludes that while these AI models are smart at talking and reasoning, they are currently unreliable at pointing.

The Bottleneck: You cannot trust the AI to guide the diagnosis if it can't accurately point to the problem first.
The Hope: If we can fix the "pointing" part (perhaps by using a specialized tool just for finding spots and then feeding that to the AI), the system could become very trustworthy.
The Reality Check: Right now, in a real hospital, relying on these models to find the problem and then diagnose it is risky because they keep getting the location wrong.

1. Problem Statement

Deploying Vision-Language Models (VLMs) in clinical settings requires more than just high-level reasoning capabilities; it demands precise visual grounding (the ability to localize anatomical structures and pathologies within an image). While frontier VLMs excel at reasoning, their performance on specialized medical inputs is poorly characterized regarding:

Perception: Can the model accurately locate specific regions (e.g., a tumor or organ) in a complex medical image?
Pipeline Integration: Does a "self-grounding" pipeline (where the model first locates a region and then answers a question based on that crop) degrade performance due to localization errors or format compliance failures?
Trustworthiness: Current benchmarks often rely on aggregate scores, failing to expose specific failure modes like "laterality confusion" (confusing left/right) which can lead to clinically dangerous errors.

2. Methodology

The authors conducted a comprehensive audit of five frontier VLMs (Gemini 2.5 Pro, GPT-5, o3, GLM-4.5V, and Qwen 2.5 VL) across three standard Medical VQA datasets: VQA-RAD, SLAKE, and PathVQA.

Experimental Setup

The evaluation followed a two-step pipeline (Figure 1 in the paper) to test three distinct configurations:

Direct VQA: The model answers the question using the full image in a single step.
Self-Grounding VQA: The model is prompted to first output a bounding box for the relevant region, then uses that cropped region to answer the question.
Oracle GT-Grounding VQA: The model is provided with ground-truth bounding boxes (from SLAKE annotations) to crop the image before answering. This serves as an upper bound to determine if the pipeline concept works if perception were perfect.

Evaluation Metrics

VQA Performance: Closed-ended accuracy and open-ended recall (using rule-based matching with synonym expansion and laterality checks).
Grounding Performance: Mean Intersection over Union (IoU), Dice coefficient, and Accuracy@0.5 (percentage of predictions with IoU $\ge$ 0.5).
Format Compliance: Analysis of parse failures (inability to extract the bounding box coordinates from the model's output).

Fine-Tuning Follow-up

To test if domain adaptation can bridge the gap, the authors performed Supervised Fine-Tuning (SFT) on Qwen 2.5 VL (7B) using combined training data from VQA-RAD, SLAKE, and PathVQA.

3. Key Contributions

Perception Audit: Construction of a dedicated visual grounding benchmark derived from SLAKE, revealing that even the best frontier models exhibit poor localization in medical imaging.
Pipeline-Integration Audit: Demonstration that "self-grounding" (localizing then answering) consistently degrades VQA accuracy across all models, driven by both inaccurate localization and severe format-compliance failures.
Oracle-Grounding Diagnostic: Proof that replacing predicted boxes with ground-truth annotations recovers and improves VQA performance, isolating the failure to the perception module rather than the decomposition strategy itself.
Forward-Looking Remedy: Demonstration that targeted SFT on a strong open-source model (Qwen 2.5 VL) significantly closes the VQA-level gap, achieving state-of-the-art recall on SLAKE, though the perception bottleneck remains unaddressed by SFT alone.

4. Key Results

A. Perception Gap (Grounding)

Poor Localization: All models performed poorly on the grounding task. The best model, Qwen 2.5 VL, achieved only 0.23 mean IoU and 19.1% Acc@0.5.
Systematic Failures:
- Laterality Confusion: Models frequently confused left and right in chest radiographs.
- Scale Mismatch: Models defaulted to large bounding boxes for small lesions (e.g., lung cancer), often covering the entire organ or image quadrant.
- Generalization Failure: Even models advertising built-in grounding (GLM-4.5V) underperformed compared to general reasoning models, indicating grounding features do not transfer well to fine-grained medical tasks.

B. Pipeline Degradation (Self-Grounding)

Catastrophic Collapse: Self-grounding degraded performance for every model.
- Gemini 2.5 Pro: Accuracy dropped from 39.9% to 0.0% (closed-ended). This was driven by a parse failure rate of 98.9% on the two-step prompt.
- GPT-5: Accuracy dropped from 67.6% to 19.3%, with a 70.3% parse failure rate.
Correlation: The severity of degradation correlated with grounding quality (IoU), though format compliance was a dominant factor for some models.

C. Oracle Recovery

When provided with Ground-Truth (GT) bounding boxes, all models improved their VQA performance compared to both Direct and Self-Grounding baselines.
Gemini 2.5 Pro recovered from near-zero to 44.3% accuracy, surpassing its direct VQA baseline. This confirms that the reasoning capability is intact, but the perception (localization) is the bottleneck.

D. Fine-Tuning Impact

Qwen 2.5 VL (SFT) achieved the highest reported SLAKE open-ended recall (85.5%) among comparable methods, outperforming LLaVA-Med and other baselines.
This proves the VQA-level gap is tractable via domain adaptation. However, since SFT was evaluated only on VQA metrics, it remains unclear if this adaptation fixed the underlying perception/grounding bottleneck.

5. Significance and Conclusion

Grounding as a Bottleneck: The study identifies visual grounding quality as the primary trustworthiness bottleneck for medical VQA in the SLAKE bounding-box setting. Current VLMs lack the spatial precision required to use grounding as a reliable intermediate step.
Pipeline Risks: Relying on "self-grounding" pipelines with current frontier models is dangerous, as it introduces a high risk of format collapse and localization errors that degrade overall performance.
Future Directions: The authors suggest:
1. Hybrid Architectures: Decoupling perception and reasoning by using specialized, domain-adapted grounding modules (e.g., fine-tuned Grounding DINO or SAM) to feed reliable spatial inputs to VLMs.
2. Granular Analysis: Stratifying failure analysis by organ type, lesion scale, and laterality.
3. Inference-Time Adaptation: Exploring test-time reinforcement learning to improve perception without additional training data.

In summary, while frontier VLMs possess strong reasoning capabilities, their inability to precisely localize medical structures renders "self-grounding" pipelines unreliable. Trustworthy medical AI requires either specialized perception modules or significant improvements in the spatial reasoning capabilities of VLMs themselves.