Vision-language models for chest radiography do not always need the image

Problem Statement
Medical vision-language models (VLMs) have demonstrated strong accuracy on chest radiograph benchmarks, leading to an implicit assumption that these systems integrate visual evidence with textual reasoning to produce clinically valid answers. However, accuracy metrics alone cannot distinguish between a model that genuinely "reads" the radiograph and one that infers answers from linguistic priors, epidemiological co-occurrences, or finding-name statistics present in its training corpus. This "shortcut learning" is a known failure mode in unimodal classifiers, but it poses a sharper risk for VLMs, which can generate credible medical responses using language alone. Existing evaluation methods, such as post-hoc saliency maps or adversarial degradation of performance, fail to causally isolate whether a specific correct answer depends on the image input for standard clinical questions.

Methodology
The authors introduce a causal audit framework designed to intervene directly on the image input to test the causal dependency of the model's output. The study evaluates nine systems, including general-purpose multimodal foundation models, specialist medical VLMs, frontier closed-source systems, text-only large language models (LLMs), and a vision-only linear probe.

The audit utilizes a probe set of 2,575 binary (yes/no) finding-presence questions derived from MS-CXR (radiologist-marked regions), MIMIC-CXR (global labels), and ReXErr (report errors). Each model is subjected to four image conditions:

1. Original: The unmodified radiograph.
2. Swap: A different patient's radiograph carrying the exact same finding label.
3. Target Mask: The original image with the radiologist-marked region for the queried finding occluded.
4. Irrelevant Mask: The original image with a same-sized region occluded away from the finding.

From these interventions, three behavioral metrics are derived:

• Causal Grounding Rate (CGR): The fraction of correct answers that flip when the target region is masked.
• Unrelated-Image Answer Rate (UAR): The fraction of correct answers preserved when the image is swapped for a different patient with the same label.
• Irrelevant-Mask Stability (IS): The fraction of answers preserved when an irrelevant region is occluded.

These metrics are used to categorize systems into three behavioral groups: Uses Image (high CGR, low UAR, high IS), Ignores Image (CGR=0, UAR=100, IS=100), and Unstable (low IS, indicating sensitivity to non-diagnostic edits). The framework is validated across a second dataset (CheXpert), varying image resolutions, and prompt phrasings, and is benchmarked against independent grading by board-certified radiologists.

Key Results

• Decoupling of Accuracy and Image Use: Benchmark accuracy does not correlate with whether a model uses the image. A text-only model (MedGemma-27B-text) achieved 60.1% accuracy, statistically indistinguishable from the best multimodal system (Gemma-4-26B at 66.2%) and significantly outperforming two specialist medical multimodal models. A 119-billion-parameter multimodal model was statistically indistinguishable from a 7-billion text-only baseline.
• Behavioral Categorization:
  • Ignores Image: Three systems (LLaVA-Med-7B, MedGemma-27B-text, DeepSeek-R1-7B) showed zero causal grounding; their answers remained unchanged regardless of image edits.
  • Unstable: One large model (Mistral-Small-4-119B) exhibited instability, changing answers under irrelevant occlusions as frequently as under target masking.
  • Uses Image (Selectively): Five systems used the image, but grounding was partial. The causal grounding rate for these models ranged from 6.4% to 33.5%. Image use was highly finding-specific (e.g., high for pneumonia and edema, near zero for atelectasis and lung opacity) and view-dependent (significantly lower on anteroposterior/portable views compared to posteroanterior views).
• Human Comparison: Against board-certified radiologists, the text-only model was statistically indistinguishable in accuracy but grounded at zero. Conversely, models that used the image grounded at rates comparable to radiologists.
• Confidence Calibration: Reported confidence only reliably flagged ungrounded answers in models that actually used the image. Text-only models and unstable models showed poor calibration, often reporting high confidence on incorrect or ungrounded answers.

Significance and Claims
The paper argues that reported accuracy is an insufficient proxy for clinical reliability in medical VLMs because it fails to verify the provenance of the answer. The authors claim that:

1. Accuracy is not evidence of radiology: High benchmark scores can be achieved by models that ignore the image entirely, relying instead on dataset-specific priors.
2. Grounding is a measurable property: Whether a model uses the image is a distinct, measurable behavioral characteristic that varies by model, finding, and view, and is not guaranteed by model size or specialization.
3. Interventional Audits are necessary: The authors propose that grounding audits, specifically causal interventions on the input, should accompany accuracy benchmarks and gate clinical deployment claims.
4. Current deployment risks: The findings suggest that relying solely on accuracy and confidence scores is unsafe, as confidence does not distinguish between grounded and ungrounded answers in models that ignore the image.

The study concludes that while medical VLMs are not necessarily unfit for clinical use, the field must shift from reporting accuracy alone to measuring and reporting causal grounding to ensure that correct answers are derived from the radiograph rather than recited from priors.