Technical Summary: Medical Context Distorts Decisions in Clinical Vision Language Models

Problem Statement

Vision-Language Models (VLMs) are increasingly proposed for clinical decision support, leveraging joint reasoning over medical images and free-text context (e.g., radiology reports, clinical history). However, their reliability in real-world scenarios remains poorly characterized. Real-world medical decisions occur under noisy, imperfect documentation streams where textual inputs may be incomplete, inconsistent, or biased.

The paper identifies three specific failure modes that threaten clinical reliability:

1. Modality Over-Reliance: An imbalance where models prioritize textual priors over visual evidence, even when visual data is available.
2. Spurious Reliance on Irrelevant History: Vulnerability to irrelevant or contradictory clinical history retrieved from Electronic Health Records (EHRs), which can systematically bias current decisions.
3. Prompt Sensitivity: Inconsistency in predictions when semantically equivalent prompt formulations are used, compromising the stability of clinical outputs.

The authors argue that current benchmark evaluations often fail to capture these failure modes, as they typically rely on clean, static test sets rather than simulating the dynamic, noisy context of clinical workflows.

Methodology

The study evaluates a diverse set of eight VLMs (including general-domain open models like LLaVA and Qwen, medically adapted models like MedGemma, and frontier proprietary models like GPT-5 and Gemini 3 Pro) on chest radiography tasks using a balanced subset of the MIMIC-CXR dataset. The task is framed as binary disease phenotyping (Abnormal vs. Normal), where "Abnormal" indicates the presence of one of five thoracic pathologies (Atelectasis, Cardiomegaly, Consolidation, Edema, Pleural Effusion).

The experimental design employs three specific protocols to isolate failure modes:

1. Selective Modality Shifting (SMS): To measure modality dominance, the authors create controlled conflicts between image and text.

   • Text Shift: The original X-ray is paired with a clinical history from a patient with the opposite diagnosis.
   • Image Shift: The original text is paired with an X-ray from the opposite class.
   • Unimodal Baselines: Performance is also measured using "Only Text" and "Only Image" inputs to determine the independent contribution of each modality.

2. Temporal Context Injection: To test robustness to irrelevant history, the models are fed the current chest X-ray and report alongside up to five clinically realistic but irrelevant prior reports (e.g., knee X-rays, brain MRIs) generated to be temporally distant and anatomically unrelated.

3. Semantic Prompt Sensitivity: The models are evaluated using four semantically equivalent prompt formulations (Standard QA, Role-Play, Formal Consult, Checklist) to assess decision stability under surface-level rephrasing.

Metrics:

• Accuracy: Measured against ground truth.
• Negative Flip Rate (NFR): A robustness metric quantifying the proportion of samples correctly classified in the baseline that become misclassified under perturbation (e.g., when text is shifted or irrelevant history is added).
• Fleiss' Kappa: Used to measure inter-prompt agreement across different prompt formulations.

Key Results

1. Modality Dominance: Text Overrides Images

The study found that VLM decisions are dominated by the text modality, even when visual evidence is available.

• Text Shift Impact: Introducing conflicting text caused a catastrophic drop in accuracy for all models. For frontier models (GPT-5, Gemini 3 Pro), accuracy collapsed from ~0.83 to ~0.17–0.18 (below random chance). Open models like LLaVA 1.5 and Qwen-2 VL dropped from ~0.80 to ~0.20.
• Image Shift Impact: Replacing the image while keeping the original text resulted in only minor accuracy changes (e.g., GPT-5 dropped from 0.83 to 0.82).
• Negative Flip Rate (NFR): Under text conflict, NFR was extremely high (0.31 to 0.66), indicating that contradictory text caused 30–66% of previously correct decisions to flip to errors. In contrast, Image Shift NFR remained low (0.02–0.15).
• Unimodal Performance: "Only Text" performance often matched or exceeded multimodal performance, while "Only Image" performance was significantly lower (often near chance level for open models).

2. Temporal Context Vulnerability

Irrelevant clinical history systematically degraded performance, though the degree varied by model family.

• General Open Models: Showed high sensitivity. LLaVA 1.5 accuracy dropped from 0.79 to 0.66 with five irrelevant reports. NFR increased significantly (up to 21% of correct predictions flipped).
• Medically Adapted Models: Showed mixed results; MedGemma remained relatively stable, while MedGemma 1.5 showed clear degradation.
• Frontier Models: Demonstrated the most robustness. GPT-5 and Gemini 3 Pro showed minimal accuracy drops and very low NFR (<3% even with five distractors), suggesting a better ability to filter long-context noise, though the failure mode (reliance on irrelevant text) was still present.

3. Prompt Sensitivity and Inconsistency

Model predictions were highly sensitive to prompt formulation, particularly for weaker models.

• Agreement (Fleiss' Kappa): Frontier models (Gemini 3 Pro, GPT-5) showed "Excellent" agreement (κ > 0.75) across prompt variations. In contrast, weaker open models like Janus-Pro and LLaVA 1.5 showed "Poor" agreement (κ < 0.40), with predictions frequently changing based on wording.
• Complexity Amplification: Prompt sensitivity was amplified when combined with long-context inputs (Multi-History setting), causing weaker models to degrade further.

Significance and Claims

The paper claims that current VLMs exhibit a systematic modality imbalance where textual signals override visual evidence, a phenomenon termed "modality collapse." This behavior persists across general-domain, medically adapted, and frontier models, suggesting that domain adaptation alone is insufficient to ensure genuine visual grounding.

The authors emphasize that:

• Benchmark Accuracy is Insufficient: High aggregate accuracy on static test sets does not guarantee reliability in clinical settings where inputs are noisy, conflicting, or retrieved from external sources.
• Context is a Safety Risk: The integration of retrieval-augmented generation (RAG) and agentic pipelines introduces significant risks, as models fail to reliably filter irrelevant or contradictory historical data.
• Prompt Fragility: Clinical reliability is compromised by prompt sensitivity, meaning that variations in documentation style across institutions could lead to divergent diagnostic outcomes for the same patient data.

The study concludes that before these models can be considered for clinical practice, explicit safeguards and stress-testing protocols are required. These protocols must specifically evaluate context dependence, modality contribution, and decision consistency under realistic perturbations, rather than relying solely on clean test-style questions.