💻 computer science

Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models

This paper demonstrates that consistency under paraphrase is a flawed proxy for reliability in medical vision-language models because models can achieve perfect consistency by ignoring images and relying on text patterns, a dangerous failure mode that standard confidence metrics miss but can be exposed by comparing predictions against a text-only baseline.

Original authors: Binesh Sadanandan, Vahid Behzadan

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Binesh Sadanandan, Vahid Behzadan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a new assistant to help you diagnose medical problems based on X-ray images. You want this assistant to be reliable. You test them by asking the same question in different ways:

"Is there fluid in the lungs?"
"Can you see fluid in the lungs?"
"Does the patient have a pleural effusion?"

If your assistant gives the exact same answer every time, no matter how you phrase the question, you think, "Great! They are consistent. They are reliable."

This paper argues that this feeling of safety is a dangerous illusion.

Here is the simple breakdown of what the researchers found, using some everyday analogies.

1. The "Stubborn Chef" Analogy

Imagine a chef who claims to cook based on the ingredients you give them.

The Test: You ask, "Is this soup salty?"
The Problem: The chef always answers "Yes," no matter what soup you show them. Even if you show them a bowl of plain water, they say "Yes." Even if you ask, "Is this soup salty?" in French, German, or a different accent, they still say "Yes."

The Trap:

Consistency: The chef is 100% consistent. They never change their answer based on how you ask.
Reliability: You might think, "Wow, they are so confident and stable!"
The Reality: The chef isn't looking at the soup at all. They are just guessing "Yes" because they know that, statistically, most soups in this restaurant are salty. They are ignoring the actual image (the soup) and relying on a text shortcut.

In the world of AI, this is called the "Dangerous" quadrant. The AI is consistent, but it's not actually looking at the X-ray.

2. The Four Types of AI Assistants

The researchers created a simple chart to sort AI assistants into four groups based on two questions:

Are they consistent? (Do they give the same answer if you rephrase the question?)
Are they looking at the picture? (If you take the picture away, do they change their answer?)

Here are the four types:

🌟 The Ideal Assistant (The Doctor):
- Behavior: They give the same answer no matter how you ask, AND they change their answer if you swap the X-ray for a blank piece of paper.
- Verdict: Safe. They are looking at the image and thinking clearly.
🥚 The Fragile Assistant (The Nervous Intern):
- Behavior: They look at the picture, but if you change the wording of the question slightly, they get confused and give a different answer.
- Verdict: Risky, but honest. They are trying to use the image, but they are easily confused by language.
☠️ The Dangerous Assistant (The Stubborn Chef):
- Behavior: They give the exact same answer no matter how you ask, BUT if you take the picture away, they give the exact same answer.
- Verdict: DEADLY. They look super reliable because they are consistent, but they are actually ignoring the patient's X-ray entirely. They are just guessing based on patterns in the text.
- The Twist: These assistants often get high scores on accuracy tests! If the disease is common, guessing "Yes" all the time gets you a high score, even if you never looked at the X-ray.
🗑️ The Worst Assistant (The Broken Robot):
- Behavior: They change their answer when you rephrase the question, AND they ignore the picture.
- Verdict: Useless. They are inconsistent and not looking at the image.

3. The Big Surprise: "Fixing" Made It Worse

The researchers tried to "fix" the AI assistants by training them to be more consistent (so they wouldn't get confused by different wordings).

The Result?
The training worked! The assistants became very consistent. But, it pushed them from the "Fragile" group into the "Dangerous" group.

They stopped getting confused by wording.
Instead, they started ignoring the X-rays entirely and just guessing the most common answer.

It's like teaching a student to pass a test by memorizing the answer key instead of learning the subject. They get a perfect score (high consistency), but they can't actually do the work (they aren't looking at the image).

4. Why Standard Tests Failed

Usually, when we test AI, we check:

Accuracy: Did they get the right answer? (The "Dangerous" AI often gets this right by luck).
Confidence: How sure are they? (The "Dangerous" AI is very confident because they have a simple rule to follow).
Consistency: Do they answer the same way if you rephrase? (The "Dangerous" AI is perfect here).

The paper says: If you only check these three things, you will hire the "Dangerous" assistant and fire the "Ideal" one. The "Dangerous" AI looks perfect on paper but is actually blind.

5. The Simple Solution: The "No-Picture" Test

The researchers propose a very simple, cheap test to catch these dangerous AI models before we use them in hospitals.

The Test:
Run the AI's question through the system twice:

Once with the X-ray image.
Once without the image (just the text question).

If the answers are different: Great! The AI is actually looking at the picture.
If the answers are the same: STOP. The AI is ignoring the picture. It doesn't matter if it's consistent or confident; it's not doing its job.

The Takeaway

In medical AI, consistency is not enough. An AI can be perfectly consistent and perfectly confident while being completely wrong because it isn't looking at the patient's scan.

To keep patients safe, we need to stop trusting AI just because it answers the same way every time. We need to make sure it's actually looking at the picture, not just guessing based on the words.

1. Problem Statement

The paper addresses a critical flaw in the current evaluation metrics for Medical Vision-Language Models (VLMs) intended for clinical deployment.

The Assumption: Consistency under paraphrase (where semantically equivalent prompts yield identical predictions) is widely used as a proxy for model reliability. A low "flip rate" (the fraction of samples where paraphrasing changes the prediction) is interpreted as a sign of a stable, trustworthy model.
The Blind Spot: The authors argue that high consistency does not guarantee safety. A model can achieve perfect consistency (0% flip rate) by relying entirely on text patterns (shortcuts) rather than analyzing the input medical image.
The Risk: Such models appear reliable because they give consistent answers, but they are "dangerous" because their predictions are invariant to the removal of the image. In a clinical setting, this leads to missed diagnoses or inappropriate treatments if the model ignores the actual radiograph.

2. Methodology

The authors propose a novel framework to evaluate VLMs at the per-sample level rather than relying on aggregate metrics.

A. Four-Quadrant Safety Taxonomy

The core contribution is a classification system based on two binary properties for each sample:

Consistency: Does the model give the same answer across $K$ semantically equivalent paraphrases?
Image Reliance: Does the model's prediction change when the image is removed (text-only baseline)?

This creates four quadrants:

Ideal: Consistent AND Image-reliant. (Stable predictions grounded in visual evidence).
Fragile: Inconsistent BUT Image-reliant. (Genuine sensitivity to phrasing, but uses the image).
Dangerous: Consistent BUT Not Image-reliant. (The "False Reliability" trap: stable predictions that ignore the image).
Worst: Inconsistent AND Not Image-reliant. (Unstable and ungrounded).

B. Experimental Setup

Models: Five configurations across two model families:
- MedGemma-4B-IT: Base, Targeted LoRA (layers 15–19), and Full LoRA (all layers).
- LLaVA-Rad: Base and LoRA-fine-tuned versions.
Datasets:
- MIMIC-CXR: Balanced binary questions (approx. 50% "yes").
- PadChest: Imbalanced binary questions (81% "yes"), serving as an Out-of-Distribution (OOD) test.
Text-Only Baseline: For every sample, the model is run with the image omitted entirely to determine if the prediction relies on visual input.
Metrics: Flip Rate, Dangerous Fraction, Per-Quadrant Accuracy, and Predictive Entropy.

3. Key Contributions

Safety Taxonomy: Introduced the four-quadrant framework to jointly evaluate consistency and image grounding, explicitly identifying the "Dangerous" quadrant.
The Consistency-Safety Paradox: Demonstrated a strong negative correlation ( $r = -0.89$ ) between flip rate and the Dangerous fraction. Models optimized for consistency (via LoRA fine-tuning) systematically shift samples from "Fragile" to "Dangerous."
Invisibility of Danger: Showed that "Dangerous" samples exhibit high accuracy (up to 99.6% on imbalanced datasets) and low entropy (high confidence). This makes them undetectable by standard uncertainty-based screening or accuracy checks.
Practical Deployment Recommendation: Proposed a low-cost mitigation strategy: pairing consistency checks with a text-only baseline (one additional forward pass per sample) to expose false reliability.

4. Key Results

Prevalence of Danger: On the PadChest dataset, the LLaVA-Rad Base model placed 98.5% of its samples in the "Dangerous" quadrant while maintaining a near-perfect 1.5% flip rate.
LoRA Effect: Fine-tuning for consistency (Targeted and Full LoRA) dramatically reduced flip rates but increased the Dangerous fraction. For example, on PadChest, Full LoRA MedGemma had a 4.1% flip rate but a 60.7% Dangerous fraction.
The Accuracy Trap: In the "Dangerous" quadrant, accuracy was often significantly higher than in the "Ideal" quadrant (e.g., 99.6% vs. 26.1% for Targeted LoRA on PadChest). This occurs because the models learn to predict the majority class ("yes") based on text statistics, which is both consistent and accurate on imbalanced data, yet clinically useless.
Entropy Failure: Standard uncertainty metrics failed to flag Dangerous samples. Dangerous samples had lower entropy (higher confidence) than Fragile samples, meaning uncertainty-based gates would filter out the "good" (image-reliant) samples while passing the "dangerous" ones.
Correlation: A strong negative correlation was observed between Flip Rate and Dangerous Fraction across all 10 model-dataset combinations, confirming that optimizing for consistency often degrades image grounding.

5. Significance and Implications

Redefining Reliability: The paper challenges the medical AI community to stop treating "consistency" as an unconditional positive. A model can be perfectly consistent and dangerously wrong if it ignores the visual modality.
Deployment Safety: Current evaluation pipelines that rely solely on accuracy, flip rate, and confidence scores are likely to select the most dangerous models for clinical use.
Actionable Solution: The authors provide a simple, computationally cheap fix: always run a text-only baseline. If a model gives the same answer with or without the image, it should be flagged as unsafe, regardless of its accuracy or consistency score.
Training Guidance: Future consistency training objectives should be constrained to ensure predictions differ from text-only baselines, forcing the model to achieve consistency through improved visual reasoning rather than text shortcuts.

In conclusion, the paper reveals that consistency is necessary but not sufficient for safety in medical VLMs. Without checking for image reliance, highly consistent models may be hallucinating based on text statistics, posing a severe risk to patient care.