Technical Summary: Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?

Problem Statement

The safe deployment of Vision-Language Models (VLMs) in clinical workflows requires reliable Uncertainty Estimation (UE). Clinicians need a signal indicating when to trust a model's prediction or escalate the case to human review. However, current evaluations of UE in clinical settings are limited. Most prior benchmarks focus on text-only tasks or non-clinical vision-language benchmarks, often relying on narrow baselines like verbalized confidence or token-level probabilities. Furthermore, standard evaluations assume well-formed inputs where the correct answer is guaranteed to be among the options. Clinical practice frequently violates these assumptions, encountering ambiguous questions, unanswerable scenarios, or cases where the correct diagnosis is not listed. The central question addressed is whether current UE methods provide a reliable safety signal in these realistic, potentially failure-prone clinical scenarios.

Methodology

The authors conducted a systematic evaluation of post-hoc UE methods across 12 VLMs (including 2 biomedical and 10 general-purpose models, ranging from 4B to 72B parameters) using the GMAI-MMBench dataset. The evaluation suite comprised 1,680 samples (210 per modality) across eight imaging modalities: CT, MRI, Endoscopy, Histopathology, Fundus Photography, X-ray, Microscopy, and Dermoscopy.

Uncertainty Estimation Methods:
Eight post-hoc UE methods spanning three families were benchmarked:

1. Logit-based: Average Negative Log-Likelihood (ANLL), Max Negative Log-Likelihood (Max NLL), and a new method, Label Negative Log-Likelihood (Label NLL). These require a single inference pass.
2. Consistency-based: Sample Consistency (SC) and Semantic Entropy (SE). These measure agreement across multiple stochastic decoding runs.
3. Embedding-based: EigenEmbed (EE) and Radial Dispersion Score (RDS). These generate multiple outputs, embed them via an external biomedical encoder, and compute geometric features.
4. Hybrid: Probability Only (PRO), requiring both logits and sampling.

Evaluation Metrics:
Performance was measured along two axes:

• Discrimination: The ability to separate correct from incorrect predictions, measured by AUROC.
• Calibration: The alignment between confidence and accuracy, measured by Expected Calibration Error (ECE) and Brier Score.

Stress Testing (NOTA Perturbations):
To probe robustness, the authors introduced "None of the Above" (NOTA) perturbations:

• +nota: Adds a "None of the options is correct" distractor while keeping the correct answer present.
• -correct+nota: Replaces the correct answer with the NOTA option, forcing the model to select an incorrect option or the NOTA option.

This design tested two hypotheses:

• H1 (Robustness): Does uncertainty rise proportionally to accuracy drops under perturbation?
• H2 (Predictive Signal): Does the baseline uncertainty of the original (unperturbed) input predict whether a prediction will flip under perturbation?

Key Findings

1. UE Quality Tracks Model Accuracy, Not Method Superiority

The study found that UE quality is not an intrinsic property of the UE method but rather tracks the underlying model's accuracy on the specific input.

• Discrimination: Label NLL consistently outperformed other methods across most models, often by a significant margin.
• Calibration: No single method dominated calibration; performance varied by model family (e.g., PRO for Qwen, RDS for Molmo).
• Contextual Degradation: Crucially, UE performance (both discrimination and calibration) degraded precisely in the clinical contexts where model accuracy was lowest (e.g., Endoscopy and Histopathology). In these high-risk domains where a safety signal is most needed, UE methods provided the least informative signals.

2. Systematic Miscalibration Under NOTA Stress

Under the -correct+nota perturbation (where the correct answer is removed), model accuracy collapsed significantly (drops of 12–32 points). However, uncertainty estimates failed to rise proportionally to match this accuracy drop.

• Models remained confidently committed to wrong answers even when the correct option was unavailable.
• This contradicts the assumption that UE serves as a reliable threshold for abstention or escalation in these failure regimes.

3. UE as a Diagnostic Tool for Fragility

Despite failing as a runtime safety signal under perturbation, the study identified a complementary utility. Baseline uncertainty estimated on the original, unperturbed input reliably predicted which predictions would collapse under NOTA perturbations.

• Samples that flipped their answers (from correct to incorrect, or vice versa) under perturbation exhibited significantly higher baseline uncertainty than stable samples.
• This suggests that while UE may not signal current error in a specific instance, it carries diagnostic information about the structural fragility of a prediction. High-uncertainty correct answers are the ones most likely to break under minor input variations.

Significance and Claims

The paper positions UE not as a uniform safety net, but as a context-dependent diagnostic tool. The authors argue that:

1. UE is not a universal safety net: It fails to flag errors precisely in the domains (low-accuracy modalities) and scenarios (missing correct options) where clinical safety is most critical.
2. Evaluation protocols must evolve: Standard calibration metrics are insufficient. Evaluation should include perturbation-based stress tests (like NOTA) to reveal systematic failure modes and decoupling of uncertainty from accuracy.
3. Diagnostic Value: The value of UE extends beyond per-prediction confidence scores. It can identify structurally fragile predictions that are prone to failure under perturbation, offering a pathway to triage which outputs require clinician review before deployment.

The authors conclude that future work should focus on developing evaluation protocols that surface this complexity and on methods that can decouple UE quality from underlying model accuracy in low-performance domains.