💬 NLP

Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models

This paper argues that medical vision-language models should be deployed for calibrated triage rather than full autonomy, demonstrating that while standard confidence metrics are poor guides, specific estimators can significantly reduce confidently wrong answers to enable safe automated handling of only a subset of cases (particularly in radiology) while routing the rest to clinicians.

Original authors: Reza Khanmohammadi, Kundan Thind, Mohammad M. Ghassemi

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Reza Khanmohammadi, Kundan Thind, Mohammad M. Ghassemi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of very smart, chatty robots (called Vision-Language Models) that are supposed to help doctors look at medical pictures like X-rays or microscope slides. These robots are great at talking; they can describe a picture fluently and sound very confident.

But here is the problem: These robots often lie about what they are looking at.

Sometimes, a robot will look at an X-ray, ignore the picture entirely, and just guess the answer based on what it has read in textbooks before. It will say, "I am 99% sure this is a broken bone," even though it didn't actually look at the bone. In medicine, this is dangerous because the robot sounds trustworthy, but it's actually guessing.

This paper asks a simple but critical question: How do we know when to trust the robot, and when to tell it to shut up and let a human doctor take over?

The researchers didn't just ask, "Is the robot smart?" They asked, "Is the robot's confidence meter reliable?"

The Experiment: A "Trust but Verify" Test

The researchers tested seven different "confidence meters" (ways to measure how sure the robot is) across three different types of medical pictures:

Radiology: X-rays and CT scans (like looking at a car engine).
Broad Clinical: A mix of many different body parts.
Pathology: Microscope slides of tissue (like looking at tiny bugs under a lens).

They used five different robot brains (models) and tested them on medical data the robots had never seen before. The goal was to see which confidence meter could correctly say, "I'm not sure, don't trust me," when the robot was actually guessing.

The Key Findings (The "Aha!" Moments)

1. The "Confidence Meter" Matters More Than the Robot
It doesn't matter if you have the smartest robot in the world. If its confidence meter is broken, you can't use it safely. The study found that the way you measure confidence is more important than which robot you are using.

2. The "High-Confidence" Trap
The most dangerous moment is when the robot is wrong but very confident.

The Bad Meters: Some methods (like asking the robot to just "tell you how sure it is") were terrible. When they were wrong, they were still confident about 40–45% of the time. It's like a weatherman saying, "I'm 100% sure it's sunny," while standing in a hurricane.
The Good Meters: The best methods (trained "probes" that look inside the robot's brain) were much better. When they were wrong, they were only confident 1–4% of the time. They knew when to back down.

3. The "Ceiling" Effect (The Glass Ceiling of Capability)
The researchers found a hard limit on how much work can be automated, and it depends on the type of picture:

Radiology (X-rays): The robots are decent here. With a good confidence meter, you can safely automate about one-third of the cases. The robot can handle the easy ones, and the meter tells you to send the hard ones to a human.
Pathology (Microscope slides): The robots are terrible here. Even with the best confidence meter, you can't safely automate almost any of these cases. The robot just isn't good enough at this specific task yet.
The Analogy: Imagine a robot trying to sort fruit.
- On apples (Radiology), it's pretty good. You can let it sort the big, obvious ones, and a human checks the weird-looking ones.
- On tiny seeds (Pathology), the robot is blind. No matter how good your "confidence meter" is, you can't let the robot sort the seeds because it keeps dropping them. The robot's skill level sets a "ceiling" on how much work you can give it.

4. The "Grounding" Problem
The best confidence meters are "grounding-aware." This means they can tell if the robot is actually looking at the picture or just guessing from memory.

If the robot ignores the image and guesses, a good meter says, "Hey, you didn't look at the picture! Lower your confidence!"
A bad meter just says, "I feel confident!" even though the robot is hallucinating.

The Bottom Line: "Calibrated Triage," Not "Autonomy"

The paper concludes that we should not try to let these robots work alone (autonomy). Instead, we should use them for Calibrated Triage.

Think of a Triage Nurse in an emergency room:

The robot acts as the first filter.
If the robot's confidence meter says, "I am 95% sure, and I actually looked at the picture," the robot handles it.
If the meter says, "I'm shaky," or "I didn't really look at the image," the robot immediately passes the patient to a human doctor.

The Takeaway:
We can't trust these robots to work alone yet. But if we use the right "confidence meter," we can safely let them handle the easy cases (like some X-rays) while routing the hard or risky cases (like pathology) to humans. The most important tool isn't a smarter robot; it's a better way to know when the robot is bluffing.

Technical Summary: Calibrated Triage, Not Autonomy

Problem Statement

Large Vision-Language Models (LVLMs) demonstrate a critical failure mode in medical imaging: they can generate fluent, confident answers to questions about radiographs or pathology slides while barely utilizing the visual input, relying instead on language priors. In clinical settings, this behavior is dangerous because the output appears trustworthy despite being ungrounded. The core problem is not merely model accuracy, but the reliability of confidence estimation. Current LVLMs are systematically overconfident, reporting similar confidence scores for correct and incorrect answers, particularly on high-risk cases. Existing confidence estimators are typically tuned for general-domain ranking under uniform error costs, failing to address the clinical requirement of selective prediction—the ability to distinguish cases that can be safely automated from those requiring clinician review while maintaining a strict error budget.

Methodology

The study evaluates seven confidence estimators across five open-weight LVLMs (ranging from 4.5B to 27B parameters) and three medical Visual Question Answering (VQA) datasets:

GMAI-MMBench: Broad clinical imaging (radiology, pathology, dermatology, etc.).
SLAKE: Radiology and anatomy (CT, MRI, X-ray).
PATH-VQA: Pathology and histopathology.

Estimators Evaluated:

Inference-only (Black-box): Prompt-based self-reports ( $P(\text{True})$ ), Self-Probing, and Prompt Ensembles.
Trained Internal Probes: SAPLMA (post-answer state), CCPS (representation stability), InternalInspector (pooled attention/feed-forward states), and BICR (grounding-aware, trained on real vs. blank image contrast).

Experimental Design:

Out-of-Distribution (OOD) Training: All trainable probes were trained exclusively on the general-domain GQA dataset (natural images) and applied to medical datasets without adaptation, simulating a realistic deployment scenario where models must generalize to unseen medical scans.
Ground Truth: Correctness labels were assigned using a single $gpt-5-mini$ judge to ensure consistent semantic equivalence grading across datasets.
Evaluation Framework: The study reframes confidence estimation as bounded selective prediction. A case is automated only if its confidence exceeds a threshold; otherwise, it is deferred. Performance is measured by:
- Discrimination: AUROC.
- Calibration: Expected Calibration Error (ECE).
- Safe Yield: The maximum fraction of cases automatable while keeping the error rate of automated cases below a tolerance (e.g., 20%).
- High-Confidence Reliability: Error rates in the top confidence decile (ErrTop10%).

Key Results

1. Standard Metrics Are Poor Guides

Global discrimination (AUROC) and average calibration (ECE) fail to identify usable estimators for clinical triage.

Discrimination: Differences in AUROC between methods are often negligible (e.g., 61.9 vs. 62.9), making methods appear interchangeable based on ranking alone.
Calibration: Average calibration can be artificially improved via off-domain temperature scaling without improving the "safe yield" (the actual volume of work that can be automated).
The Critical Gap: The differentiator is behavior in the high-confidence region. Inference-only baselines assign high confidence (>0.8) to wrong answers 41–45% of the time. In contrast, the best trained probes (BICR and InternalInspector) reduce this to 1–4%.

2. The Capability-to-Automatability Gradient

Safe automation is governed by two levels:

Base Model Competence (The Ceiling): The maximum possible automation is strictly limited by the model's base accuracy on a specific domain.
- Radiology (SLAKE): Base accuracy ~61%. Safe yield at 20% error tolerance reaches ~34.6% (BICR).
- Broad Clinical (GMAI): Base accuracy ~47%. Safe yield drops to ~4.5%.
- Pathology (PATH-VQA): Base accuracy ~39.5%. No method safely automates any cases at a 20% error tolerance.
Confidence Layer (The Reach): Even within the achievable ceiling, the confidence layer determines how much can be safely reached. On pathology, the confidence layer fails to identify the low-error subset that the base model's competence theoretically permits.

3. Domain and Model Specificity

No single estimator is universally best.

BICR and InternalInspector generally outperform others in radiology and broad clinical domains.
BICR excels in pathology discrimination but fails to improve safe yield due to the low base competence ceiling.
Prompt Ensembles sometimes outperform complex internal probes in pathology, suggesting that for domains where the model is fundamentally unreliable, expensive internal instrumentation offers no tail-safety advantage.
Grounding Awareness: Only grounding-aware methods (like BICR) significantly lower confidence when the model answers without using the image (e.g., knowledge-based questions), a failure mode critical for clinical safety.

Significance and Claims

The paper argues that the immediate role of medical LVLMs is calibrated triage, not autonomy.

Triage over Autonomy: Systems should automate only the cases marked as safe by a calibrated score and route all others to clinicians.
Measurement Matters More Than Model: How confidence is measured and aggregated is more critical than which specific LVLM produces the answer. A well-calibrated, grounding-aware estimator can convert an unreliable LVLM into a useful triage tool, recovering roughly one-third of radiology workload at moderate error tolerances.
Rejection of Pooled Benchmarks: Aggregated leaderboard scores are misleading. Estimators must be validated per model and per clinical category (e.g., department or anatomical region) because performance varies significantly across domains and imaging modalities.
Safety Signal: The most valuable signal for a clinician is a low rate of confident errors in the high-confidence region. Standard ranking metrics do not measure this; only specific, grounding-aware probes provide this safety.

The study concludes that while safe automation is currently bounded by base model competence (especially failing in pathology), a calibrated, grounding-aware confidence layer provides the necessary signal to safely delegate a portion of the workload, provided the deployment is strictly limited to cases where the confidence score indicates safety.