Technical Summary: Mind the Gap – Benchmarking LLM Uncertainty and Calibration in Clinical QA

Problem Statement

The deployment of Large Language Models (LLMs) in high-stakes clinical domains, such as clinical question answering (QA), is hindered by the models' tendency toward overconfidence. Reliable Uncertainty Quantification (UQ) is critical for safe deployment, requiring not only discrimination (the ability to distinguish correct from incorrect predictions) but also calibration (the alignment between predicted confidence and actual accuracy).

Existing literature often fails to address UQ in clinical settings comprehensively. Prior studies are frequently limited to:

• Narrow sets of outdated models, excluding recent reasoning-oriented and domain-specific LLMs.
• Aggregated metrics that obscure performance variations across specific medical specialties and question types.
• A lack of systematic comparison between different UQ paradigms (score-based vs. set-based) and model architectures (general-purpose vs. biomedical vs. reasoning).
• A focus on classification tasks, leaving the challenges of uncertainty in natural language generation (NLG) largely unexplored in clinical contexts.

Methodology

The authors conducted a comprehensive benchmark across 10 open-source LLMs (including general-purpose, biomedical, and reasoning-focused models) and representative proprietary models. The evaluation utilized two primary datasets: S-MedQA (covering 6 common medical specialties) and MedExQA (covering 5 less frequent specialties), totaling 1,810 questions.

Data Annotation

A novel annotation scheme was introduced to map QA pairs onto 6 distinct question types: Diagnosis, Treatment, Diagnostic Test, Definition, Procedure/Operation, and Other. This allows for granular analysis of how task difficulty and uncertainty profiles shift across clinical contexts.

Uncertainty Estimation Methods

The study evaluated four categories of UQ approaches:

1. Score-based (Single-pass):
   • Logit-based: Average token-level probability (greedy decoding) and G-NLL (negative log-likelihood of the greedily decoded output).
   • Linguistic Confidence: A two-step approach where the model generates an answer and then assigns a scalar certainty score.
2. Consistency-based (Sampling-based):
   • Semantic Entropy: Generates multiple responses (10 per question), groups them by selected option, and computes the entropy of the resulting distribution.
3. Set-based:
   • Conformal Prediction: Constructs prediction sets by sampling responses to achieve a target marginal coverage (90%).
4. Behavioral Features (Novel Case Study):
   • A lightweight, single-pass method extracting linguistic cues from reasoning traces of reasoning-oriented models (e.g., DeepSeek-R1, s1.1). Features include the number of reasoning tokens, self-posed questions, and occurrences of the token "Wait." These features are fed into a supervised calibration regression model to predict uncertainty.

Evaluation Metrics

Performance was assessed using:

• Discrimination: Area Under the Receiver Operating Characteristic Curve (AUROC).
• Calibration: Expected Calibration Error (ECE) and Brier Score.
• Set-based Metrics: Empirical coverage and average set size.

Key Results

1. Heterogeneity Across Specialties and Question Types

Uncertainty reliability is not a monolithic property. Significant variations were observed:

• Specialty: Performance fluctuates drastically across the 11 clinical specialties. For instance, all models performed near chance levels on Speech-Language Pathology questions, while reasoning models excelled in Pediatrics and Infectious Diseases.
• Question Type: "Procedure/Operation" questions proved most challenging, with all models underperforming. Conversely, "Definition" questions were the easiest.
• Decoupling: High accuracy does not guarantee good calibration or discrimination. For example, the smallest model (Llama-3.2-3B) achieved the highest AUROC in some specialties despite low accuracy, while larger models sometimes exhibited overconfidence (high accuracy but poor calibration).

2. Performance of UQ Methods

• Semantic Entropy: Consistently provided the strongest overall performance in both calibration and discrimination across models, though it requires multiple generations (sampling).
• Single-pass Methods: Most single-pass methods (e.g., Avg Token LogProb, G-NLL) failed to achieve both good calibration and discrimination simultaneously. G-NLL underperformed despite prior success in non-clinical QA.
• Behavioral Features: The novel method using behavioral cues from reasoning traces (e.g., "Wait" tokens) achieved performance close to Semantic Entropy but with only a single forward pass. This offers a lightweight, interpretable alternative for latency-sensitive applications.
• Model Types: Reasoning-oriented models (DeepSeek-R1, s1.1) generally demonstrated superior accuracy and calibration. Biomedical models showed advantages at smaller scales (8B) but did not consistently outperform general-purpose models at larger scales (70B).

3. Conformal Prediction

Set-based methods revealed that high accuracy does not automatically translate to reliable set coverage.

• DeepSeek-R1-70B achieved the best balance, reaching 93.8% coverage with compact sets.
• Llama-3.1-70B fell short of the 90% coverage target (85.5%), indicating overconfidence and poor generalization from score-based metrics to set-based reliability.
• Llama3-OpenBioLLM-70B achieved high coverage (90.1%) but with larger set sizes, indicating conservativeness, and failed to meet targets in specific categories like Infectious Diseases.

Key Contributions

1. Granular Benchmarking: The first large-scale evaluation of UQ methods across 11 clinical specialties and 6 question types, revealing that aggregate metrics obscure critical reliability differences.
2. Comprehensive Model Comparison: A systematic comparison of general-purpose, biomedical, and reasoning-focused LLMs, highlighting that domain-specific tuning does not guarantee superior uncertainty estimation.
3. Novel Behavioral Proxy: Introduction of a lightweight, single-pass uncertainty estimation method based on reasoning traces, which approximates the performance of resource-intensive sampling methods.
4. Paradigm Comparison: Demonstration of the misalignment between score-based and set-based uncertainty estimates, emphasizing the need for multi-faceted evaluation.

Significance and Claims

The paper argues that uncertainty reliability is context-dependent, varying significantly by clinical specialty and question type. The authors claim that "one-size-fits-all" evaluations are insufficient and potentially misleading for high-stakes clinical deployment.

The primary contribution is a cautionary yet constructive perspective:

• Caution: Aggregate metrics can hide dangerous overconfidence in specific sub-domains.
• Construction: The proposed behavioral feature method offers a practical, interpretable path forward for real-world clinical decision support where sampling is infeasible.

The authors position this work as a first step toward developing LLMs that are both accurate and trustworthy in clinical use, advocating for evaluations that account for specific clinical contexts rather than relying on global averages. They emphasize that future work must involve clinicians to validate which uncertainty signals best indicate the need for human oversight.