Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text

Technical Summary: Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text

Problem Statement
While Large Language Models (LLMs) are increasingly deployed for clinical text tasks such as summarization and report revision, their ability to correctly preserve diagnostic uncertainty remains under-explored. In clinical practice, uncertainty expressions (e.g., "possible pneumonia" vs. "pneumonia") are not merely cautious wording; they communicate the strength of evidence, indicate alternative diagnoses, and guide decisions regarding follow-up testing and treatment. Current evaluation metrics for clinical LLMs primarily focus on fluency, coherence, and factual consistency, often failing to detect when models alter the degree of uncertainty. This paper posits that altering these expressions can change the clinical meaning entirely, potentially leading to harmful downstream decisions, yet this specific failure mode is not captured by standard evaluation protocols.

Methodology
The study addresses this gap through a two-step approach: the construction of a specialized benchmark and a dual-mode evaluation of three LLMs.

1. Benchmark Construction:

   • Data Source: The authors constructed a dataset of 1,200 clinical documents derived from MIMIC-IV-Note (discharge summaries and radiology reports) and TCGA-Reports (pathology reports).
   • Annotation Schema: A five-level uncertainty schema was defined, ranging from UL1 (Certain absent) to UL5 (Non-asserted), with intermediate levels for Probable present, Possible present, and Indeterminate/unresolved.
   • Granularity: Unlike document-level labeling, the benchmark utilizes proposition-level annotations. Each document is represented as a set of 9,184 cue-target pairs, where a "cue" is the uncertainty expression (e.g., "cannot exclude") and the "target" is the clinical concept (e.g., "pulmonary embolism").
   • Extraction Pipeline: A rule-based pipeline (utilizing regex and medical NER tools like medSpaCy and QuickUMLS) was employed to extract these pairs to avoid circular evaluation. Human validation on a subset of 90 documents yielded an average accuracy of 82% and a Cohen's $\kappa$ of 0.61.

2. Evaluation Framework:
   The study evaluates three LLMs (gpt-oss-120b, gemini-2.5-flash, and claude-haiku-4.5) using two complementary research questions (RQs):

   • RQ1 (Indirect Assessment): Evaluates uncertainty preservation during downstream text transformation tasks: Clinician Handoff Summarization and Patient-Friendly Revision. These were tested under two prompt conditions: Baseline (standard instruction) and Guarded (explicit instructions to preserve uncertainty and avoid adding interpretations).
   • RQ2 (Direct Assessment): Tests the model's explicit ability to identify and rank uncertainty cues. This involved a single-signal setting (classifying one pair) and a multi-signal setting (classifying and ranking bundles of 2–5 pairs simultaneously).

Key Contributions

• Benchmark Creation: The paper introduces a reusable benchmark of 1,200 documents and 9,184 annotated cue-target pairs covering six clinical text types and five distinct uncertainty levels.
• Empirical Evaluation: It provides a systematic evaluation of three state-of-the-art LLMs, demonstrating that standard fluency metrics fail to capture uncertainty distortion.
• Identification of Failure Modes: The study identifies a systematic bias in LLMs toward "certainty assertion," where hedged language is collapsed into definite claims, a pattern invisible to document-level evaluation.

Results

• Indirect Assessment (RQ1):
  • Poor Preservation: Under baseline conditions, LLMs preserved the original uncertainty level less than 50% of the time (Uncertainty Retained Rate [URR] ranged from 33.64% to 45.55%).
  • Dominant Failure Mode: The primary error was Certainty Assertion (CAR). Roughly 40% of retained targets were rewritten as definite assertions (e.g., changing "possible" to a definitive statement), while partial shifts and over-hedging accounted for less than 8% of errors.
  • Task Variance: Patient-friendly revision tasks resulted in more aggressive content removal and lower uncertainty preservation compared to clinician summarization.
  • Prompt Efficacy: While "Guarded" prompting improved URR by 7–19 percentage points and reduced certainty assertion, it did not eliminate distortion; the best configuration still achieved only ~62.5% URR.
• Direct Assessment (RQ2):
  • Classification vs. Ranking: The model (gemini-2.5-flash) achieved moderate classification accuracy (78.4% ULA) in single-signal settings. However, exact rank accuracy degraded significantly as the number of signals increased (dropping from 86% at N=2 to 48% at N=5).
  • Granularity Issues: The model performed well on categorical distinctions (e.g., certain absent) but struggled with fine-grained distinctions between adjacent levels, particularly between "Probable present" and "Possible present."

Significance and Claims
The paper claims that uncertainty preservation must be treated as a distinct evaluation dimension separate from fluency and factual consistency. The authors argue that the dominant failure mode is not random noise but a systematic bias toward certainty, which poses a significant risk in clinical workflows where tentative diagnoses require different management than confirmed ones.

The study concludes that while explicit prompting offers partial mitigation, it is insufficient for safe deployment. The findings highlight an urgent need for more sophisticated interventions, such as fine-tuning on uncertainty-aware benchmarks or preference optimization, to ensure LLMs do not inadvertently distort clinical meaning by collapsing uncertainty into false certainty. The authors emphasize that their work reveals a failure mode not captured by standard metrics, necessitating a shift in how clinical NLP systems are evaluated and deployed.