💬 NLP

Modeling Clinical Uncertainty in Radiology Reports: from Explicit Uncertainty Markers to Implicit Reasoning Pathways

This paper introduces a two-part framework that quantifies explicit uncertainty through hedging phrase ranking and models implicit uncertainty via diagnostic pathway expansion to create Lunguage++, an enriched benchmark for structured radiology reports that enables uncertainty-aware clinical analysis.

Original authors: Paloma Rabaey, Jong Hak Moon, Jung-Oh Lee, Min Gwan Kim, Hangyul Yoon, Thomas Demeester, Edward Choi

Published 2026-03-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Paloma Rabaey, Jong Hak Moon, Jung-Oh Lee, Min Gwan Kim, Hangyul Yoon, Thomas Demeester, Edward Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a radiologist looking at a chest X-ray is like a detective solving a mystery. They look at the clues (the images) and write a report for the doctor who will treat the patient. But here's the catch: detectives aren't always 100% sure. Sometimes they say, "I think there's a broken bone," or "This might be pneumonia." Other times, they leave out the small details because they assume the doctor already knows them, like saying "The patient has pneumonia" without listing the specific fever or cough that led to that conclusion.

This paper is about teaching computers to understand these two types of "uncertainty" in medical reports, so AI can read them just as carefully as a human doctor.

Here is the breakdown of their solution, using some everyday analogies:

1. The "Maybe" Problem (Explicit Uncertainty)

The Issue: Radiologists use "hedging" words like "possible," "suggests," "likely," or "cannot be excluded."

Old Way: Computers used simple rules. If they saw the word "possible," they just marked it as "uncertain." But "possible" in one sentence might mean a 30% chance, while in another, it might mean a 70% chance. It's like a traffic light that just says "Go or Stop" but doesn't tell you if you have 5 seconds or 30 seconds left.
The New Solution (The "Taste-Test" Panel): The researchers built a system using advanced AI (Large Language Models) to act like a panel of expert judges.
- They took thousands of sentences and asked the AI: "Which of these two sentences sounds more certain?"
- Imagine a game of "Rock, Paper, Scissors" played millions of times. The AI compares phrases like "probably" vs. "suggests" against each other to build a ranking list.
- Once they have the ranking, they can assign a specific probability score (like 0.65 or 0.82) to every finding. Now, instead of just knowing something is "uncertain," the computer knows exactly how uncertain it is.

2. The "Missing Puzzle Pieces" Problem (Implicit Uncertainty)

The Issue: Radiologists often skip steps in their reasoning to save time. If they write "Congestive Heart Failure," they might not explicitly write down "enlarged heart" or "fluid in the lungs," even though those are the reasons they diagnosed the failure.

The Analogy: Imagine a chef writes a recipe that just says "Make a Lasagna." They don't list "boil pasta," "make sauce," or "layer cheese" because they assume you know how to make lasagna. If a robot tries to follow that recipe, it might get confused about what ingredients are actually needed.
The New Solution (The "Recipe Expansion" Kit): The researchers created 14 "Diagnostic Pathways" (think of them as detailed flowcharts or recipe books) for common diseases.
- When the computer sees "Heart Failure," it doesn't just stop there. It looks at the flowchart and says, "Ah, if they have Heart Failure, they must also have an enlarged heart and fluid in the lungs."
- The system then fills in the missing pieces automatically. It adds those hidden details back into the report, marking them as "inferred" but highly likely. This makes the computer's understanding of the patient's condition much more complete.

3. The Result: "Lunguage++"

By combining these two tricks, the team created a new, super-charged dataset called Lunguage++.

Before: A computer read a report and saw a list of findings with vague labels like "maybe" or "definitely."
After: The computer sees a list where every finding has a precise confidence score (e.g., "Pneumonia: 65% likely") AND it has filled in the missing logical steps (e.g., "Pneumonia -> implies Fever + Cough").

Why Does This Matter?

Think of medical AI as a student learning to be a doctor.

If you teach the student with vague notes, they will guess wrong.
If you teach them with Lunguage++, they learn to understand the nuance of doubt and the logic behind a diagnosis.

This helps AI:

Read reports better: It won't miss a subtle "maybe" that could be a serious warning.
Reason like a human: It understands that a diagnosis is built on a chain of smaller clues, even if the radiologist didn't write them all down.
Make safer decisions: By knowing exactly how uncertain a finding is, AI can tell doctors, "I'm not sure about this one, please double-check," rather than confidently giving a wrong answer.

In short, this paper teaches computers to stop reading medical reports like a robot and start reading them like a thoughtful, experienced detective.

1. Problem Statement

Radiology reports are critical for clinical decision-making but inherently contain two distinct types of uncertainty that hinder automated analysis and AI training:

Explicit Uncertainty: Arises from hedging phrases (e.g., "possible," "suggests," "may represent") used by radiologists to qualify findings. Current rule-based systems often treat these as binary "uncertain" labels, failing to capture the degree of uncertainty, which varies significantly based on context and specific phrasing.
Implicit Uncertainty: Arises from the selective nature of reporting. Radiologists often omit intermediate reasoning steps or sub-findings to maintain conciseness (e.g., stating "Congestive Heart Failure" without explicitly listing "cardiomegaly" or "pulmonary edema"). This creates ambiguity: are omitted findings truly absent, or simply unrecorded? Misinterpreting unrecorded evidence as absence biases data and distorts diagnostic reasoning models.

Existing datasets (like CheXpert or MIMIC-CXR) often lack fine-grained uncertainty quantification or mechanisms to reconstruct these implicit reasoning pathways.

2. Methodology

The authors propose a two-part framework to address these issues, resulting in the release of Lunguage++, an expanded version of the existing Lunguage benchmark dataset.

A. Quantifying Explicit Uncertainty (Continuous Probability)

Instead of binary labels, the authors map findings to a continuous probability scale ($0$ to $1$) representing the likelihood of a finding's presence.

Vocabulary Extraction: Using an LLM (Gemini), they extracted 42 common hedging phrases from 2,066 "tentative" finding-sentence pairs in the Lunguage dataset.
Reference Ranking via TrueSkill: To rank these phrases by certainty, they employed the TrueSkill algorithm (a Bayesian rating system used in competitive gaming).
- LLM as Judge: Four LLMs (Gemini, GPT-4o, Claude, MedGemma) performed pairwise comparisons of sentences containing different hedging phrases to determine which implied higher certainty.
- Aggregation: 8,610 pairwise comparisons were conducted. The TrueSkill algorithm inferred a global skill level ( $\mu$ ) for each phrase, creating a robust reference ranking.
Mapping to Probability: The TrueSkill scores were mapped to a probability scale ( $p \in [0, 1]$ ) using a sigmoid function. The transformation was anchored by expert radiologists who assigned probabilities to the highest ("most likely") and lowest ("less likely") ranked phrases.
Sentence Fitting: For new finding-sentence pairs, the system iteratively compares the sentence against the reference ranking using TrueSkill to assign a specific probability score.

B. Modeling Implicit Uncertainty (Pathway Expansion)

To address omitted findings, the authors introduced a Pathway Expansion Framework based on expert-defined diagnostic logic.

Diagnostic Pathways (DAGs): Experts constructed 33 unique Directed Acyclic Graphs (DAGs) for 14 common Chest X-ray (CXR) diagnoses (e.g., Pneumonia, CHF, Pneumothorax). These graphs map high-level diagnoses to characteristic sub-findings (e.g., CHF $\to$ $\to$ Cardiomegaly + Pulmonary Edema + Dyspnea).
- Pathways account for context: imaging view (PA vs. AP), patient position, and acuity (acute vs. chronic).
Expansion Algorithm:
- Deduplication: Merges semantically similar findings to avoid redundancy.
- Pathway Matching: Aligns original findings with the correct diagnostic pathway variant based on location, attributes, and view.
- Recursive Expansion: If a diagnosis is present (positive status), the framework recursively infers and adds all child sub-findings from the DAG.
- Property Propagation: Inferred sub-findings inherit the diagnostic status, certainty level, and probability of the parent diagnosis.
Conflict Resolution: A rule-based protocol resolves contradictions between original reports and inferred findings (e.g., if the report says "no edema" but the pathway infers it, the original report takes precedence).

3. Key Contributions

Continuous Uncertainty Quantification: A novel framework using LLM-based pairwise comparisons and TrueSkill to assign precise probability scores to explicit uncertainty, moving beyond discrete classification.
Implicit Reasoning Reconstruction: The first framework to systematically model implicit uncertainty by expanding reports using expert-defined diagnostic pathways, effectively "filling in" omitted diagnostic evidence.
Lunguage++ Dataset: The release of an enriched benchmark containing:
- 14,049 original structured findings.
- 4,761 newly inferred sub-findings (+33.9% expansion).
- Continuous probability scores for all tentative findings.
- Expert-validated reference rankings of hedging phrases.
Expert-Validated Resources: Publication of a reference ranking of 42 hedging phrases and 33 diagnostic pathways for 14 common CXR conditions.

4. Results and Analysis

Explicit Uncertainty:
- The reference ranking of 42 phrases showed high stability (Spearman correlation > 0.99 between different random seeds).
- Expert evaluation confirmed strong agreement between human radiologists and the LLM-generated ranking (Average agreement ~0.78).
- The resulting probabilities for tentative findings ranged from 0.102 to 0.892, with a mean of 0.459, providing a nuanced view of diagnostic confidence.
Implicit Uncertainty:
- The expansion framework identified 2,639 findings (18.7% of the dataset) eligible for expansion.
- It successfully inferred 4,761 additional sub-findings, significantly enriching the dataset's granularity.
- Conflict Rate: Conflicts between original and expanded data were rare (3.2% of cases), and the resolution protocol successfully maintained logical and clinical consistency.
Cost Efficiency: The LLM-based ranking process cost approximately $92.16 total ($0.045 per pair), demonstrating scalability compared to manual expert rating.

5. Significance and Future Impact

Improved AI Training: Lunguage++ enables the training of "uncertainty-aware" AI models. Image classifiers can learn to predict not just the presence of a disease, but the confidence associated with it, aligning better with clinical reality.
Faithful Reasoning: By reconstructing implicit pathways, the dataset allows Vision-Language Models (VLMs) to learn the full chain of diagnostic reasoning, rather than just the final conclusion.
Clinical Decision Support: The framework provides a structured way to quantify and visualize diagnostic uncertainty, which is crucial for downstream clinical outcomes and risk stratification.
Generalizability: While demonstrated on CXR reports, the methodology (LLM-based ranking and pathway expansion) is applicable to other medical domains and datasets (e.g., CheXpert, MIMIC-CXR).

Limitations: The authors note that the probability mapping relies on a small pool of radiologists for anchoring, and the pathway expansion assumes a unidirectional (top-down) logic, which may not capture complex, bidirectional clinical reasoning or cyclic dependencies. Future work aims to incorporate visual grounding to verify inferred findings against actual images.