💻 computer science

Explaining Sources of Uncertainty in Automated Fact-Checking

The paper introduces CLUE, a plug-and-play framework that generates natural language explanations for automated fact-checking uncertainty by identifying and verbalizing conflicts and agreements between text spans, thereby producing more faithful, informative, and logically consistent outputs than existing baselines.

Original authors: Jingyi Sun, Greta Warren, Irina Shklovski, Isabelle Augenstein

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingyi Sun, Greta Warren, Irina Shklovski, Isabelle Augenstein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Black Box" Detective

Imagine you hire a super-smart AI detective to check if a news story is true. The detective reads a claim (e.g., "Cats can spread a virus") and looks at two different news articles (Evidence 1 and Evidence 2) to decide.

The detective gives you an answer: "This is likely true." But then, it adds a weird, unhelpful note: "I'm 73% sure."

This is frustrating for a human fact-checker. Why are you only 73% sure?

Is it because the two articles contradict each other?
Is it because one article is vague?
Is it because the detective is just guessing?

Current AI systems are like detectives who give you a confidence score but refuse to show their work. They don't tell you which part of the evidence made them hesitate. If the two articles disagree, the AI doesn't point out the specific sentence in Article 1 that clashes with Article 2. It just says, "I'm not 100% sure," leaving you to guess why.

The Solution: CLUE (The "Show Your Work" Framework)

The authors created a new tool called CLUE (Conflict-&Agreement-aware Language-model Uncertainty Explanations).

Think of CLUE as a translator that sits between the AI detective and the human. Instead of just giving a number, CLUE forces the AI to write a plain English explanation that highlights exactly where the confusion lies.

Here is how CLUE works, step-by-step:

1. The "Spot the Difference" Game (Span Interaction)

First, CLUE looks at the Claim and the two Evidence articles. It acts like a highlighter pen, scanning for specific "span interactions."

The Match: It finds where the Claim and Evidence 1 agree (e.g., both mention "cats").
The Clash: It finds where Evidence 1 and Evidence 2 disagree (e.g., Evidence 1 says "cats can spread it," but Evidence 2 says "no transmission was found").

CLUE doesn't just see the whole document; it zooms in on the specific sentences that are talking to (or fighting with) each other.

2. The "Confidence Meter" (Uncertainty Scoring)

The AI calculates a "worry score" (mathematically called entropy). If the AI is confused by conflicting evidence, this score goes up. If the evidence is clear and agrees, the score goes down.

3. The "Storyteller" (Explanation Generation)

This is the magic part. CLUE takes the "worry score" and the "highlighted clashes" and asks the AI to write a story.

Old Way: "I am 73% confident."
CLUE Way: "I am somewhat uncertain because while Evidence 1 agrees that cats can get the virus, Evidence 2 contradicts the part about them spreading it to others. This specific disagreement is why I'm not 100% sure."

How They Tested It

The researchers tested this on three different AI models and two real-world fact-checking datasets (one about health, one about general news).

They compared CLUE against a "standard" AI that just gets a prompt to explain itself without any help.

The Result: The standard AI often made up reasons or gave vague explanations that didn't match its actual confusion.
The CLUE Result: CLUE's explanations were much more honest. They accurately pointed out the specific sentences causing the doubt.

Human testers (real people) preferred CLUE. They said the explanations were:

More helpful: They actually helped them understand the problem.
Less repetitive: They didn't just repeat the text over and over.
More logical: The reasoning made sense.

The Trade-Off (The "Strict" vs. "Chatty" Detective)

The paper found two ways to run CLUE, and they have a slight personality clash:

The Strict Detective (CLUE-Span+Steering): This version forces the AI to stick very closely to the highlighted sentences. It is extremely accurate (faithful) to the math, but sometimes the explanation feels a bit robotic or choppy because it's so focused on the specific words.
The Chatty Detective (CLUE-Span): This version is a bit more flexible. It flows better and sounds more natural to humans, but it might occasionally miss a tiny detail or add a tiny bit of fluff.

The Bottom Line

The paper argues that for AI to be useful in high-stakes jobs like fact-checking, it can't just give a verdict and a number. It needs to show its work.

CLUE is the first tool that can look at a messy pile of conflicting documents, find the exact sentences that are fighting each other, and explain to a human: "I'm unsure because these two specific sentences disagree." This turns the AI from a mysterious oracle into a transparent partner that helps humans verify the truth.

1. Problem Statement

Automated fact-checking using Large Language Models (LLMs) often involves reasoning over multiple documents where evidence may conflict or agree. While LLMs can predict verdicts (Supports, Refutes, Neutral), current explainable AI (XAI) methods fail to adequately communicate model uncertainty to human users.

Limitations of Current Methods: Existing approaches either provide only numerical confidence scores (hard to contextualize) or vague natural language hedges (e.g., "I'm not sure") that do not faithfully reflect the model's internal state.
The Gap: Crucially, existing systems do not explain why a model is uncertain. They fail to surface the specific conflicts and agreements between the claim and evidence, or between different evidence pieces, which are the primary drivers of uncertainty in fact-checking tasks. This prevents human fact-checkers from effectively resolving disagreements or verifying specific points of contention.

2. Methodology: The CLUE Framework

The authors propose CLUE (Conflict-&Agreement-aware Language-model Uncertainty Explanations), a plug-and-play, white-box framework that generates natural language explanations (NLEs) grounded in specific evidence interactions. CLUE operates entirely at inference time without requiring fine-tuning or gold-label explanations.

The framework consists of three stages:

A. Uncertainty Scoring

The system quantifies model uncertainty using predictive entropy derived from the model's output logits.

For a given input $X$ (Claim + Evidence), the entropy $u(X)$ is calculated over the softmax distribution of candidate labels $\{SUPPORTS, REFUTES, NEUTRAL\}$ .
This provides a scalar uncertainty score that is model-agnostic and does not require multiple sampling runs.

B. Conflict and Agreement Span Interaction Identification

To identify the sources of uncertainty, CLUE extracts salient text-span interactions between the Claim ( $C$ ) and Evidence pieces ( $E_1, E_2$ ), as well as between evidence pieces themselves.

Extraction: Using attention mechanisms from the model's final layer, the system identifies important token interactions between input parts. It applies the Louvain algorithm to cluster these tokens into communities, extracting specific span pairs (e.g., a phrase in the claim and a phrase in the evidence).
Relation Labeling: An auxiliary model ( $L_{rel}$ ) labels each extracted span pair as Agree, Disagree, or Unrelated based on the full context.
Selection: The top $k$ (e.g., 3) most important span interactions are selected based on their attention scores.

C. Uncertainty Explanation Generation

CLUE converts these extracted interactions into natural language explanations using two distinct variants:

CLUE-Span (Instruction-Driven): The extracted span pairs and their relation labels are inserted directly into a few-shot prompt. The model is instructed to explain how these specific agreements or conflicts influence its confidence.
CLUE-Span+Steering (Intrinsic Attention Steering): In addition to the prompt, the system modifies the model's attention weights during generation. It down-weights non-target tokens and up-weights the tokens within the selected span interactions, forcing the model to focus its reasoning on the identified conflicts/agreements.

3. Key Contributions

First Framework for Uncertainty-Oriented Explanations: CLUE is the first system to generate NLEs that explicitly trace model uncertainty to specific conflicting or agreeing evidence spans.
White-Box & Plug-and-Play: It operates without fine-tuning, utilizing internal model signals (logits and attention) to ensure the explanations are faithful to the model's actual reasoning process.
Dual Mechanism: It introduces a comparison between instruction-driven prompting and attention steering to optimize the generation of uncertainty explanations.
Human-Centric Design: The framework addresses the practical needs of fact-checkers by "showing the work" behind a decision, specifically highlighting where evidence tensions exist.

4. Experimental Results

The framework was evaluated across three LLMs (Qwen2.5-14B, Gemma-2-9B, OLMo-2-13B) and two datasets (HealthVer and DRUID).

Automatic Evaluation

Faithfulness (Entropy-CCT): The authors introduced Entropy-CCT, a metric measuring the correlation between the presence of a perturbed token in the explanation and the resulting change in the model's predictive entropy.
- Result: Baseline prompting methods showed negative faithfulness (they mentioned unimportant tokens). Both CLUE variants reversed this, with CLUE-Span+Steering achieving the highest positive correlation (up to $r_{pb} = 0.102$ ), indicating the explanations faithfully reflect the model's uncertainty sources.
Span Coverage & Extraneousness: CLUE variants covered more of the extracted critical spans and introduced fewer irrelevant spans compared to baselines.
Label-Explanation Entailment: CLUE explanations were more logically consistent with the predicted verdicts.

Human Evaluation

Participants: 12 human raters evaluated 120 explanations.
Preference: CLUE-generated explanations were ranked as the first preference 67–78% of the time, compared to 24–34% for the baseline.
Qualitative Metrics: CLUE explanations were rated significantly higher in Helpfulness, Coverage, Non-redundancy, and Consistency.
Trade-off: A slight trade-off was observed between the two CLUE variants: CLUE-Span+Steering achieved higher faithfulness (strictly adhering to extracted spans), while CLUE-Span was sometimes rated slightly higher in overall quality/plausibility, likely due to better fluency when extracted spans were fragmentary.

5. Significance and Impact

Bridging the Human-AI Gap: By explicitly linking uncertainty to evidence conflicts, CLUE enables human fact-checkers to understand why an AI is unsure, allowing them to target their verification efforts more effectively.
Reliability in High-Stakes Tasks: The method moves beyond simple verdict prediction to provide actionable insights, which is critical for domains like health and news where misinformation can cause harm.
Generalizability: The framework is applicable to any multi-document reasoning task involving conflicting information, such as question answering or retrieval-augmented generation (RAG).
Future Directions: The paper highlights the need for better span extraction (to avoid fragmentary spans) and further research into balancing faithfulness with plausibility in explanation generation.

In conclusion, CLUE represents a significant step forward in Explainable AI (XAI) for fact-checking, shifting the focus from justifying a verdict to explaining the sources of uncertainty through grounded, evidence-based reasoning.