💬 NLP

Uncertainty-Aware Web-Conditioned Scientific Fact-Checking

This paper presents an uncertainty-aware, web-conditioned framework for scientific fact-checking that decomposes claims into atomic facts, verifies them against local context, and selectively invokes web searches only for uncertain cases to achieve interpretable, cost-efficient, and high-accuracy verification in specialized domains.

Original authors: Ashwin Vinod, Katrin Erk

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ashwin Vinod, Katrin Erk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, you are investigating a scientific claim like, "This new drug cures headaches in 10 minutes."

In the past, AI detectives (Large Language Models) would often guess the answer based on what they "remembered" from their training. Sometimes they were right, but often they would hallucinate—making up facts that sounded plausible but were completely wrong. They were like a detective who confidently tells you the butler did it, even though they never actually looked at the evidence.

This paper introduces a new, smarter detective system called Atomic+Search. It's designed to be careful, honest, and efficient. Here is how it works, broken down into simple steps:

1. The "Lego Block" Strategy (Atomic Decomposition)

Imagine the scientific claim is a giant, complicated Lego castle. If you try to check if the whole castle is real at once, it's overwhelming. You might miss a fake brick hidden in the middle.

Instead, this system breaks the claim down into tiny, individual Lego blocks (called "atomic facts").

Old way: Check the whole sentence.
New way: Break it down.
- Claim: "The drug cures headaches in 10 minutes."
- Block 1: "The drug exists."
- Block 2: "The drug treats headaches."
- Block 3: "It works in 10 minutes."

By checking these tiny blocks one by one, the system can pinpoint exactly which part of the claim is true and which part is fake.

2. The "Local Librarian" (Context-Only Check)

First, the system looks at the document you gave it (the "evidence snippet"). Think of this as a Local Librarian who only has access to the specific book you handed them.

The system asks the Librarian: "Does this book say Block 1 is true?"
If the book clearly says "Yes," the Librarian stamps it Supported.
If the book clearly says "No," it stamps it Refuted.
Crucial Step: If the book is silent or confusing, the Librarian doesn't guess. They mark it as "Uncertain" (or NEI - Not Enough Information). They refuse to lie just to give you an answer.

3. The "Uncertainty Gate" (When to Call the Internet)

This is the most important innovation. In many AI systems, if the AI isn't sure, it might just guess or search the whole internet blindly, which is slow and expensive.

This system has a Gatekeeper.

If the Local Librarian is 100% sure (High Confidence), the Gatekeeper stays closed. No internet search is needed.
If the Librarian is "Uncertain" (the middle ground), the Gatekeeper opens.
The Rule: The system only calls the "Internet" (a specialized search engine) for the specific blocks it is unsure about. It doesn't waste time searching for things it already knows.

4. The "Trusted Source" Search (Domain-Restricted)

When the Gate opens, the system doesn't just search Google (which is full of blogs, ads, and fake news). It searches a VIP Club of trusted scientific sources like the NIH, WHO, CDC, and PubMed.

It asks: "Hey, do these trusted medical sources confirm Block 3?"
If the new evidence agrees, great!
The Safety Net: If the new evidence from the internet contradicts the original book you gave us, the system abstains. It says, "I can't decide because the sources disagree," rather than forcing a wrong answer. It chooses to say "I don't know" rather than lie.

5. The "Chief Detective" (Final Verdict)

Finally, a "Chief Detective" (a Judge AI) looks at all the stamped Lego blocks.

If most blocks are "Supported," the whole claim is Supported.
If the key blocks are "Refuted," the claim is Refuted.
If the system had to say "I don't know" on a critical block, the whole claim gets an NEI (Not Enough Information) label.

Why is this a big deal?

It's Cheaper: Because it only searches the internet when it's truly confused, it saves money and time.
It's Honest: It admits when it doesn't know, rather than making things up.
It's Explainable: You can see exactly which Lego block failed. Instead of just saying "This is fake," it can say, "The drug exists, but it doesn't work in 10 minutes."
It's Safe: It's perfect for high-stakes fields like medicine, where a wrong guess could be dangerous.

In short: This paper teaches AI to stop guessing, break big problems into small pieces, only look up answers when it's stuck, and always admit when it doesn't have enough proof. It's like upgrading from a confident but unreliable guesser to a meticulous, evidence-based scientist.

1. Problem Statement

Scientific fact-checking in high-stakes domains (e.g., biomedicine, materials science) faces significant challenges with current Large Language Models (LLMs):

Hallucination & Inconsistency: Models often rely on surface cues or parametric memory, leading to inconsistent reasoning and hallucinations when local evidence is sparse.
Complexity: Technical claims often involve complex predicate-argument structures that sentence-level models fail to interpret correctly.
Cost vs. Accuracy: Existing Retrieval-Augmented Generation (RAG) systems often retrieve indiscriminately, increasing latency and cost, or fail to handle cases where retrieved evidence conflicts with the provided context.
Lack of Traceability: Many systems provide a verdict without clear, interpretable rationales or the ability to abstain (declare "Not Enough Information") when evidence is insufficient.

The authors propose a system that verifies claims with atomic granularity, uses calibrated uncertainty to trigger external searches, and maintains strict source control to ensure reliability.

2. Methodology: The Atomic+Search Pipeline

The proposed framework, Atomic+Search, is a modular pipeline that couples symbolic decomposition with probabilistic verification and targeted retrieval. It operates in five stages:

A. Atomic Fact Decomposition

A reasoning-capable LLM decomposes a complex natural language claim ( $c$ ) into a set of atomic facts ( $F = \{f_i\}$ ).
Each atomic fact is a minimal predicate-argument tuple (max 25 words), ensuring that each unit targets a single verifiable assertion.
Output is structured as strict JSON.

B. Semantic Local-Evidence Selection

The input document ( $D$ ) is chunked into overlapping windows (~420 characters).
For each atomic fact, the system computes embeddings (using text-embedding-3-large) to find the local snippet with the highest cosine similarity.
If embeddings are unavailable, a token-overlap heuristic is used.

C. Lightweight Verification (Grounded Checker)

A compact, evidence-grounded verifier (MiniCheck-7B) evaluates each atomic fact against its selected local snippet.
It outputs a calibrated support probability $p_i \in [0, 1]$ .
Thresholds:
- $p_i \geq 0.80$ : Labeled Supported.
- $p_i \leq 0.25$ : Labeled Refuted.
- $0.25 < p_i < 0.80$ : Labeled Uncertain.

D. Uncertainty-Gated Web Corroboration

Selective Retrieval: Web search is only triggered for facts falling in the "Uncertain" band ( $0.25 < p_i < 0.80$ ).
Domain Restriction: Queries are restricted to authoritative, vetted domains (e.g., PubMed, WHO, CDC, FDA, NIH, ClinicalTrials.gov, Wikipedia).
Conflict Handling: If retrieved web evidence conflicts with the original local context, the system abstains (outputs NEI) rather than overriding the provided context. This prevents the system from hallucinating based on potentially noisy external data.
Augmentation: Retrieved snippets are summarized with citations and concatenated to the original evidence for re-evaluation by MiniCheck.

E. Final Deductive Judge

A Judge LLM aggregates the results of the atomic facts.
It receives the original claim, the document, and the sets of Supported ( $S$ ) and Refuted ( $R$ ) facts.
It outputs a final verdict (Supported, Refuted, or NEI) and a short, interpretable rationale referencing specific fact IDs.
The system supports both binary (2-way) and tri-valued (3-way) classification.

3. Key Contributions

Atomic Decomposition Strategy: Moving from sentence-level verification to predicate-argument decomposition significantly improves precision and interpretability.
Uncertainty-Gated Corroboration: The system avoids routine web searching, invoking external evidence only when local confidence is low. This balances cost/latency with accuracy.
New Dataset (PubMedFact1k): The authors released a 1,000-example, three-way (Supported/Refuted/NEI) medical claim verification dataset derived from PubMedQA to support future research.
Conservative Abstention Mechanism: A novel protocol where the system refuses to override provided context with conflicting web evidence, prioritizing traceability and safety in high-stakes settings.
Comprehensive Evaluation: Demonstrated performance across biomedical (BIONLI-300, PubMedFact1k) and climate (CLIMATE-FEVER) domains, showing strong out-of-domain generalization.

4. Experimental Results

The system was evaluated against strong baselines, including compact verifiers (MiniCheck), tool-augmented LLMs (GPT-5 Mini, Qwen-32B), and retrieval-centric systems (RARR).

BIONLI-300 (Biomedical, 2-way):
- Atomic+Search achieved 66.7% F1 and 68.7% Balanced Accuracy.
- This represents a +6.0 F1 improvement over sentence-level MiniCheck (60.7%) and a +4.7 F1 gain over the strongest baseline (GPT-5 Mini + Search).
- Ablation studies showed that removing web search dropped F1 by 4.7%, while removing atomic decomposition dropped it by 6.4%, indicating decomposition is the primary driver of performance.
PubMedFact1k (Biomedical, 3-way):
- Achieved the top Macro-F1 of 73.7%, outperforming GPT-o1 (71.2%) and GPT-5 Mini + Search (72.5%).
- Significantly reduced ambiguous NEI vs. Supported/Refuted errors.
CLIMATE-FEVER (Climate, 2-way):
- Achieved 73.83% Balanced Accuracy and 70.04% F1, demonstrating strong transferability to non-biomedical domains without retraining.
Efficiency: Web corroboration was invoked for only a minority of atomic facts on average, confirming the efficiency of the uncertainty-gating mechanism.

5. Significance and Conclusion

The paper demonstrates that combining atomic reasoning with calibrated uncertainty handling yields a fact-checking system that is more accurate, interpretable, and cost-effective than current state-of-the-art approaches.

Interpretability: By breaking claims into atomic units, the system provides traceable rationales for every part of a verdict.
Cost Control: The "uncertainty-gated" approach ensures expensive web searches are only used when necessary, making the system suitable for high-throughput or latency-constrained environments.
Safety: The conservative handling of conflicting evidence (abstaining rather than overriding) makes the system suitable for high-stakes domains like healthcare, where hallucinations can have severe consequences.

The authors conclude that this pipeline offers a robust framework for scientific fact-checking that prioritizes provenance, traceability, and conservative decision-making, setting a new standard for reliable AI in specialized domains.