💬 NLP

Contradiction to Consensus: Dual Perspective, Multi Source Retrieval Based Claim Verification with Source Level Disagreement using LLM

This paper proposes a novel open-domain claim verification system that leverages large language models and multi-source retrieval to aggregate evidence from diverse perspectives, including contradictory information, thereby enhancing verification accuracy, transparency, and interpretability compared to traditional single-source approaches.

Original authors: Md Badsha Biswas, Ozlem Uzuner

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Md Badsha Biswas, Ozlem Uzuner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to settle a heated argument between two friends about a specific fact, like "Does eating chocolate cure a cold?"

In the past, automated fact-checkers were like a single, overworked librarian. You would ask them, "Is this true?" and they would rush to one specific bookshelf (usually Wikipedia), find the first page that mentions chocolate, and say, "Yes, it's true!" or "No, it's fake!" They rarely looked at other bookshelves, and they never asked, "What if the opposite were true?"

This paper introduces a new, super-smart fact-checker that acts more like a team of investigative journalists working together. Here is how it works, broken down into simple steps:

1. The "Devil's Advocate" Strategy (Dual-Perspective)

Most fact-checkers only look for evidence that supports a claim. This paper says, "That's biased! Let's look at the other side, too."

The Old Way: You ask, "Is it true that chocolate cures colds?" The system searches for "chocolate cures colds."
The New Way: The system asks two questions at once:
1. "Is it true that chocolate cures colds?"
2. "Is it true that chocolate does NOT cure colds?" (The negated claim).

The Analogy: Imagine you are hiring a lawyer. The old way is hiring a lawyer who only tries to prove you are innocent. The new way hires two lawyers: one who tries to prove you are innocent, and another (the "Devil's Advocate") who tries to prove you are guilty. By hearing both sides, the judge (the AI) gets a much clearer picture of the truth.

2. The "All-Hands Meeting" (Multi-Source Aggregation)

The old librarians only checked Wikipedia. This new system checks three different libraries at the same time:

Wikipedia: Good for general knowledge.
PubMed: The giant medical library (great for health claims).
Google: The vast, messy internet search.

The Analogy: Instead of asking one person for the weather, you ask a farmer, a meteorologist, and someone who just looked out the window. If all three say "It's raining," you are very confident. If the farmer says "rain" but the meteorologist says "sunshine," you know there is a conflict and you need to be careful.

3. The "Conflict Detector" (Measuring Disagreement)

This is the coolest part. The system doesn't just give you a "True" or "False" answer. It also tells you how much the sources agreed.

High Confidence: If Wikipedia, PubMed, and Google all say "True," the system gives you a high-confidence "True" and shows you a green light.
Low Confidence/Disagreement: If Wikipedia says "True" but Google says "False," the system says, "Well, it's complicated. Here is what Source A thinks, and here is what Source B thinks."

The Analogy: Think of a weather forecast.

Old System: "It will rain." (Period.)
New System: "It will rain. However, the National Weather Service is 90% sure, but the local farmer is only 40% sure because his barometer is acting up. Here is the data from both."

Why Does This Matter?

The researchers tested this system on four different types of tricky topics (science, health, politics, and general news) using five different powerful AI brains (LLMs).

The Results:

Better Accuracy: By looking at both sides of the argument and checking three different libraries, the system got it right much more often than systems that only looked at one side or one library.
More Honest: It doesn't hide the confusion. If the sources disagree, the system admits it. This helps humans make better decisions because they see the full picture, not just a polished summary.

The Bottom Line

This paper teaches us that to find the truth in a messy world, you shouldn't just ask one question from one place. You need to:

Ask the opposite question too.
Ask multiple experts.
Listen to where they agree and where they fight.

By doing this, we build a fact-checking system that is not just smart, but also transparent and trustworthy, showing us the complexity of the real world rather than pretending everything is simple.

1. Problem Statement

The rapid spread of misinformation and disinformation poses significant societal risks, particularly in high-stakes domains like healthcare, finance, and public safety. While automated claim verification (fact-checking) systems exist, they suffer from two critical limitations:

Single-Source Reliance: Most systems depend on a single knowledge source (e.g., Wikipedia), ignoring the wealth of information in secondary sources (e.g., scientific literature, web search) and failing to capture the complexity of real-world information distribution.
Confirmation Bias in Retrieval: Existing methods typically retrieve evidence only to support the original claim, overlooking contradictory evidence. This limits the system's ability to handle nuance, reduces transparency, and fails to account for disagreements between different knowledge bases.

The authors argue that effective verification requires embracing diversity, contradiction, and aggregation across multiple sources to build transparent and reliable systems.

2. Methodology

The proposed system is an Open-Domain Claim Verification (ODCV) framework that integrates Dual-Perspective Retrieval and Multi-Source Aggregation. The architecture consists of three main stages:

A. Dual-Perspective Evidence Retrieval

Instead of querying only the original claim, the system generates a negated version of the claim (e.g., changing "Vitamin B12 deficiency increases homocysteine" to "Vitamin B12 surplus decreases homocysteine").

Process: Both the original claim ( $c$ ) and its negation ( $\bar{c}$ ) are used as queries.
Goal: This ensures the retrieval of both supporting and refuting evidence, creating a balanced "dual-perspective" evidence pool that mitigates confirmation bias.

B. Multi-Source Aggregation

The system retrieves evidence from three distinct knowledge sources, each with different strengths:

Wikipedia: Effective for popular/trending claims.
PubMed: Optimized for specialized, technical, and biomedical claims.
Google: Provides broad web search results and current news.

Pipeline Steps:

Retrieval: Top- $k$ documents are retrieved for both $c$ and $\bar{c}$ from each source.
Selection & Filtering: Sentences are extracted and ranked using SPICED embeddings (cosine similarity) to identify the most relevant sentences.
Deduplication & Merging:
- Evidence sets from the original and negated claims are normalized.
- Overlapping sentences are removed (symmetric difference).
- The remaining sentences are merged and re-ranked to form a unified evidence set ( $E_i$ ) per source.
- Finally, evidence from all three sources is aggregated into a cumulative set.

C. Verification and Disagreement Quantification

LLM Verification: A zero-shot Large Language Model (LLM) is prompted with the claim and the aggregated evidence to predict veracity (e.g., Supported, Refuted, Not Enough Evidence).
Confidence Scoring: The system extracts log-probabilities (logprobs) for the predicted label from the LLM.
Disagreement Analysis: The system compares the confidence scores (logprobs) across the three sources.
- Low dispersion in logprobs indicates consensus.
- High dispersion indicates disagreement or uncertainty.
- These metrics are visualized using Kernel Density Estimation (KDE) to provide interpretability to end-users.

3. Key Contributions

Novel Retrieval Strategy: Introduction of a dual-perspective approach that explicitly retrieves evidence for both the original and negated forms of a claim, capturing a more comprehensive view of the truth.
Multi-Source Aggregation: A systematic framework for aggregating evidence from Wikipedia, PubMed, and Google, demonstrating that combining diverse sources outperforms reliance on a single repository.
Interpretability via Disagreement: A method to quantify and visualize inter-source disagreement using LLM confidence scores (logprobs), moving beyond simple binary verdicts to reveal the "complexity and contentious nature" of knowledge.
Comprehensive Evaluation: Extensive zero-shot evaluation across four benchmark datasets (SciFact, PubHealth, Averitec, LIAR) using five state-of-the-art LLMs (Llama 70B/405B, Phi-4, Qwen 2.5, Mistral).

4. Results

The experiments yielded two robust trends:

Impact of Negated Claims: Augmenting claims with their negated counterparts consistently improved performance.
- Gains: Typical relative improvements of +2–10% in Accuracy and +2–8% in Macro-F1.
- Example: On the SciFact dataset with Llama 70B and Google, accuracy rose from 0.550 to 0.607 (+10.4%) when using the dual-perspective approach.
Impact of Multi-Source Aggregation: Aggregating evidence from all three sources generally outperformed single-source baselines, particularly for weaker sources.
- Example: On SciFact, Llama 70B's merged F1 score exceeded Wikipedia by +41.9% and Google by +10.9%.
- Exception: For the LIAR dataset, Google alone was already strong, and aggregation did not surpass Google's individual performance but still outperformed Wikipedia and PubMed.
Disagreement Visualization: KDE plots of confidence scores showed a clear correlation between source agreement and model certainty. Unanimous agreement across sources resulted in sharp, high-confidence peaks, while partial or no agreement resulted in broader, lower-confidence distributions. This held true across all datasets, though the effect was more pronounced in structured domains (SciFact, PubHealth) than open-domain ones (LIAR).

5. Significance and Conclusion

This paper demonstrates that embracing contradiction is essential for robust fact-checking. By retrieving evidence for both a claim and its negation, and by aggregating data from diverse, independent sources, the system achieves higher accuracy and better interpretability without requiring fine-tuning.

Key Takeaways:

Reliability: Multi-source aggregation provides complementary information that single sources miss, leading to more reliable verdicts.
Transparency: Visualizing source-level disagreement (via logprobs) empowers users to understand why a verdict was reached and where uncertainty lies, rather than presenting a "black box" decision.
Scalability: The zero-shot nature of the approach makes it adaptable to new domains and claims without retraining.

Limitations: The study notes challenges with long-context truncation, the presence of noisy/adversarial claims in open-domain datasets, and the lack of time-aware retrieval (which could lead to outdated evidence). Future work aims to address temporal reasoning, multilingual support, and hallucination mitigation.