💬 NLP

ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence

This paper introduces ConflictScore, a novel metric and benchmark designed to quantify how well language models acknowledge conflicting evidence in their grounding documents by decomposing responses into atomic claims and measuring both the proportion of conflicting claims and the balance between supporting and contradicting evidence.

Original authors: Siyi Liu, Aaron Halfaker, Dan Roth, Patrick Xia

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyi Liu, Aaron Halfaker, Dan Roth, Patrick Xia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of one witness, you have a room full of ten people giving you testimony. Some say the suspect was at the park; others swear they saw him at the library.

The Problem: The "Yes, But..." Blind Spot
Current AI models (like the chatbots you might use) are great at finding information. But when they find conflicting stories, they often act like a stubborn child who picks the loudest voice and ignores the rest. They might say, "The suspect was at the park!" and stop there, completely forgetting that five other witnesses said otherwise.

Existing tools for checking if an AI is telling the truth are like a simple "Pass/Fail" test. They ask: "Did any witness support this story?" If one person said "Park," the AI gets a passing grade, even if the other nine said "Library." This gives a false sense of security.

The Solution: ConflictScore
The authors of this paper introduce a new tool called ConflictScore. Think of it as a "Conflict Detective" that doesn't just ask if a story is supported, but asks: "Is this story being argued against by other witnesses?"

Here is how it works, broken down into three simple steps:

Breaking it Down (Claim Decomposition):
Imagine the AI writes a long paragraph. ConflictScore breaks this paragraph into tiny, individual sentences (atomic claims). It's like taking a big cake and slicing it into individual pieces to inspect each one.
The Witness Stand (Evidence Evaluation):
For every single sentence, the tool checks it against every single document the AI used. It asks: "Does Document A support this sentence? Does Document B contradict it?"
- Support: The document agrees.
- Contradict: The document disagrees.
- Irrelevant: The document doesn't talk about it.
The Scorecard (The Metrics):
The tool gives two scores:
- ConflictScore-Count (CS-C): This is like a "Trouble Count." It tells you what percentage of the AI's sentences are fighting with the evidence. If 4 out of 6 sentences have witnesses arguing against them, the score is high (meaning the AI is in trouble).
- ConflictScore-Ratio (CS-R): This is a "Balance Scale." It measures how the evidence is split. Is it a 50/50 split (a real debate), or is it 9 vs. 1 (a landslide)? This helps understand how severe the disagreement is.

The Test Drive: ConflictBench
To see if their new detective tool works, the authors built a gym called ConflictBench. They gathered thousands of questions where the answers were messy:

Ambiguity: Questions like "How big is Cleveland?" (There are many cities named Cleveland).
Contradiction: Documents that explicitly say opposite things (e.g., "The event was in 1990" vs. "The event was in 1995").
Opinion: Questions where people genuinely disagree (e.g., "Is this web design technique good?").

They tested their tool on this gym and found it was very good at spotting when the AI was ignoring the arguing witnesses.

What They Found

AI is Overconfident: Even when the documents clearly disagreed, the AI models tended to pick one side and act like it was the absolute truth. They often ignored the "library" witnesses while shouting about the "park."
Talking Doesn't Fix It: The authors tried giving the AI "gentle reminders" in its instructions (like "Please be careful and consider all sides"). This helped a little bit, but the AI still often got it wrong. It's like telling a stubborn dog to "be nice," but it still bites the mailman.
The "Do-Over" Trick: The most exciting finding was in the TruthfulQA experiment. When they used ConflictScore to tell the AI, "Hey, you missed a contradiction, try again," the AI actually got better. It was like a student getting a red pen correction on a test, realizing their mistake, and rewriting the answer correctly.

The Bottom Line
ConflictScore is a new way to measure if an AI is being honest about the messiness of the real world. It doesn't just check if the AI has some proof; it checks if the AI is brave enough to admit when the proof is messy and conflicting.

Limitations (The Catch)
The authors admit this tool takes a lot of computer power to run because it has to check every sentence against every document. It's like hiring a team of 100 lawyers to review a single paragraph—it's accurate, but it's expensive and slow. Also, the tool treats all documents as equal; it doesn't know if one document is a famous newspaper and another is a random blog post. It just sees "Document A" vs. "Document B."

In short, ConflictScore helps us see when an AI is confidently wrong because it ignored the arguments right in front of its nose.

Technical Summary: ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence

Problem Statement

Large Language Models (LLMs) are increasingly deployed in retrieval-augmented generation (RAG) settings for tasks such as question answering and report generation, where they must synthesize information from multiple sources. However, these sources frequently contain conflicting evidence due to ambiguity, divergent opinions, or contradictory facts. Existing metrics for evaluating factuality and faithfulness (e.g., FactScore, SAFE) typically treat the entire set of grounding documents as a single, unified reference corpus. Consequently, if at least one document supports a claim, the metric often labels the claim as supported, overlooking the presence of other documents that contradict it. This global framing fails to capture the nuance of inter-evidence disagreement, leading to the evaluation of overconfident or misleading responses that ignore conflicting perspectives.

Methodology

The authors introduce ConflictScore, a novel metric designed to quantify how well a model's response acknowledges and navigates conflicting evidence within its grounding documents. Unlike prior approaches that collapse the evidence set, ConflictScore operates on a per-document basis. The framework consists of three stages:

Claim Decomposition: The model's response is decomposed into a set of atomic, minimal factual statements (claims).
Evidence Evaluation: Each claim is independently evaluated against every grounding document. A document is labeled as SUPPORT, CONTRADICT, or IRRELEVANT relative to the claim. Crucially, a document is considered to support a claim even if it only partially supports it, encouraging nuanced reasoning. A claim is deemed to have "conflicting evidence" if the set of supporting documents ( $D^+$ ) and the set of contradicting documents ( $D^-$ ) are both non-empty.
Metric Calculation: The framework aggregates these labels into two complementary measures:
- ConflictScore-Count (CS-C): The proportion of claims in a response that exhibit conflicts (i.e., have both supporting and contradicting evidence). Lower scores indicate fewer contentious claims.
- ConflictScore-Ratio (CS-R): The balance between supporting and contradicting evidence. For each conflicted claim, it calculates the ratio $|D^-| / (|D^+| + |D^-|)$ and averages this across all claims. This captures the severity of the disagreement (e.g., a 1:1 split vs. a 9:1 imbalance).

To evaluate this metric, the authors constructed ConflictBench, a benchmark aggregating and transforming existing datasets (ContraQA, MacNoise, AmbigDocs, ConflictingQA) to cover diverse conflict types, including counterfactuals, ambiguity, and divergent opinions.

Key Contributions

The paper makes three primary contributions:

ConflictScore Metric: A robust, conflict-aware extension of atomic-claim factuality pipelines that explicitly measures how responses handle simultaneous support and contradiction from grounding documents.
ConflictBench: A systematic benchmark designed to evaluate conflict detection and calibration across diverse conflict types, filling a gap in existing evaluation protocols.
Diagnostic and Corrective Utility: The demonstration that ConflictScore serves not only as a diagnostic tool for identifying overconfident claims but also as a corrective signal. By feeding conflict signals back to the model, the authors show improvements in response truthfulness.

Results

Conflict Detection Performance:
Experiments on ConflictBench using frontier models (e.g., Gemini-3.5-Flash, GPT-5.4-Mini) show that the underlying conflict detection subroutine achieves high precision (0.98) and recall (0.90), with an overall F1 score of 0.94. This indicates the metric is well-calibrated to detect conflicts across different datasets and model families.

Benchmarking LLMs:
When applied to state-of-the-art LLMs (GPT-5.4-Mini, GPT-5.4-Nano, Qwen3-32B) in a report-generation task with contradictory evidence, the results reveal a significant gap in conflict awareness. Despite explicit prompting strategies encouraging "balanced" or "super-balanced" responses, models frequently commit to a single stance. CS-C scores remained consistently high (around 0.50–0.55), indicating that even with instructions to hedge, models often fail to acknowledge the full scope of conflicting evidence.

Corrective Feedback (TruthfulQA):
In a case study on TruthfulQA, the authors tested whether feeding conflict signals back to the model could improve truthfulness. Using a "Regenerated-RAG" setting where the model regenerates its answer after being informed of detected conflicts, they observed consistent improvements over standard RAG and prompt-based control settings.

For models like gemini3.1-flash-lite, the regeneration process corrected up to 74% of originally wrong answers while introducing very few new errors (harm rate < 3%).
This suggests that conflict signals can effectively guide models toward more cautious and accurate reasoning, particularly in multiple-choice settings.

Significance and Limitations

The paper argues that ConflictScore provides a necessary diagnostic for the "overconfidence" problem in RAG systems, where models synthesize contradictory sources into definitive, potentially misleading answers. By quantifying the tension between supporting and contradicting evidence, it offers a pathway to more truthful and nuanced AI responses.

The authors acknowledge several limitations:

Computational Cost: The quadratic complexity of evaluating every claim against every document can be expensive for long-form outputs, though lightweight variants (e.g., sampling salient claims) are proposed.
Source Reliability: The current framework treats all grounding documents equally, without weighting them by source trustworthiness. While the framework is compatible with future reliability-aware extensions, it does not inherently distinguish between conflicts among credible sources and those driven by low-quality evidence.
Upstream Dependency: The metric relies on the accuracy of upstream claim decomposition and evidence labeling. Errors in these steps can propagate to the final score, and the choice of claim granularity can influence the evaluation.

In summary, the paper establishes that current LLMs struggle to naturally handle conflicting evidence and that explicit conflict detection and feedback mechanisms are effective tools for improving the truthfulness and reliability of generated content.