💬 NLP

Explanation Generation for Contradiction Reconciliation with LLMs

This paper introduces the task of reconciliatory explanation generation, where models must hypothesize explanations to render contradictory statements compatible, and evaluates 18 large language models to reveal their current limitations in this under-explored aspect of human-like reasoning.

Original authors: Jason Chan, Zhixue Zhao, Robert Gaizauskas

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jason Chan, Zhixue Zhao, Robert Gaizauskas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for clues to prove someone guilty, your job is to prove that two seemingly impossible stories can actually both be true at the same time.

That is essentially what this paper is about. The researchers are teaching AI (Large Language Models) a very human skill: how to stop arguing and start making sense of contradictions.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Either/Or" Trap

Usually, when an AI sees two sentences that clash, it acts like a rigid judge.

Statement A: "Cassie hates coffee."
Statement B: "Cassie buys coffee every single day."

A standard AI says: "Error! These contradict. One of them must be a lie. I will delete Statement B."

But in real life, humans don't just delete facts. We get curious. We ask, "Wait a minute... maybe Cassie hates coffee, but she buys it every day because she's the office manager and has to buy it for her 50 coworkers?"

This is called reconciling a contradiction. It's the art of finding a hidden context that makes both statements true. The paper argues that while AI is getting smarter, it's still terrible at this specific type of "detective work."

2. The New Game: "Reconciliatory Explanation Generation"

The authors created a new game for AI. The rules are:

Give the AI two contradictory sentences.
Ask the AI to write a short story (an explanation) that connects them.
If the story makes the contradiction disappear, the AI wins.

The Analogy: Imagine you are playing a game of "Connect the Dots," but the dots are far apart and look like they belong to different pictures. The AI has to draw a line that shows how they are actually part of the same picture.

3. How They Tested It (The "Magic Mirror" Method)

Testing this is tricky. Usually, you need a human to say, "Yes, that explanation is good." But humans are slow and sometimes disagree with each other.

So, the researchers used a clever trick:

They took old datasets where humans had already argued about whether sentences contradicted each other.
They used AI judges (other AI models) to grade the answers.
The Logic: If an AI judge originally thought "A" and "B" were enemies, but after reading the new explanation, it changes its mind and says, "Oh! Now I see how they are friends," then the explanation was a success.

It's like having a panel of critics watch a magic trick. If they were convinced the magician was cheating, but then he explains the secret, and they say, "Ah, that makes sense!", the trick worked.

4. What They Found (The Surprising Results)

The researchers tested 18 different AI models, from tiny ones to massive ones. Here is what happened:

The "Thinking" Paradox: You might think that if you tell an AI to "think harder" or "take a moment to reason" before answering, it would get better.
- The Result: For medium-sized models, yes, thinking helped. But for the biggest, smartest models? Thinking actually made them worse at this specific task.
- The Metaphor: It's like asking a genius to write a poem. If they overthink every word, they might freeze up and write something boring. Sometimes, the "flow" is better than the "struggle."
The "Restating" Cheat: Some small, cheap AI models tried to cheat. When asked to explain why "Cassie hates coffee" and "buys coffee" fit together, they just wrote: "She buys coffee because she buys coffee."
- The AI judges were fooled into thinking this was a good explanation because it technically didn't contradict anything, but it was useless. It was like a student copying the question as the answer.
The Gap: The most expensive, proprietary AI models (like the ones from OpenAI) were significantly better at this than the open-source ones. They were better at finding those "hidden contexts" (like the office manager story).

5. Why Does This Matter?

This isn't just a logic puzzle. This skill is crucial for the future of AI in the real world.

In Chatbots: If a user says, "I love my job, but I'm quitting," a bad bot says, "You are confused." A good bot asks, "Are you quitting because you're moving to a new city?"
In Science: Scientists often find data that contradicts their theories. Instead of throwing the data away, they use it to create new theories (like how light can be both a particle and a wave). AI needs to learn this "reconciling" skill to help us discover new things.
In Law: Lawyers often have to read two laws that seem to clash and find a way to interpret them so they work together.

The Bottom Line

The paper concludes that while AI is getting very good at detecting lies and contradictions, it is still struggling to resolve them with creativity. It's like a robot that is great at spotting a broken bridge but hasn't learned how to build a detour around it yet.

The researchers hope that by teaching AI to "reconcile" rather than just "reject," we can build smarter assistants that understand the messy, complicated, and often contradictory nature of human life.

1. Problem Definition

Current Natural Language Processing (NLP) research predominantly treats contradictions as errors to be resolved by arbitrating which statement is true and which is false (arbitration). However, human reasoning in social, professional, and scientific contexts often relies on abductive reasoning to generate explanations that reconcile seemingly contradictory statements, rendering them compatible rather than discarding one.

The paper identifies a gap in Large Language Model (LLM) capabilities: while models are increasingly used as conversational agents and scientific aids, their ability to hypothesize reconciliatory explanations for contradictions remains largely unexplored. Existing work focuses on detecting conflicts or choosing a "winner," but rarely on generating the context that makes both statements true simultaneously.

2. Methodology

Task Formulation: Reconciliatory Explanation Generation (REG)

The authors define a new task where a model ( $M_{expl}$ ) is given a Premise ( $P$ ) and a Hypothesis ( $H$ ) that a judge model initially labels as a Contradiction. The goal is to generate an Explanation ( $E$ ) such that:

Effectiveness: When $E$ is combined with $P$ , the resulting context entails $H$ (i.e., $P + E \models H$ ).
Coherence: $E$ does not contradict $P$ (i.e., $P$ and $E$ are consistent).

Data Curation

Instead of creating a new dataset from scratch, the authors propose a novel framework to repurpose existing Natural Language Inference (NLI) datasets:

Source: They utilize the ChaosNLI-MNLI subset (derived from MultiNLI), which contains premise-hypothesis pairs annotated by 100 human annotators.
Selection Strategy: They select instances where the majority label is "Contradiction" but acknowledge that human annotators often disagree (no instance was unanimously labeled as a contradiction). This inherent disagreement is treated as a feature, representing the "latent context" needed for reconciliation.
Dataset: The final dataset (ChaosNLI-MNLI-C) consists of 275 instances where the mean proportion of annotators labeling it as a contradiction is 0.62.

Evaluation Framework

The paper introduces a scalable, automatic evaluation method using LLM-as-a-Judge:

Judges: Multiple NLI models (selected from the 18 tested models based on their accuracy in predicting the majority label of the dataset) act as judges.
Metrics: For each generated explanation, judges evaluate three relationships:
1. Does $P + E$ entail $H$ ? (Effectiveness)
2. Does $P$ contradict $E$ ? (Coherence, checked bidirectionally)
3. Success Rate: The percentage of explanations that satisfy both effectiveness and coherence.
Advantage: This approach avoids the noise and subjectivity of human annotation for evaluation, relying on consistent, reproducible model judgments.

Experimental Setup

Models: 18 LLMs were evaluated, including Qwen3 (various sizes and "thinking" modes), Llama-3.1-Tulu-3, Olmo-3, DeepSeek-R1, and proprietary OpenAI models (gpt-5-mini, gpt-5.2).
Modes: Models were tested in both "Non-thinking" (direct output) and "Thinking" (Chain-of-Thought) modes to assess the impact of extended reasoning.

3. Key Results

Overall Performance: Most models struggle with the REG task. The best-performing open-weight model (Llama-3.3-70B-Instruct) achieved a success rate of 26.20%, while proprietary models performed significantly better (gpt-5-mini: 40.25%, gpt-5.2: 31.56%).
Coherence vs. Effectiveness: Models generally find it easier to generate coherent explanations (not contradicting the premise) than effective ones (actually entailing the hypothesis).
The "Thinking" Paradox:
- Coherence: Enabling "thinking" modes generally decreased coherence scores. Models often acknowledged contradictions in their thought traces but still generated explanations that failed to resolve them.
- Effectiveness: For mid-sized models (e.g., Qwen3-4B, 8B), "thinking" significantly improved effectiveness. However, for larger models (e.g., Qwen3-32B), the benefit of "thinking" plateaued or even diminished.
- Small Models: For the smallest models (0.6B, 1.7B), "thinking" reduced effectiveness, suggesting that forced reasoning chains can destabilize models with fewer parameters.
Human Correlation: A weak but statistically significant negative correlation (-0.2257) was found between the proportion of humans labeling a pair as a contradiction and the model's success rate. This suggests that if humans find a contradiction hard to resolve, models also struggle.
Failure Modes: Qualitative analysis revealed models often resort to:
- Restating the hypothesis without adding new context.
- Misinterpreting linguistic nuances (e.g., treating a metaphorical "dog" as a literal animal).
- Generating nonsensical scenarios (e.g., announcing dates after an event has passed).

4. Key Contributions

New Task Definition: Introduced Reconciliatory Explanation Generation (REG), shifting the focus from arbitration to abductive reasoning for contradiction resolution.
Scalable Data & Evaluation: Proposed a method to curate REG data from existing NLI datasets by leveraging annotator disagreement, and introduced automatic metrics (Effectiveness/Coherence) using LLM judges, eliminating the need for costly human evaluation.
Empirical Insights: Provided extensive benchmarks across 18 models, revealing that current LLMs have limited capacity for this specific reasoning task and that "thinking" does not linearly improve performance as model size increases.

5. Significance and Implications

Beyond Arbitration: The work highlights a critical limitation in current LLMs: they are good at deciding "who is right" but poor at "how both can be right." This is essential for applications like chatbots (handling user confusion), scientific discovery (resolving paradoxical observations), and legal reasoning (harmonious reading of statutes).
Reasoning Limits: The finding that "thinking" modes plateau in effectiveness for larger models suggests that simply increasing compute or chain-of-thought length is insufficient. Future research must focus on the structure of reasoning and how models retrieve and integrate latent knowledge to generate novel hypotheses.
Future Directions: The authors call for future work to expand this framework to document-level contradictions, diverse languages, and to incorporate criteria for factual grounding and diversity of hypotheses.