🤖 AI

When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews

This paper introduces RevCI, a new benchmark with evidence-level annotations and graded intensity labels for scientific peer review contradictions, alongside IMPACT, a multi-agent framework, and its distilled model TIDE, which together outperform existing baselines in identifying and evaluating the severity of reviewer disagreements.

Original authors: Sandeep Kumar, Yash Kamdar, Abid Hossain, Bharti Kumari, Tanik Saikh, Asif Ekbal

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sandeep Kumar, Yash Kamdar, Abid Hossain, Bharti Kumari, Tanik Saikh, Asif Ekbal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the editor of a massive, high-stakes science fair. Thousands of researchers submit their projects, and you hire hundreds of expert judges to review them. The problem? These judges often disagree. One might say, "This is a brilliant, groundbreaking idea!" while another says, "This is a confused mess with no new insights."

Usually, when judges disagree, they just give a "Yes" or "No" to the project. But in the real world, it's rarely that simple. Sometimes the disagreement is a tiny whisper of doubt; other times, it's a screaming match about the core of the science.

This paper introduces a new way to handle these disagreements, treating them not as a simple "Yes/No" switch, but as a complex conversation that needs to be understood, measured, and resolved.

Here is the breakdown of their solution, using simple analogies:

1. The Problem: The "Binary" Trap

Previous methods tried to find disagreements by looking at two sentences at a time and asking, "Do these contradict?" It's like a referee blowing a whistle every time two players bump shoulders.

The Flaw: This misses the big picture. It doesn't tell you how bad the disagreement is. Is it a minor difference in opinion (a gentle tap), or a fundamental clash of values (a full-on collision)? The old methods treated both the same.

2. The New Tool: "RevCI" (The Scorecard)

The authors created a new dataset called RevCI. Think of this as a massive, expert-annotated library of "Judge vs. Judge" arguments.

What's special: Instead of just marking "Disagreement," human experts went through and labeled the intensity of the fight.
- Level 1 (Low): "I think this is vague," vs. "It's pretty clear." (A mild difference).
- Level 2 (Medium): "The math is shaky," vs. "The math is solid." (A clear conflict).
- Level 3 (High): "This is the best paper ever," vs. "This is garbage." (A total explosion).
They also marked exactly which sentences were fighting each other, like highlighting the specific words in a legal brief.

3. The Brain: "IMPACT" (The Panel of Judges)

To solve these problems, they built a system called IMPACT. Imagine a high-tech courtroom where the "judge" isn't one person, but a team of AI agents working together.

The Detective (ACEA): First, an agent scans the reviews to find the specific sentences that are fighting. It's like a detective finding the smoking guns in a messy room.
The Debaters (DIA Agents): Two AI agents take the "smoking guns" and argue about how serious the fight is.
- Crucial Rule: They are told not to just agree to end the argument quickly. They must stick to their initial opinion and defend it with evidence. This forces them to dig deeper and find the real nuance, rather than just saying "Okay, let's call it a draw."
The Referee (Adjudication Agent): After the debaters have argued their points, a third agent (the Referee) listens to the whole debate and makes the final call on the intensity score.
The Bouncer (Validity Gate): Finally, a gatekeeper checks: "Is this actually a fight, or just two people talking about different things?" If it's not a real contradiction, it gets tossed out.

The Result: This multi-agent "courtroom" is much better at understanding the severity of the disagreement than a single AI or a simple sentence-matcher.

4. The Speedster: "TIDE" (The Intern)

The "IMPACT" courtroom is very smart, but it's slow and expensive because it requires a whole team of AIs to debate for every single review. It's like hiring a panel of 10 professors to grade one essay.

To fix this, they created TIDE.

The Analogy: Imagine the "IMPACT" team (the professors) spends hours debating and writing detailed notes on how to grade a paper. They then take those notes and teach a very smart, fast intern (TIDE) how to do the same job in seconds.
The Magic: TIDE is a smaller, faster AI model. It doesn't need to hold a debate; it just looks at the reviews and instantly predicts the contradiction and the intensity score, having "learned" from the expensive debates of the IMPACT team.
The Benefit: TIDE is almost as accurate as the big team but runs much faster and costs much less.

Summary

The paper argues that scientific peer review is too complex for simple "Yes/No" checks.

They built a dataset (RevCI) that measures how bad the disagreements are.
They built a smart system (IMPACT) that uses a "debate" between AI agents to figure out the severity of those disagreements.
They distilled that smart system into a fast, lightweight model (TIDE) that can do the same job quickly, making it practical for real-world use.

The goal isn't to replace human editors, but to give them a tool that highlights exactly where and how strongly experts disagree, so they can make better decisions.

Technical Summary: When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews

Problem Statement
Scientific peer review is the cornerstone of academic publishing but faces increasing scrutiny due to systemic fragilities, including lack of transparency, inherent biases, and inconsistency in evaluations. A critical challenge for Area Chairs and editors is reconciling conflicting expert judgments, particularly as submission volumes surge (e.g., over 13,000 reviewers for EMNLP 2025). Existing computational approaches, such as ContraSciView, typically frame reviewer disagreement as a binary contradiction detection task over isolated sentence pairs. This formulation abstracts away the full review-level context and collapses graded differences in evaluative conflict into coarse binary labels. Consequently, these methods fail to capture the severity of disagreements or the nuanced, discourse-level reasoning required to resolve conflicts, limiting their utility for editorial decision-making.

Methodology
The authors propose a fine-grained formulation of reviewer contradiction analysis that operates over full peer reviews. The methodology consists of three primary components:

RevCI Dataset: The authors introduce RevCI (Review Contradiction Intensity), an expert-annotated benchmark derived from the ASAP-Review corpus (ICLR and NeurIPS reviews). Unlike prior datasets, RevCI features:
- Evidence-level annotations: Identifying specific spans of text that constitute contradictions.
- Graded intensity labels: Assigning severity scores (1: Low/Implicit, 2: Moderate/Explicit, 3: High/Direct) rather than binary labels.
- Aspect conditioning: Categorizing contradictions by review aspects (e.g., Novelty, Clarity, Soundness).
- Human-written explanations: Rationales for the assigned intensity.
- The dataset contains 800 review pairs, filtered via an LLM-based screening process to identify potential contradictions before expert annotation.
IMPACT Framework: The authors propose IMPACT (Intensity-based Multi-Agent Contradiction estimation), a structured multi-agent framework designed to jointly identify contradiction evidence and estimate disagreement intensity. The framework operates through a deliberative reasoning protocol:
- Aspect-Conditioned Evidence Agent (ACEA): Retrieves candidate evidence spans conditioned on specific review aspects to ensure high recall.
- Deliberative Intensity Agents (DIA-A/B): Two agents independently assign intensity scores and generate reasoning based on the full review context.
- Disagreement Orchestrator: If agents disagree on intensity, this component enforces a "score-locking" constraint, preventing agents from drifting toward lazy consensus. Instead, they engage in structured debate, grounding arguments in evidence and addressing counter-arguments.
- Adjudication Agent: Resolves the debate by selecting the final intensity score and producing a consolidated reasoning trace.
- Contradiction Validity Gate (CVG): Filters out evidence pairs assigned a null label (0), ensuring only valid contradictions are retained.
TIDE Model: To address the computational cost of the iterative multi-agent process, the authors distill IMPACT into TIDE, a Small Language Model (SLM). TIDE is trained via supervised fine-tuning (using LoRA) on the synthetic data generated by IMPACT's deliberation traces. This allows TIDE to predict contradiction evidence, intensity scores, and reasoning in a single forward pass, significantly reducing inference latency while maintaining performance.

Key Contributions

Task Formulation: A shift from binary contradiction detection to fine-grained analysis involving evidence extraction, aspect conditioning, and graded intensity scoring.
RevCI Benchmark: A new dataset with expert-annotated evidence spans, graded intensity labels, and human-written rationales, enabling fine-grained evaluation of reviewer disagreement.
IMPACT Framework: A novel multi-agent architecture that integrates aspect-conditioned extraction with deliberative debate and adjudication to model contradiction intensity, outperforming single-agent and generic multi-agent baselines.
TIDE Model: A distilled SLM that achieves competitive performance with human annotations at a fraction of the inference cost, demonstrating that fine-grained intensity reasoning can be effectively transferred to smaller models.

Experimental Results
Experiments were conducted on the RevCI test set, comparing IMPACT (both proprietary and open-access variants) and TIDE against strong baselines including single-agent models (CoT prompting), generic multi-agent frameworks (e.g., ChatEval, CourtEval), and the prior system ContraSciView.

Performance: IMPACT substantially outperforms all baselines. The proprietary variant (IMPACT-P) reduced average detection error (FNR and FPR) by 31.2% and improved average agreement scores ( $\kappa$ , $\rho$ , $\tau$ ) by 52.0% compared to the strongest baseline, CourtEval. The open-access variant (IMPACT-OA) also showed significant gains, indicating that the framework design, rather than model scale, drives performance.
Efficiency: TIDE achieves agreement with human annotations comparable to much larger models (e.g., GPT-5.2) but with significantly lower inference costs. Token efficiency analysis shows TIDE uses substantially fewer tokens per sample compared to multi-agent baselines.
Ablation Studies:
- Deliberation Depth: Increasing discussion rounds in IMPACT improves agreement up to four rounds, after which gains saturate.
- Component Impact: Aspect conditioning significantly reduces False Negative Rates (FNR). The inclusion of intensity examples and reasoning in TIDE training improves alignment with human judgments. The Adjudication and Disagreement Orchestrator components are crucial for reducing False Positive Rates (FPR).

Significance and Claims
The paper claims that reviewer disagreement is inherently fine-grained and context-dependent, rendering binary detection over isolated sentences insufficient for the needs of Area Chairs and editors. The authors assert that their structured multi-agent approach, driven by deliberative reasoning rather than simple prompting, successfully captures the severity and nuance of conflicts. They further demonstrate that this complex reasoning can be distilled into an efficient small language model (TIDE) without sacrificing alignment with human judgment. The work provides a practical tool to assist editors in identifying and interpreting conflicting reviews, potentially improving the fairness and consistency of the peer review process, while explicitly noting that the system is an assistive tool requiring human oversight and is not designed to automate editorial decisions.