DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Review

DeepReviewer 2.0 is a traceable agentic system that generates auditable scientific peer reviews by enforcing an output contract for anchored evidence and executable follow-ups, demonstrating superior performance in major-issue coverage and blind comparisons against both advanced AI models and human committees on ICLR 2025 submissions.

The Gist

Imagine you are a judge on a high-stakes cooking competition. You have to taste 134 different dishes and decide which ones are good enough to win.

The Problem with Current AI Judges:
Most current AI reviewers are like a food critic who writes a very smooth, confident-sounding paragraph: "This dish lacks depth and the seasoning is off."
But here's the catch: They don't tell you which spice is missing, where in the recipe the chef went wrong, or exactly what to change. If you ask them, "Show me the evidence," they just shrug. They sound nice, but they aren't helpful because you can't verify their claims.

The Solution: DeepReviewer 2.0
DeepReviewer 2.0 is like a super-organized, forensic food inspector who doesn't just taste the food—they bring a magnifying glass, a notebook, and a red pen.

Here is how it works, broken down into simple concepts:

1. The "Red Pen" Approach (Traceability)

Instead of writing a generic essay, DeepReviewer 2.0 treats the paper like a map.

• Old Way: "The experiments are weak."
• DeepReviewer Way: "On Page 4, Paragraph 2, you claim the speed improved by 50%. However, Table 3 shows the baseline was actually much lower than you stated. You need to fix the math here."
• The Analogy: It's like a teacher grading a math test who doesn't just write "Wrong" at the top. Instead, they circle the specific number where the student made a mistake and write, "You forgot to carry the one."

2. The "Detective's Notebook" (The Ledger)

Before writing the final review, the system acts like a detective building a case file.

• It creates a Claim-Evidence-Risk Ledger.
• Claim: "This is the first time this has been done."
• Evidence: "I checked 50 other papers. None of them did exactly this."
• Risk: "If I'm wrong about this, the whole paper is a fraud."
• The Analogy: Think of it as a lawyer building a case before going to court. They don't just say "He's guilty"; they list the specific evidence, the witness, and the timeline before they make the final accusation.

3. The "Matched-Setting" Rule (Fair Comparison)

When checking if an idea is truly new, the system is very strict about fairness.

• It won't compare a Ferrari to a bicycle just because they both have wheels.
• It only compares the paper to other research that used the exact same tools, the same dataset, and the same rules.
• The Analogy: Imagine a race. You can't say a runner is the "fastest in the world" if they ran on a flat track while everyone else ran up a mountain. DeepReviewer 2.0 ensures everyone is running on the same track before declaring a winner.

4. The "Safety Gate" (The Export Gate)

This is the most important part. The system has a "Do Not Export" button.

• If the AI tries to write a review but hasn't found enough evidence, or if it can't point to a specific page in the paper, it refuses to send the review.
• It forces itself to be honest. If it doesn't know, it says, "I can't verify this yet," rather than making up a confident-sounding lie.
• The Analogy: It's like a factory quality control robot. If a car comes off the assembly line with a missing wheel, the robot doesn't just paint over it and ship it. It stops the line and says, "This car is not ready. Fix the wheel first."

Why Does This Matter?

The paper tested this system against 134 real scientific papers and compared it to human experts and other AI systems.

• The Result: DeepReviewer 2.0 found more major problems than the other AIs.
• The Human Test: When humans (who are experts in the field) read the reviews, they preferred DeepReviewer 2.0 71% of the time.
• Why? Because the humans could actually use the feedback. They knew exactly what to fix.

The Bottom Line

DeepReviewer 2.0 isn't trying to replace human scientists. It's trying to be the ultimate assistant.

Think of it as a co-pilot for scientists. It does the boring, tedious work of checking facts, finding missing data, and pointing out contradictions. It leaves the final decision (the "verdict") to the human, but it gives the human a clear, evidence-based map to make that decision safely.

In short: It turns "I think this is wrong" into "Here is exactly where it is wrong, here is the proof, and here is how to fix it."

Technical Summary

1. Problem Statement

Current automated peer review systems are primarily evaluated on their ability to generate fluent, coherent prose. However, this approach suffers from a critical failure mode: lack of auditability. Reviewers and Area Chairs cannot verify automated judgments because critiques are often unanchored to specific locations in the manuscript, lack concrete evidence, and do not specify actionable follow-up steps.

• The Gap: Existing systems (e.g., OpenReviewer, ReviewAgents) often treat review generation as a one-shot text generation task. They fail to provide a "traceable review package" where every claim is linked to specific text spans, figures, or tables, and where novelty checks are performed against matched-setting prior work.
• The Goal: To build an automated reviewing system that prioritizes traceability, evidence grounding, and actionability over mere fluency, ensuring that judgments can be audited and repaired.

2. Methodology: The Agentic Cognitive Chain

DeepReviewer 2.0 reframes peer review not as a monologue but as a process-controlled agentic cognitive chain. It enforces a strict "output contract" where the system must produce a structured review package only after meeting minimum traceability and coverage budgets.

The system operates in two distinct stages:

Stage I: Independent Pre-Review (Manuscript-Only)

• Input Parsing: The system parses the PDF into semantic units (paragraphs, captions, equations) paired with stable anchors (page index and line/paragraph spans).
• Claim-Evidence-Risk Ledger: The agent constructs a ledger ($L^{(0)}$) identifying core claims, their supporting evidence within the paper, and associated scientific risks.
• Investigation Agenda: Based on the ledger, the system generates a prioritized list of questions ($Q$) to resolve in Stage II, focusing on high-risk defects (e.g., internal contradictions, missing baselines).
• Page-wise Re-reading: Unlike single-pass generators, the system iteratively selects paper spans to re-read, updating the ledger and annotations to mitigate "lost-in-the-middle" errors.

Stage II: Verification-Oriented Annotation & Synthesis

• Agenda-Driven Retrieval: For each agenda item, the system uses an academic paper-search agent (PASA) to retrieve comparable prior work.
• Matched-Setting Gate: A critical constraint is applied: retrieved papers are only used for novelty checks if they match the same task definition, dataset/benchmark, and primary evaluation metric. This prevents false novelty overlaps due to mismatched settings.
• Conservative Novelty Tagging: Claims are assigned tags: Supported, Partially Overlapping, Substantially Overlapped, or Unclear.
• Anchored Annotations: The system writes critiques as structured units ($A$) containing:
  • Location: Page/paragraph anchor.
  • Issue Category & Severity: (e.g., Major/Minor).
  • Risk Explanation: Why the issue matters scientifically.
  • Repair Action: Concrete, executable instructions for the author (e.g., "Add ablation study," "Clarify metric").
• Export Gate: The system does not export a review package until it meets strict process constraints:
  • Structured report schema compliance.
  • Minimum literature check budget ($\alpha$).
  • Minimum distinct question coverage ($\beta$).
  • Minimum number of anchored annotations ($\gamma$).

3. Key Contributions

1. Traceable Output Contract: The paper introduces a review interface that bundles a global report, anchored annotations (linked to specific PDF regions), a prioritized repair plan, and a novelty assessment. Traceability is a mandatory requirement, not an optional feature.
2. Process-Controlled Workflow: The system enforces a staged cognitive chain with a "claim-evidence-risk" ledger and a matched-setting retrieval gate. This ensures that novelty checks are rigorous and critiques are grounded in specific evidence.
3. Three-Protocol Evaluation: The authors evaluate DeepReviewer 2.0 using:
   • Strict Issue Coverage: Measuring diagnostic recall against human-identified issues.
   • Anonymous System Ranking: Comparing against other automated systems using Bradley-Terry Elo.
   • Blind Comparison vs. Humans: Comparing against a human review committee under the same rubric.

4. Experimental Results

The system was evaluated on 134 ICLR 2025 submissions using an untuned 196B parameter model (Step-3.5-Flash).

• Strict Issue Coverage (RQ1):
  • DeepReviewer 2.0 achieved 37.26% coverage on major issues, significantly outperforming the Gemini-3.1-Pro baseline (23.57%) and other automated systems.
  • It reduced the critical miss rate for major issues to 62.74% (vs. ~76% for baselines).
• Anonymous Ranking (RQ2):
  • In pairwise comparisons against other automatic systems, DeepReviewer 2.0 won 71.63% of non-tied comparisons.
  • It achieved the highest Bradley-Terry Elo score (2057) among all automatic systems, outperforming the next best (Gemini-3.1-Pro at 1915) by a significant margin.
• Blind Comparison vs. Humans (RQ3):
  • When compared anonymously against a human review committee, DeepReviewer 2.0 was preferred in 71.63% of cases.
  • Strengths: It excelled in Constructive Value (84.5% win) and Communication Clarity (86.05% win), demonstrating its ability to provide actionable, localized feedback.
  • Weaknesses: Gains were smaller in Technical Accuracy (59.69%) and Analytical Depth (58.14%), indicating that while process constraints improve organization and guidance, they do not fully solve complex technical reasoning errors.
  • Ethics Gap: The system scored 0% on ethics-sensitive checks, highlighting a critical blind spot for future work.

5. Significance and Positioning

• Shift in Paradigm: DeepReviewer 2.0 moves the field from "generating fluent text" to "generating auditable artifacts." It proves that enforcing process constraints (traceability, evidence linking) significantly improves the utility of automated reviews.
• Assistive Tool, Not a Proxy: The authors explicitly position the system as an assistive tool for reviewers and authors. It is designed to augment human oversight by making judgments checkable and repairs executable, rather than replacing human decision-making.
• Safety and Reliability: By making evidence links explicit and conditioning export on minimum requirements, the system mitigates risks associated with hallucinated critiques and hidden prompt injections in PDFs.
• Future Directions: The paper identifies the need for dedicated objectives to improve ethics-sensitive critique and acknowledges that while the system improves diagnostic recall, it is not yet a substitute for human semantic truth verification.

In summary, DeepReviewer 2.0 demonstrates that process-aware agentic systems can outperform both baseline LLMs and human committees in specific dimensions of review quality (constructiveness, clarity, and traceability), offering a robust framework for the next generation of automated scientific peer review.