Technical Summary: Benchmarking Agentic Review Systems

Problem Statement

The peer review system faces increasing pressure from the rising volume of AI-assisted submissions, a dynamic formalized as a "review death spiral" where overwhelmed reviewer capacity leads to degraded accuracy and random acceptance decisions. While AI-assisted reviewing is proposed as a remedy, the emergence of various agentic review systems (both open-source and proprietary) has created a landscape where it is unclear how these systems compare. Existing benchmarks are often limited in scale, evaluate raw Large Language Models (LLMs) rather than full system pipelines, or rely on outdated models. There is a critical need to evaluate full agentic review systems backed by state-of-the-art models on real research papers to determine their ability to track paper quality, detect errors, and provide value to users.

Methodology

The authors evaluate four review systems across six LLMs (spanning frontier and efficient models) using two primary evaluation axes and a real-world deployment study:

1. Systems and Models

• Systems:
  • Zero-shot: A single prompt asking the model to review the entire paper in one pass.
  • OpenAIReview: An open-source system that processes papers sequentially, maintaining a running summary of definitions, equations, and claims to check passages against context and neighbors.
  • 'coarse: An open-source multi-agent system combining macro-level and per-section agents with editorial deduplication.
  • Reviewer3: A proprietary commercial system emitting high-priority comments (treated as a fixed system).
• Models: The open systems were tested with frontier models (GPT-5.5, Claude Opus 4.7) and efficient models (DeepSeek-V4-Flash, Qwen3.6-35B, Gemini-3.1-Flash-Lite, Grok-4.1-Fast).

2. Evaluation Axis 1: Correlation with Human Quality Signals

• Dataset: Real ICLR/NeurIPS papers (2021–2022) linked to citation counts, acceptance decisions, and reviewer scores.
• Quality Proxies: Papers were categorized into high and low quality based on four proxies: community citations, conference awards/acceptance, mean reviewer scores, and a composite of these.
• Metric: Pairwise accuracy. For randomly sampled (low, high) quality pairs, the system is evaluated on whether it generates more comments for the lower-quality paper.

3. Evaluation Axis 2: Perturbation Benchmark (Error Detection)

• Dataset: 74 clean papers from 8 arXiv subject classes (e.g., Econometrics, Genomics, Machine Learning).
• Method: Controlled errors were injected into papers across four categories:
  1. Surface: Local math edits (sign flips, index changes).
  2. Claim: False theoretical or empirical statements.
  3. Reasoning: Broken logic (circular reasoning, invalid implications).
  4. Experimental: Flawed design (reversed causality, p-hacking).
• Metric: Recall (detection rate). A comment is counted as a detection if it contains the perturbed text and an LLM judge confirms the explanation identifies the specific error.

4. Real-World Deployment

• Setup: OpenAIReview was deployed as a public web tool.
• Data: 1,360 reviews of 1,100 papers.
• Metrics: User voting (likes/dislikes), resolution rates (marking comments as addressed), and qualitative analysis of downvoted comments.

Key Results

1. Correlation with Quality

• All systems performed above chance in distinguishing low-quality from high-quality papers based on comment volume.
• Best Performance: OpenAIReview backed by GPT-5.5 achieved the highest pairwise accuracy at 83.0%.
• Efficient Models: Even efficient models (e.g., Grok-4.1-Fast) performed above chance (80%), though with fewer total comments.
• Observation: Systems were not explicitly trained to predict acceptance; they surfaced issues as a byproduct of their design, yet still tracked quality signals effectively.

2. Error Detection (Perturbation Benchmark)

• Best Performance: OpenAIReview + GPT-5.5 detected 71.6% of injected errors.
• Error Types: The system showed significant gains over zero-shot baselines on prose-level errors (claims, reasoning, experimental design), where recall increased by 10–18 percentage points. Gains on local math-token edits were smaller.
• Complementarity: Different backend models detected different subsets of errors. The union of detections across all six models under OpenAIReview reached 83.3% recall, significantly outperforming any single model.
• System Comparison: 'coarse and Reviewer3 underperformed zero-shot baselines in raw recall, likely because they optimize for editorial prioritization (fewer, high-priority comments) rather than exhaustive enumeration.

3. Human vs. AI Complementarity

• Overlap: AI systems and human reviewers flagged largely disjoint sets of paragraphs (Jaccard similarity of 0.230).
• Focus: Humans concentrated on paper-level critiques (novelty, scope, motivation), while AI systems over-indexed on surface-level details, claim validity, experimental design, and formal math.
• Conclusion: AI systems appear positioned to serve as a complement to humans, handling exhaustive technical enumeration while humans focus on high-level judgment.

4. User Feedback

• Sentiment: Users voted positively on comments, with a 1.44:1 ratio of likes to dislikes.
• Action: 4.9% of comments were marked as resolved.
• Weaknesses: The primary complaints (71% of downvotes) were "unhelpful" comments, specifically false positives, trivial nitpicks, and unreasonable demands for detail. This indicates a precision problem rather than a recall problem.

Significance and Claims

The paper claims that while AI review systems are not yet perfect, they have reached a stage where they are practically useful:

1. Quality Tracking: Current systems can track human quality judgments well without explicit training on acceptance decisions.
2. Error Detection: Strong configurations can catch the majority of injected errors, with significant room for improvement through ensemble methods (combining models).
3. Practical Value: Real-world users find the comments actionable and valuable, though the systems currently struggle with precision (false positives).
4. Future Direction: The results suggest that "harness design" (how models are orchestrated) is critical. Combining complementary models can significantly boost performance, and systems should be tailored to specific audiences (e.g., authors vs. area chairs) to mitigate precision issues.

The authors conclude that AI review systems are a viable component of the peer review ecosystem, capable of handling the "exhaustive enumeration" of technical issues, but they require better calibration to reduce noise and false positives before fully replacing or significantly augmenting human review capacity.