Technical Summary: PRISM – A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers

Problem Statement

The scientific peer-review system is facing unprecedented strain due to the exponential growth in submissions to top-tier machine learning venues (e.g., NeurIPS submissions surged from 15,671 in 2024 to 21,575 in 2025). This volume has strained the reviewer pool, complicated paper-to-reviewer matching, and degraded review quality and consistency. In response, Large Language Models (LLMs) have rapidly evolved from proofreading aids to autonomous reviewer agents, with estimates suggesting 17–21% of reviews at recent top-tier venues now involve LLM assistance.

However, a critical gap remains in understanding the actual efficacy of these systems. It is unclear whether LLMs can match human reviewers in identifying scientific gaps, diagnosing methodological flaws, or providing actionable feedback. Existing evaluation methods are insufficient: surface-level metrics like ROUGE and BLEU reward linguistic fluency over scientific rigor, while "LLM-as-a-judge" approaches often conflate verbosity with quality and suffer from inherent biases. There is a need for a structured, multi-dimensional framework that moves beyond aggregate scores to evaluate the specific cognitive competencies required for scientific peer review.

Methodology: The PRISM Framework

The authors introduce PRISM (Peer Review Intelligence via Structured Multi-dimensional assessment), a benchmarking framework designed to evaluate review quality across four distinct, independent dimensions. Unlike holistic scoring, PRISM decomposes evaluation into structured evidence-extraction tasks using dedicated pipelines grounded in argument mining, retrieval-augmented verification, and consensus-based scoring.

The framework processes peer review text and manuscript content through four modular pipelines:

1. Depth of Analysis (DoA):

   • Goal: Determine if reviewers substantiate claims with objective, well-grounded premises rather than generic assertions.
   • Pipeline: Extracts Argumentative Discourse Units (ADUs) and classifies them as Claims or Premises. Premises are scored on a grounding scale: Level 0 (Vague), Level 1 (Internal to manuscript), or Level 2 (External to literature).
   • Metric: A harmonic mean of the Premise Ratio (evidence coverage) and Normalized Average Grounding Score (evidence quality).

2. Novelty Assessment:

   • Goal: Verify if novelty judgments are grounded in retrievable prior literature or rely on unverified assertions.
   • Pipeline: Extracts novelty claims from reviews and constructs deterministic queries to retrieve prior work (via Semantic Scholar). An LLM judge verifies each claim against the paper context and retrieved literature, assigning a support score from -2 (contradicted) to +2 (supported).
   • Metric: A relevance-weighted aggregation of claim scores, distinguishing well-grounded critiques from hallucinations.

3. Flaw Identification & Major Issues Prioritization:

   • Goal: Measure the accuracy of detecting genuine methodological weaknesses and the ability to prioritize critical issues over minor formatting errors.
   • Pipeline: Merges findings from human and LLM reviewers to establish a "consensus ground truth" of valid flaws, classified as Critical (methodological) or Minor (editorial).
   • Metric: Recall for Critical and Minor flaws, and a Critique Prioritization Score (nCPS) based on NDCG, measuring whether critical flaws appear earlier in the review than minor ones.

4. Multi-dimensional Constructiveness:

   • Goal: Quantify the helpfulness of feedback in terms of actionability, specificity, justification, solution orientation, and tone.
   • Pipeline: Decomposes reviews into Atomic Review Comments (ARCs) and scores each on five dimensions (0–2 scale).
   • Metric: Mean Constructiveness Score (MCS), averaging normalized scores across all ARCs.

Experimental Setup:
PRISM was applied to a stratified corpus of 1,000 manuscripts from ICLR (2024–2026), ICML (2025), and NeurIPS (2025). The study benchmarked five leading automated reviewer systems (TreeReview, Reviewer2, SEA-E, DeepReview, CycleReviewer) against human reviewers. All evaluations were performed using Gemini 2.5 Flash Lite as the judge model.

Key Results

The evaluation reveals that no single LLM system consistently matches the balanced performance of human reviewers across all dimensions. Instead, each system exhibits a distinct specialization profile with characteristic blind spots:

• Depth of Analysis: CycleReviewer and DeepReview matched human analytical depth, primarily by generating a high volume of substantiated premises. In contrast, TreeReview fell into a "surface-level trap," over-indexing on presentation anomalies (clarity) at the expense of methodological rigor.
• Novelty Assessment: SEA-E outperformed human reviewers in grounding novelty claims in retrievable literature, achieving the highest evidence-grounding scores. However, this appeared to stem from a tendency to endorse novelty claims that were easily verifiable rather than exercising the same level of skepticism as humans.
• Flaw Identification: Reviewer2 acted as a high-sensitivity scanner, achieving the highest recall for both critical and minor flaws, significantly outperforming humans in detecting valid issues. However, other systems (e.g., CycleReviewer) showed higher rates of hallucinated minor flaws. Crucially, all systems (including humans) achieved near-perfect prioritization scores (nCPS ≈ 0.97), indicating that ordering critical issues first is a standard behavior across all evaluated models.
• Constructiveness: DeepReview produced the most actionable feedback, significantly outperforming humans in Solution Orientation (proposing specific fixes) and Tone. Human reviewers, while highly specific in their diagnoses, frequently failed to provide explicit solutions, acting more as diagnosticians than collaborators.

Significance and Claims

The paper argues that LLM peer reviewers should be understood not as general-purpose replacements for human expertise, but as targeted supplements effective within specific dimensions.

• Specialization over Replacement: The results demonstrate that while LLMs can match or exceed humans in specific tasks (e.g., exhaustive flaw scanning, grounding novelty claims, or formulating actionable solutions), they lack the holistic balance of human reviewers. No single system dominates all four dimensions simultaneously.
• Structured Failure Modes: Aggregate metrics often miss distinct failure modes, such as the "surface-level trap" (focusing on formatting over science) or "solution gaps" (identifying problems without prescribing fixes). PRISM's multi-dimensional approach is necessary to expose these blind spots.
• Deployment Guidance: The authors propose an ensemble deployment strategy where different systems are assigned specific roles within a human-assisted pipeline:
  • Use Reviewer2 for exhaustive flaw scanning (high diagnostic recall).
  • Use DeepReview for drafting constructive feedback (high actionability and solution density).
  • Use SEA-E for novelty-grounding checks (high literature support rate).

The work concludes that LLMs are powerful, task-matched specialists. Their deployment is most effective when they serve as "co-pilots" to augment human review, rather than as autonomous agents intended to replace human judgment entirely.