Position on LLM-Assisted Peer Review: Addressing Reviewer Gap through Mentoring and Feedback

Technical Summary: Position on LLM-Assisted Peer Review

Problem Statement: The Reviewer Gap
The paper identifies a critical structural mismatch in the AI research ecosystem termed the "Reviewer Gap," driven by the unprecedented expansion of AI submissions (e.g., 20–30% annual increases at major conferences) outpacing the growth of the review infrastructure. This gap manifests in two reinforcing dimensions:

1. Volume Gap: Submission volumes exceed processing capacity, leading to excessive reviewer assignments, fatigue, and superficial, perfunctory assessments.
2. Quality Gap: The pool of reviewers is expanding to include junior researchers and authors with limited reviewing experience, creating a deficit in domain expertise. Without systematic training, this leads to the entrenchment of low-quality review norms.

The authors critique existing Large Language Model (LLM) approaches that focus on automated review generation. They argue that direct LLM generation risks hallucination, superficial summarization, and the propagation of biases, failing to address the root cause: the lack of reviewer competence.

Methodology: A Human-Centered Dual-System Framework
The paper proposes a paradigm shift from automation to augmentation, positioning LLMs as educational assistants and feedback tools rather than autonomous reviewers. This approach is grounded in five foundational principles of high-quality peer review: Fidelity (accuracy to facts), Clarity (unambiguous, evidence-based language), Fairness (bias-free, neutral tone), Proportionality (critique weight commensurate with contribution), and Constructiveness (actionable suggestions).

The proposed solution consists of two complementary systems:

1. LLM-Assisted Reviewer Mentoring System (Education):

   • Goal: To systematically cultivate reviewer competencies through a "safe-to-fail" learning environment.
   • Structure: A three-stage pedagogical curriculum:
     • Guided Recognition: Learners analyze examples of high- and low-quality reviews against the five principles to internalize criteria.
     • Review Refinement Practice: Learners revise drafts containing intentional flaws, receiving targeted feedback to identify deficiencies.
     • Full Simulation: Learners write complete reviews, which the LLM analyzes to generate a comprehensive mentoring report highlighting specific successes and failures relative to the principles.
   • Outcome: Upon completion, reviewers receive a voluntary "Reviewer Certification," signaling enhanced expertise without mandating participation.

2. LLM-Assisted Reviewer Feedback System (Verification):

   • Goal: To bridge the gap between training and execution by supporting reviewers during the actual review process.
   • Workflow:
     • Detection & Cross-Verification: An LLM "critic agent" scans submitted draft reviews for violations of the five principles (e.g., factual errors, overlooked content, definitive claims).
     • Evidence-based Feedback Generation: The system extracts specific evidence from the paper to help reviewers justify critiques (e.g., clarifying vague statements about dataset insufficiency by referencing specific benchmarks used in the paper).
     • Reliability Testing: Before delivery, feedback undergoes a check to ensure appropriate tone, avoid reviewer-to-author phrasing, and prevent uninformative praise.
     • Optional Refinement: Feedback is shared privately with reviewers and Area Chairs. Revision is non-mandatory, preserving reviewer autonomy while enabling self-correction.

Key Contributions

• Conceptual Reframing: The paper argues against replacing human judgment with LLMs, proposing instead a dual-system architecture where LLMs serve as mentors (long-term skill building) and feedback mechanisms (real-time quality assurance).
• Standardized Evaluation Rubric: It defines a unified set of five core principles (Fidelity, Clarity, Fairness, Proportionality, Constructiveness) to serve as the objective basis for LLM-based mentoring and feedback.
• Architectural Design: The proposal outlines a specific workflow integrating educational simulation with post-draft verification, ensuring that learned principles are applied in real-world scenarios without imposing additional burdens on reviewers.
• Collaborative Improvement Loop: The framework includes a mechanism for human experts (e.g., Area Chairs) to provide meta-feedback on the LLM's suggestions, creating a closed loop for continual system refinement and bias mitigation.

Results and Claims
As a position paper, this work does not present empirical experimental results or quantitative metrics from a deployed system. Instead, the "results" are theoretical and structural:

• The paper claims that shifting from automated generation to LLM-assisted mentoring and feedback addresses both the quantitative (capacity) and qualitative (expertise) dimensions of the Reviewer Gap.
• It posits that this human-centered approach mitigates the risks of automation bias and deskilling by keeping human reviewers in the loop as the primary decision-makers, with LLMs acting as supportive tools for error detection and skill reinforcement.
• The authors claim the system respects reviewer autonomy through its optional refinement structure, preventing the homogenization of reviews while ensuring adherence to professional standards.

Significance
The paper asserts that its significance lies in offering a sustainable pathway for the scholarly ecosystem. By focusing on cultivating reviewer expertise rather than merely generating text, the proposed framework aims to:

1. Break the vicious cycle of low-quality reviews and research stagnation.
2. Establish a virtuous cycle where improved reviews lead to more rigorous research, which in turn advances AI technologies that further strengthen the review ecosystem.
3. Provide a scalable, cost-effective solution to the Reviewer Gap that does not rely on the impossible task of rapidly expanding the pool of expert human reviewers, but rather on systematically upskilling the existing community.

Ultimately, the paper envisions a future where the peer review system is not just a bottleneck to be automated, but a robust, self-improving component of scientific progress, supported by AI that enhances rather than replaces human expertise.