Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review

Based on the provided text, here is a detailed technical summary of the paper "Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review":

Problem
The paper addresses the critical challenge of scaling peer review in machine learning conferences, where an explosion in submissions has led to reviewer overload, inconsistent judgments, and superficial evaluations. While Large Language Models (LLMs) offer a potential solution for automating peer review, the specific impact of reviewer guideline design on the quality and alignment of automated reviews remains under-investigated. The authors seek to determine which types of guidelines—official conference policies versus those generated by imitating human reviewers—best guide LLMs to produce scores and justifications consistent with human judgments.

Methodology
The study employs a systematic experimental framework using three open-weight LLMs (Qwen3-30B-A3B, DeepSeek-R1-32B, and Qwen3-32B) on a dataset of 1,500 submissions and reviews from ICLR 2024. The evaluation metric is the Root Mean Squared Error (RMSE) between LLM-generated scores and the average human review scores.

The authors compare several guideline conditions:

1. No-Guideline: A zero-shot baseline where the model rates the paper without specific instructions.
2. Official Conference Guidelines: Direct usage of guidelines from NeurIPS, ICLR, and the ACL Rolling Review (ARR). These were tested in their original text format and as "instruction-style" prompts (rewritten by ChatGPT into imperative bullet points).
3. Reviewer-Imitating Guidelines: Guidelines generated by an LLM (Gemini 2.5 Pro) based on high-quality ("Good"), medium-quality ("Middle"), and low-quality ("Bad") human reviews. These reviews were filtered using programmatic proxies (length, submission date, citation quality, and score consistency).
4. Rubric vs. Non-Rubric: For the reviewer-imitating guidelines, the authors tested two scoring modes:
   • Non-rubric: Free-form evaluation based on the generated checklist.
   • Rubric: A strict additive scoring system where the model assigns +1 or -1 points for specific checklist items, which are then normalized to a 1–10 scale.

Key Contributions

1. Superiority of Official Guidelines: The study demonstrates that providing guidelines significantly improves automated review performance compared to a no-guideline baseline. Crucially, official conference guidelines (particularly from NeurIPS and ICLR) consistently yield the lowest RMSE, outperforming guidelines generated by imitating human reviewers. This suggests that evaluation criteria refined over time by the research community serve as a more effective foundation for automation than patterns distilled from individual reviews.
2. Ineffectiveness of Reviewer-Imitating Guidelines: Guidelines generated by imitating human reviewers generally performed worse than official guidelines. In some cases (e.g., with the Qwen3-30B-A3B model), using these generated guidelines actually degraded performance compared to the no-guideline baseline.
3. Detriment of Strict Rubrics: The study finds that enforcing strict, additive rubric-style scoring (computing scores by summing point allocations for checklist items) consistently degrades performance. Models perform better when allowed to determine scores subjectively and holistically based on the guidelines, rather than following rigid, mechanical scoring logic.
4. Impact of Presentation: The format of the guidelines matters. Converting official policy text into "instruction-style" prompts altered model behavior, sometimes flipping the relative ranking of which conference's guidelines were most effective.

Results

• RMSE Reduction: Across all models, providing guidelines reduced RMSE. For instance, DeepSeek-R1-32B showed substantial error reduction across all venues when using official guidelines compared to the no-guideline baseline.
• Venue Performance: With official guidelines, NeurIPS guidelines generally produced the best alignment (lowest RMSE), followed by ICLR and ARR. However, this ranking shifted when guidelines were converted to instruction-style prompts, indicating that presentation style interacts with model architecture.
• Rubric Failure: In the reviewer-imitating experiments, the "Non-Rubric" setting consistently outperformed the "Rubric" setting. For example, with Qwen3-30B-A3B, the RMSE for the "Good" group was 3.22 (non-rubric) versus 4.42 (rubric), indicating that rigid point-allocation systems significantly increase deviation from human judgments.

Significance
The paper claims that the design of prompts for automated peer review is not merely an administrative detail but a critical determinant of review quality. The findings suggest that:

• Automated systems should prioritize official, community-refined guidelines over synthetic guidelines derived from human review patterns.
• The holistic, subjective nature of human evaluation is difficult to replicate with rigid rubrics; allowing LLMs to interpret guidelines flexibly yields better alignment with human judgment.
• The study provides a reproducible, open-weights framework for evaluating automated review, emphasizing the need for transparency and privacy (avoiding uploading unpublished manuscripts to non-secure commercial APIs).

The authors conclude that while LLMs can assist in peer review, their effectiveness is heavily dependent on the quality and format of the guidance provided, with established conference policies offering the most reliable path forward.