Judgment-Grounded Expansion for Peer Review Generation

Problem
Automatic review generation is a promising direction for accelerating scientific progress, yet most existing systems adopt an end-to-end setup that autonomously generates reviews with little or no human intervention. While this minimizes human effort, it limits human control during content creation and raises significant accountability concerns. Fully automated reviews have been shown to become homogenized across different papers, exhibit verification failures, and be susceptible to manipulation. There is a need to better balance automation with accountability, particularly in formal settings like conference reviewing where human oversight is crucial.

Methodology
The authors formalize judgment-grounded expansion, a human-AI collaboration mode where a reviewer provides an evaluative claim (a key point and its corresponding judgment) regarding a paper, and the system expands it into one or several review comment candidates grounded in the submission.

The process is modeled as a structured generate-check-refine workflow:

1. Human Input: The reviewer provides a concise evaluative claim (e.g., "Novelty is a strength").
2. Generation: The system generates a set of candidate comments. Unlike single-point prediction, set prediction is used to increase the likelihood of covering the reviewer's intended comment.
3. Quality Check: The reviewer assesses candidates based on two dimensions:
   • Intention: Relevance (does it address the key point?) and Specification (does it capture the intended specificity?).
   • Grounding: Evidence (is it hallucination-free, matching human evidence, or sufficiently supportive?) and Reasoning (is the logical connection clear?).
4. Refinement: If a candidate fails the check, the reviewer can refine the input (e.g., clarifying specificity or requesting alternative evidence) to trigger re-generation.

To address the challenge of scalable evaluation (which typically requires costly human participation), the authors develop simulation methods using two types of proxies for human input: an exemplar-based proxy (using real user inputs as in-context examples) and an aspect-based proxy (using paper aspects and strength/weakness labels). They also simulate quality checks using an LLM-as-a-judge.

To address candidate set curation (balancing the need for coverage against reviewer cognitive load), the authors cast the problem as a conformal prediction task. This allows the system to generate variable-size prediction sets with a formal guarantee of target coverage, expanding the set only when the model is uncertain.

Key Contributions

1. New Task Formulation: The paper formalizes judgment-grounded expansion as a specific collaboration mode and models it as a generate-check-refine process, distinguishing it from general writing expansion by anchoring generation in the reviewer's own evaluative claim.
2. New Interaction Data: A user study was conducted with 6 participants across 14 papers, collecting 81 human-model interaction instances. This dataset characterizes reviewer input patterns, quality check behaviors, and user experience.
3. New Evaluation Pipeline: A novel simulation method is developed to enable reproducible, large-scale evaluation of judgment-grounded expansion, allowing for the analysis of how backbone models and task factors affect generation.
4. Conformal Prediction Application: The paper demonstrates that conformal prediction is well-suited for balancing candidate set size and target coverage, offering a principled way to manage the trade-off between coverage and reviewer burden.

Results

• User Study: Concise inputs (average 17.5 words) were found to support successful expansion, with 72.8% of interactions resulting in immediate acceptance. Failure cases often stemmed from inputs that were too brief, hedged, or referenced external papers. Participants reported moderate effort in correcting generated reviews but high satisfaction with the final output and a strong preference for judgment-grounded expansion over writing from scratch.
• Simulation and Evaluation: Large-scale evaluation comparing judgment-grounded expansion (JGE) with end-to-end (E2E) baselines showed that JGE generally aligns better with reference reviews, particularly for larger models and those trained with reasoning objectives (e.g., DeepSeek-R1 distillation). JGE also demonstrated higher practical utility in terms of technical accuracy and analytical depth.
• Robustness: JGE showed significantly greater adversarial robustness than E2E generation. When subjected to prompt injection attacks, JGE exhibited much smaller changes in score predictions, indicating that anchoring generation in reviewer judgment makes the system less susceptible to manipulation.
• Judgment-Utilization Score (JUScore): A diagnostic measure was introduced to quantify how much a specific model benefits from reviewer judgment. Results indicated that models with higher end-to-end performance do not necessarily have high JUScores; for instance, Qwen3-32B showed the largest gains from reviewer judgment.
• Candidate Curation: Experiments showed that conformal prediction using rank-based nonconformity scores provided a better size-coverage trade-off than fixed score-based ranking, allowing practitioners to tune the system for either broader coverage or reduced cognitive load.

Significance
The paper establishes judgment-grounded expansion as a concrete task and provides empirical and methodological foundations for the design of future collaborative review generation systems. It argues that this approach accommodates a genuine need to balance automation and accountability by ensuring the reviewer remains responsible for the evaluative stance while delegating the expansion work to the system. The work offers a systematic study and resources (data, simulation methods, and evaluation metrics) to advance the development of accountable AI-assisted peer review.