Technical Summary: Expert Preference-based Evaluation of Automated Related Work Generation

Problem Statement

Scientific writing, particularly the generation of "Related Work" (RW) sections, demands extensive domain knowledge and the ability to reason over novel information in relation to existing literature. While Large Language Models (LLMs) show promise in this domain, evaluating the quality of automatically generated scientific text remains a critical open issue. Conventional automatic metrics (e.g., ROUGE, BERTScore) and standard "LLM-as-a-judge" systems are insufficient because they lack domain-specific criteria, fail to capture expert preferences, and often suffer from pretraining biases and a lack of transparency. Existing evaluation frameworks do not adequately distinguish between valid generations that differ in surface-level lexicons or identify incorrect generations that share similar lexical traits with correct ones. Furthermore, current RW generation datasets often rely on limited context (e.g., titles and abstracts), failing to provide the depth required for comprehensive literature reviews.

Methodology

To address these gaps, the authors propose GREP (Granular Related-work Evaluation based on Preferences), a multi-turn evaluation framework designed to simulate realistic human-AI collaborative writing.

1. Dataset Construction

The authors constructed a novel RW generation dataset comprising 50 main papers from top-tier NLP and Computer Vision venues, with an average of 13.5 cited papers each (totaling 725 papers). Unlike previous datasets that utilized only titles and abstracts, this dataset includes Introduction sections from cited papers to provide rich contextual information necessary for understanding relations between works. The data was manually curated to ensure quality and completeness.

2. Evaluation Criteria

GREP decomposes evaluation into Hard Constraints (essential requirements for a valid RW section) and Soft Constraints (reflecting expert preferences among multiple valid drafts):

• Hard Constraints:

  • Citation Verification: Measures "Missing Ratio" (uncited provided papers) and "Hallucination Ratio" (citations not in the provided list).
  • Faithfulness: Evaluates factual alignment between citation sentences and the source papers using Natural Language Inference (NLI). A citation is faithful only if all claims are fully supported by the source.
  • Positioning Existence: Checks for statements that differentiate the main paper's contributions from prior work.

• Soft Constraints:

  • Length: Ensures the token count falls within a tolerance ratio of a gold standard.
  • Citation Emphasis: Measures the allocation of content to specific citations compared to a gold standard.
  • Positioning Type & Ratio: Evaluates whether contributions are highlighted in every paragraph or summarized in the final paragraph, and the ratio of paragraphs satisfying the expected expression type.

3. The GREP Framework

GREP operates as an iterative loop (Figure 1 in the paper):

1. Generation: An LLM (the generator) produces a draft based on main paper details and a list of cited papers.
2. Evaluation: Dedicated modules evaluate the draft against hard and soft constraints.
   • Faithfulness and Positioning are evaluated using LLMs. To mitigate the limitations of standard LLM judges, GREP employs Localized Judgment (evaluating specific contexts rather than holistic scores) and Manipulated Contrastive Examples (few-shot examples showing correct and incorrect reasoning to guide the judge).
   • Hard constraints (like missing citations) are verified deterministically.
3. Feedback Generation: The evaluation report is converted into natural language feedback.
4. Refinement: The generator uses this feedback to produce a revised draft in the next iteration.

The framework offers two variants: PreciseGREP (using proprietary models like GPT-4o and o3-mini for higher accuracy) and OpenGREP (using open-weight models like Gemma 3 and Llama 3.3 for cost-efficiency).

Key Contributions

1. Novel Dataset: A high-quality RW generation dataset enriched with Introduction sections from cited papers, addressing the context limitations of prior datasets.
2. Fine-Grained Evaluation Rubric: A comprehensive set of hard and soft constraints that capture both validity and expert stylistic preferences.
3. GREP Framework: A multi-turn evaluation system that integrates deterministic checks with localized, contrastive-example-augmented LLM judging. It unifies verifiable criteria with criteria requiring deep natural language understanding.
4. Expert Validation: A study involving 16 domain experts (NLP and CV) who evaluated LLM generations in a pairwise, multi-turn setting. The study validates that GREP's judgments closely align with human expert assessments, outperforming standard judge models.

Results

• Alignment with Experts: In expert studies, GREP variants demonstrated significantly higher agreement with human judgments than baseline judge models (e.g., Self-Taught Evaluator, DeepSeek-GRM). PreciseGREP achieved a 78% match rate in citation faithfulness compared to experts, versus 53% for specialized SoTA LLMs. OpenGREP also showed strong alignment (66%).
• Generator Performance: State-of-the-art LLMs struggle to generate valid RW sections without assistance. Even the best-performing model (o3-mini) achieved perfect citation faithfulness in only 20% of initial drafts.
• Feedback Utilization: LLMs generally struggle to incorporate feedback effectively. While some models improved on specific metrics (e.g., length adjustment), many failed to track multiple improvement aspects simultaneously or introduced inconsistencies that degraded quality.
• Dynamic Adaptability: When user preferences changed dynamically (e.g., introducing new papers mid-interaction or changing positioning styles), models showed a severe decline in performance. They often failed to integrate new information or adapt to style changes immediately, with performance not fully recovering even after repeated feedback.

Significance

The paper claims that GREP provides a robust, domain-grounded evaluation framework that bridges the gap between automated metrics and human expert judgment in scientific writing. By decomposing complex evaluation tasks into localized, semi-objective tests and utilizing contrastive examples, GREP addresses the "what to judge" and "how to judge" problems inherent in current LLM-as-a-judge systems.

The authors emphasize that their work is the first to conceptualize automated RW evaluation as an expert domain task with domain-specific utilities. The framework offers a reliable proxy for human judgment, reducing the cost of human-in-the-loop experimentation while supporting the development of realistic human-AI collaborative writing tools. The study highlights that current frontier LLMs, despite their reasoning capabilities, still struggle with the nuanced constraints of scientific literature synthesis, particularly regarding citation faithfulness and dynamic preference adaptation.