RWGBench: Evaluating Scholarly Positioning in Related Work Generation

Technical Summary: RWGBench

Problem Definition
The paper addresses a critical gap in the evaluation of Automated Related Work Generation (RWG). While Large Language Models (LLMs) demonstrate strong fluency in scientific writing, existing evaluation metrics largely rely on summarization-oriented approaches (e.g., lexical overlap, semantic similarity to reference text). The authors argue that RWG is fundamentally distinct from generic summarization; it is a scholarly positioning task. This task requires not just text synthesis, but complex citation-level decision-making: selecting relevant prior work, organizing it to highlight gaps, and framing the target paper's contribution relative to existing research. Current metrics fail to detect academically critical failures, such as inappropriate citation selection, misplaced references, or the omission of central prior work, even when the generated text appears fluent and coherent.

Methodology
To address these limitations, the authors introduce RWGBench, a benchmark designed to evaluate RWG through the lens of citation decision-making rather than surface-level text similarity.

• Dataset Construction:

  • Source: The benchmark is built from 40,108 computer science papers published on arXiv between 2020 and 2025.
  • Retrieval Corpus: A large-scale corpus of 1.09 million documents was constructed by aligning citations from the source papers with the SurGE corpus and supplementing missing entries via the arXiv API.
  • Test Set: A high-fidelity test set of 100 papers was curated. These papers feature professionally written related work sections and have a 97.7% citation coverage rate within the retrieval corpus, ensuring realistic evaluation constraints.

• Evaluation Framework:
  Instead of relying solely on ROUGE or BERTScore, the authors propose a multi-dimensional framework assessing four key aspects of scholarly positioning:

  1. Citation Appropriateness:
     • Precision/Recall: Measures alignment between generated and gold-standard citation sets.
     • Claim-Support Consistency (CSC): Uses Natural Language Inference (NLI) to verify if the cited paper's abstract supports the specific claim in the generated sentence.
     • Topical Relevance (TRT): Measures semantic similarity between the target paper and cited papers at the document level.
  2. Citation Organization:
     • Citation Density (CD): Evaluates the rate of citations per sentence to detect over- or under-citing.
     • Citation Distribution Balance (CDB): Measures the proportion of unique citations to assess diversity versus repetitive reliance on a small set of papers.
  3. Structural Coherence:
     • Positioning Structure Similarity (PSS): Compares the distribution of discourse roles (Background, Gap, Positioning) in generated text against human references using Jensen–Shannon divergence.
  4. Text Quality:
     • Includes standard metrics (ROUGE-L, BERTScore) and an LLM-as-Judge score, though the authors note these are supplementary to citation-centric metrics.

• Experimental Setup:
  The study evaluates four generation paradigms (RAG, ALCE, TAG, Workflow) across three frontier LLMs (DeepSeek-V3, Llama-3-8B, Qwen-2.5-7B) using both sparse (BM25) and dense retrieval methods. An "Oracle" setting is also employed, where models are provided with ground-truth citations to isolate generation-level failures from retrieval errors.

Key Results
Experiments on RWGBench reveal systematic limitations in current RWG systems that are obscured by standard evaluations:

1. Retrieval vs. Generation Bottlenecks: While dense retrieval improves semantic similarity, it yields only marginal gains in citation accuracy. Oracle experiments show that even with perfect citations, models struggle with placement and framing, but the primary bottleneck remains retrieval.
2. Decoupling of Fluency and Accuracy: Models can generate fluent, high-scoring text (according to LLM-as-Judge or ROUGE) while failing to identify necessary citations or selecting irrelevant ones. Surface-level metrics fail to detect these scholarly reasoning errors.
3. Systematic Organizational Deviations: Generated sections exhibit higher citation density and lower distribution balance compared to human-authored sections, indicating a tendency toward repetitive emphasis and citation clustering.
4. Contextual Justification Gap: Even when models select the correct papers, they struggle to integrate them into coherent argumentation (placement and rhetorical framing). This gap persists even in Oracle settings.
5. Metric Alignment: Human evaluation confirms that citation-specific metrics (CSC, TRT, Recall) align significantly better with expert judgment (60–69% pairwise accuracy) than surface-level metrics like BERTScore (44.4%) or LLM-as-Judge scores (35.8%), which often favor fluent but citation-deficient outputs.

Significance and Contributions
The paper claims four primary contributions:

1. Reformulation: It reframes RWG as a scholarly positioning task centered on citation decision-making, moving beyond text generation.
2. Benchmark: It provides RWGBench, a realistic benchmark with curated targets and a massive retrieval corpus to support end-to-end evaluation.
3. Framework: It introduces a behavior-oriented evaluation framework that enables fine-grained diagnosis of failure modes in citation selection, placement, and organization.
4. Empirical Insight: It exposes critical limitations in current systems, demonstrating that retrieval quality and citation reasoning are distinct challenges that standard metrics fail to capture.

The authors conclude that RWGBench offers a necessary testbed for developing RWG systems that align with actual scholarly writing practices, emphasizing that rigorous benchmarking is essential to understand where LLM assistance is appropriate and where it falls short in consequential academic tasks.