Technical Summary: SurGE – A Benchmark and Evaluation Framework for Scientific Survey Generation

Problem Statement

The exponential growth of scientific literature, exemplified by the doubling of daily arXiv submissions between 2019 and 2024, has rendered manual creation of comprehensive survey papers increasingly impractical. While recent Large Language Model (LLM)-based agents have demonstrated the feasibility of automating survey generation through hierarchical decomposition and memory-driven mechanisms, the field lacks a standardized, scalable evaluation protocol. Existing evaluation methods suffer from two primary limitations:

1. Scalability Issues: Many approaches, such as StepSurvey, rely heavily on human evaluation, which is labor-intensive and difficult to reproduce.
2. Bias and Lack of Comparability: Other methods, like AutoSurvey and SurveyForge, utilize custom evaluation frameworks designed to validate their specific pipelines. This creates a self-validation loop that introduces systematic bias, favoring design-specific assumptions and preventing fair, cross-system comparisons.

Consequently, there is an urgent need for a method-agnostic, reproducible, and diagnostic benchmark that can disentangle the ability to retrieve relevant literature from the ability to synthesize it into a coherent survey.

Methodology

The authors introduce SurGE (Survey Generation Evaluation), a comprehensive benchmark and evaluation framework that formalizes survey generation as a two-stage process: (1) Information Collection, where a system retrieves relevant papers from a corpus, and (2) Survey Generation, where a system synthesizes these papers into a structured article with proper citations.

1. Dataset Construction

SurGE provides a rigorously constructed dataset comprising:

• Ground-Truth Surveys: 205 high-quality survey papers sourced from arXiv (2020–2024) that explicitly identify as surveys or systematic reviews and possess a minimum of 20 citations. These were subjected to a secondary manual filtering process by four Ph.D. students and a faculty member, evaluating citation impact, content coverage, structural coherence, and citation quality. Papers were retained only if both annotators deemed them "usable," resulting in a Cohen's Kappa of 0.792.
• Retrieval Corpus: A large-scale academic corpus of 1,086,992 unique papers built entirely from arXiv metadata (titles, authors, abstracts, categories, etc.). The corpus was constructed by retrieving metadata for papers cited in the ground-truth surveys and expanding the pool with topic-related papers and a random sample of CS papers to mitigate bias. The dataset adheres to ethical standards by excluding full-text PDFs, relying solely on metadata permitted under CC0 licenses.

2. Evaluation Framework

SurGE proposes a multi-dimensional, automated evaluation framework assessing four key dimensions against the Ground Truth (GT):

• Comprehensiveness: Measured by Recall, calculating the overlap between the generated survey's references and the GT reference list. This serves as a pragmatic metric for identifying the core body of work validated by the research community.
• Citation Accuracy: Evaluated at three levels (Document, Section, Sentence) using a Natural Language Inference (NLI) model (nli-deberta-v3-base). The model determines if a cited paper is thematically relevant to the topic, semantically appropriate for the section, and supportive of the specific claim in the sentence. Scores are derived from the model's Entailment, Neutral, and Contradiction probabilities.
• Structural Quality: Assessed via two metrics:
  • Structure Quality Score (SQS): An LLM-as-a-Judge metric comparing the hierarchical organization, meaning, and wording of generated headings against GT headings.
  • Soft-Heading Recall (SHR): A metric using semantic embedding similarity to measure the alignment between generated and GT headings, robustly handling paraphrasing via soft cardinality overlap.
• Content Quality: Measured by the Content Quality Score (CQS), where an LLM (GPT-4o) evaluates each section based on fluency, logical clarity, redundancy avoidance, description clarity, and error absence.

The framework's reliability was validated through a meta-evaluation, showing high correlation (Spearman's $\rho \approx 0.70$–$0.88$) between the automated metrics and human expert rankings, particularly in distinguishing machine-generated text from human-level scholarship.

Key Contributions

1. SurGE Benchmark: The introduction of a method-agnostic benchmark featuring expert-validated ground-truth surveys and a large-scale retrieval corpus of over one million papers.
2. Automated Evaluation Framework: A robust protocol assessing survey quality across four dimensions: comprehensiveness, citation accuracy, structure, and content, utilizing a hybrid of objective metrics and LLM-based judgments.
3. Empirical Analysis: Extensive experiments benchmarking state-of-the-art systems, providing diagnostic insights into the specific bottlenecks of current agentic pipelines.

Experimental Results

The authors benchmarked several systems, including standard Retrieval-Augmented Generation (RAG) baselines and specialized agentic pipelines (AutoSurvey, StepSurvey, SurveyForge), all utilizing a shared dense retriever and the Qwen2.5-14B-Instruct backbone for fair comparison.

• Retrieval Performance: The fine-tuned dense retriever achieved a Recall@1000 of 68.05% for ground-truth references, significantly outperforming the BM25 baseline (27.15%). This confirms that the corpus contains the necessary information and that the bottleneck lies in the generation stage, not data availability.
• Generation Performance:
  • Fluency vs. Accuracy: Standard RAG models (e.g., RAG-Qwen) achieved competitive n-gram scores (ROUGE-L, BLEU) but suffered from the lowest Citation Accuracy and Recall. This highlights a "fluency hallucination" trap where models generate coherent but factually ungrounded text.
  • Structural Quality: Specialized agentic pipelines consistently outperformed standard RAG baselines in Structural Quality (SQS) and Soft-Heading Recall (SHR), demonstrating the necessity of explicit planning and hierarchical decomposition.
  • Agent Capabilities: While agentic pipelines improved structural quality and citation accuracy, they still struggled with information utilization. For instance, SurveyForge excelled in retrieving accurate literature (highest Recall and Doc-Acc), while StepSurvey achieved the highest Content Quality Score (CQS) due to its granular, step-by-step drafting process.
  • Model Capacity: The results suggest that well-structured agentic frameworks can compensate for limited model capacity; SurveyForge (using Qwen-14B) surpassed RAG-Gemini in citation accuracy, indicating that decomposition strategies can enable smaller models to produce more faithful outputs.

Significance

The paper posits that SurGE addresses a critical gap in the field by establishing the first method-agnostic benchmark for scientific survey generation. By decoupling retrieval from generation and providing a diagnostic evaluation protocol, SurGE enables researchers to identify whether performance failures stem from information acquisition or content synthesis. The authors conclude that while agentic pipelines represent a significant advancement over standard RAG, current systems still face challenges in citation accuracy and full information utilization. SurGE provides the necessary infrastructure to rigorously assess these systems and guide future research toward more trustworthy and effective automated survey generation.