💬 NLP

OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources

This paper introduces OARelatedWork, the first large-scale dataset enabling full-text related work generation from open-access sources, which reveals that while modern LLMs struggle with synthesizing massive contexts compared to abstracts, they can outperform human baselines in evidence-grounded factuality, necessitating new statement-level evaluation frameworks to replace inadequate reference-based metrics.

Original authors: Martin Docekal, Martin Fajcik, Pavel Smrz

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Martin Docekal, Martin Fajcik, Pavel Smrz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Writing the "Chapter of Others"

Imagine you are writing a novel. Before you tell your own story, you have to write a chapter called "Related Work." In this chapter, you have to summarize what other authors have written before you, explain how their stories are similar to yours, and show where your story is different.

For a long time, computers trying to do this were like students who were only allowed to read the blurbs on the back of books (the abstracts). They had to guess the whole story based on a tiny summary. This paper introduces a new dataset called OARelatedWork that changes the rules: now, the computer gets to read the entire library (the full text of every book) before it tries to write its chapter.

The Problem: The "Blurb" vs. The "Book"

Previously, researchers built datasets where computers only saw the short summaries of cited papers.

The Old Way: It's like trying to write a book review based only on the back cover. You might get the main idea, but you miss the details, the nuances, and the specific evidence.
The New Way (OARelatedWork): This dataset gives the computer the full text of 94,000 papers and the full text of the millions of books they reference. It forces the AI to read the whole book, not just the blurb.

What They Found: The "Smart" vs. The "Human"

The researchers tested this new dataset with various AI models and compared them to human writers. They found some surprising things:

More Information Can Be a Trap: When the AI was given the full text of books instead of just the blurbs, it actually made more mistakes about where to put the citations. It was like giving a student a 1,000-page textbook and asking them to find one specific fact; they got overwhelmed and sometimes pointed to the wrong page.
Humans "Make Things Up" (Abstraction): Human authors are very creative. They often read a book, close it, and then write a sentence that sounds like it came from that book, even if the exact words weren't there. They mix ideas from different parts of the book to create a smooth story.
AI is the "Strict Librarian": Because the AI is trained to be very careful and only say things it can prove with a direct quote, it turned out to be more factually accurate than humans when judged strictly.
- The Analogy: Imagine a human writer is a jazz musician who improvises a beautiful melody that feels right but might not follow the sheet music exactly. The AI is a strict conductor who checks every note against the sheet music. In this specific game of "did you follow the rules?", the AI won.

The Scoreboard: How Do We Grade?

The paper argues that the old way of grading these summaries (counting how many words overlap between the AI's work and a human's work) is broken.

The Old Score: Like grading a student's essay by counting how many words they used that were in the teacher's example. If the student used different words to say the same thing, they got a bad grade.
The New Score: The authors created a "Statement-Level Judge." Instead of looking at the whole essay, they break it down sentence by sentence and ask: "Is this fact true? Is the citation correct?"
- They found that standard grading tools are terrible at this. You need a smart AI judge (like a senior professor) to check the facts, not just a spell-checker.

The Conclusion

This paper says: "Stop giving AI just the back-cover blurbs. Give it the whole library."

The Dataset: It's a massive collection of open-access papers with full texts.
The Lesson: When AI has to read the whole book, it struggles to synthesize the information as smoothly as humans do, but it becomes incredibly good at sticking to the facts.
The Future: To build better tools for researchers, we need to stop testing AI on short summaries and start testing it on the full, messy, complex reality of reading entire academic papers.

In short: This paper built a giant library for AI to practice writing literature reviews. It discovered that while AI is better at following the rules of fact-checking than humans, humans are still better at weaving those facts into a smooth, natural story.

Technical Summary: OARelatedWork

Problem Statement

The generation of "Related Work" sections is a critical yet under-addressed challenge in Natural Language Processing (NLP). While related work generation extends multi-document summarization, it uniquely requires not only summarizing referenced papers but also positioning a target work within prior research through comparative discourse. Existing approaches face three primary limitations:

Input Constraints: Most datasets and models rely exclusively on paper abstracts due to the technical difficulty of processing long full-text inputs. This restricts the model's ability to synthesize deep, evidence-grounded insights.
Output Granularity: Prior benchmarks often limit generation to single paragraphs, failing to evaluate a model's ability to produce well-structured, multi-section related work chapters.
Evaluation Inadequacy: Standard reference-based metrics (e.g., ROUGE, BERTScore) are insufficient for long-form, structured generation. They struggle to assess factual faithfulness and often fail to correlate with human judgment in long-context scenarios.

Furthermore, the field lacks a large-scale dataset where every cited paper in the validation and test splits is available in full text, preventing controlled evaluation of full-text synthesis capabilities.

Methodology

Dataset Construction: OARelatedWork

The authors introduce OARelatedWork, the first large-scale multi-document summarization dataset designed for generating entire related work sections from open-access sources.

Corpus Source: The dataset is built by merging the CORE corpus (5.4M documents in XML/TEI format) and the Semantic Scholar corpus (129.2M documents, including S2ORC and API data).
Processing Pipeline:
- Bibliography Linking: A multi-stage process using Microsoft Academic Graph (MAG), the Semantic Scholar citation graph, and a custom searcher to link citations to full-text documents. This increased identified bibliographic entries by 0.6% (approx. 11.6M links).
- Filtering: Documents were filtered to ensure related work sections contain at least three sentences, cite at least two documents, and have linked full-texts for all cited papers in the validation and test splits.
- Structure: Data is represented in JSON with a hierarchical tree structure (sections, subsections, paragraphs, sentences) to facilitate future use cases.
Statistics: The final dataset contains 94,450 papers with related work sections and 5.8 million unique cited papers. Crucially, the validation and test sets ensure 100% full-text availability for cited references, a feature missing in prior datasets like S2ORC, Delve, and Multi-XSci.

Task Definition

The task involves generating a related work section $Y$ given:

A combination of input content types (abstracts and/or full texts of the target paper and cited papers).
A target length $L$ .
Structural instructions (whether to use subsections).

The system must synthesize information from multiple sources to produce a coherent, structured narrative.

Evaluation Framework

To address the inadequacy of traditional metrics, the authors propose a statement-level evaluation framework:

LLM Judges: A quantized Gemma 4 31B model (without reasoning) is used as a judge to extract factual statements from generated text and classify them into four categories: True, True but wrong citation, Unverifiable, and False.
Human Evaluation: A fine-grained human evaluation was conducted on 40 papers (120 sections, 408 statements) to validate the LLM judges and assess faithfulness, relevance, and language quality.
Metrics: In addition to the statement-level faithfulness metrics, the study reports ROUGE scores, BERTScore, and citation coverage F1.

Key Contributions

OARelatedWork Dataset: A large-scale resource enabling the shift from abstract-only to full-text related work generation. It provides the first benchmark where test sets guarantee full-text access for all cited references.
Empirical Analysis of Human Writing: Through human evaluation, the authors reveal that human authors frequently introduce abstractive claims that are ungrounded in localized source texts (high "Unverifiable" rates in strict evaluations).
LLM vs. Human Baselines: The study demonstrates that advanced LLMs (specifically GPT-4o-mini) can surpass human baselines in strict, evidence-grounded factuality when evaluated against the provided context.
Meta-Evaluation of Metrics: The paper proves that standard reference-based metrics are inadequate for this task. It validates the use of LLM-based judges for reliable evaluation of structured, long-form generation.

Results

Model Performance

The authors benchmarked a spectrum of models, including extractive baselines (LEAD, TextRank), a fine-tuned multi-document summarization model (PRIMERA), and zero-shot LLMs (Llama 3.2 3B, GPT-4o-mini).

Impact of Full-Text Inputs: Providing full texts instead of abstracts significantly challenges models. For GPT-4o-mini, the "True" evidence-grounded rate dropped from 92.9% (abstracts only) to 83.8% (full texts), accompanied by increases in "Wrong Citation" and "Unverifiable" rates. This suggests that synthesizing massive full-text contexts remains a significant challenge even for modern LLMs.
Extractive vs. Abstractive: Extractive baselines achieved very high "True" scores (95–99%) but this is attributed to reusing source text rather than synthesis. Conversely, abstractive models showed higher rates of hallucination or citation errors but better coherence.
Human vs. Model: In a pairwise human evaluation, GPT-4o-mini overwhelmingly dominated both PRIMERA and human authors in Relevance, Language, and Overall Ranking. In Faithfulness, GPT-4o-mini statistically tied with PRIMERA but outperformed human authors in strict statement-level factuality.

Human Writing Behavior

Analysis of 408 human-written statements revealed:

High Unverifiability: Human authors had a high rate of "Unverifiable" statements (18.5% in the gold set), often due to integrating information distributed across the document rather than relying on localized evidence spans.
Citation Errors: Humans frequently introduced claims ungrounded in the specific localized source text provided, leading to a higher rate of "True but wrong citation" or "Unverifiable" classifications under strict evaluation.

Metric Validation

The meta-evaluation confirmed that reference-based metrics (ROUGE, BERTScore) correlate poorly with human judgment on this task. LLM-based judges (specifically Gemini 3.1 Pro and Gemma 4) showed strong correlation with human annotations, validating the proposed statement-level framework.

Significance and Claims

The paper claims that OARelatedWork shifts the field toward a realistic, full-section paradigm by leveraging comprehensive full-text contexts. Its primary significance lies in:

Challenging the Status Quo: It demonstrates that moving beyond abstracts to full-text synthesis is necessary for realistic related work generation, despite the increased difficulty.
Re-evaluating Human Baselines: It challenges the assumption that human writing is the gold standard for factual faithfulness in this context, showing that LLMs can be more strictly grounded in provided evidence.
Evaluation Standards: It establishes that standard metrics are insufficient for long-form structured outputs and introduces a robust, statement-level evaluation framework as the new standard for this domain.

The authors conclude that while current models struggle with the sheer volume of full-text context, the combination of structured data, fine-grained factual annotations, and specialized evaluation frameworks provides a necessary foundation for future research in automated scientific writing.