💬 NLP

Large-scale dataset of automatically classified rhetorical sections in scientific papers

This paper presents a large-scale dataset of automatically classified rhetorical sections for 15.6 million scientific papers from the Semantic Scholar Open Research Corpus, validated to achieve human-level agreement and enabling granular computational analysis of scientific writing patterns.

Original authors: Daniel Verdi, Jacob Aarup Dalsgaard, Roberta Sinatra

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Verdi, Jacob Aarup Dalsgaard, Roberta Sinatra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a massive library containing 15.6 million scientific papers. If you tried to read them all, you'd be overwhelmed. But even more confusing is that every paper is a bit of a puzzle: some have clear chapters like "Introduction" or "Methods," while others use weird titles like "The Journey of Discovery" or "What We Found."

This paper is about building a super-fast, automated librarian that can walk through this entire library, look at every single paper, and instantly sort the pages into their correct chapters.

Here is how they did it, explained simply:

1. The Problem: The "Messy Desk"

Scientists usually write papers in a standard format called IMRaD (Introduction, Methods, Results, and Discussion). It's like a recipe: you say what you're cooking, how you'll cook it, what happened, and what you think about it.

But in the real world, scientists are messy.

One paper might call the "Methods" section "Experimental Design."
Another might call "Results" "Findings."
Some papers mix sections together, like "Results and Discussion."

Trying to find the "Methods" section in millions of papers by hand is impossible. Previous attempts to do this with computers were like trying to find a needle in a haystack using a tiny magnet—they only worked on small piles of papers and didn't scale up.

2. The Solution: The "Rule-Based Detective"

Instead of using a complex, expensive AI that tries to "think" like a human (which takes forever and costs a fortune), the authors built a Rule-Based Detective.

Think of this detective as a very strict librarian with a checklist:

Pass 1 (The Exact Match): The detective looks for the word "Introduction." If it sees it, it tags it.
Pass 2 (The Pattern Match): The detective looks for variations. If it sees "Study Design" or "How We Did It," it knows, "Ah, that's the Methods section!"
The "Fill-in-the-Blanks" Trick: Sometimes the detective misses a header. But it knows that if the page before says "Methods" and the page after says "Results," everything in between must be part of the "Methods" section. It fills in the gaps automatically.

Why this approach?
The authors chose this "dumb" but fast method over a "smart" AI because:

Speed: It processed 15 million papers in just a few hours on normal computers.
Transparency: You can look at the rules and say, "Oh, it labeled this 'Methods' because it saw the word 'Design'." With a complex AI, it's often a "black box" where you don't know why it made a choice.
Cost: It was incredibly cheap to run.

3. The Result: A Clean, Organized Library

After running the detective through the library, they filtered out the papers that were too messy (like a paper with only one section, which isn't a real scientific study).

The Final Dataset:

13.5 million papers are now neatly organized.
Every paper has its sections labeled (Introduction, Methods, Results, etc.).
It covers mostly science, tech, engineering, and math (STEM), with a huge chunk being medical and biology papers.

4. Did It Work? (The "Taste Test")

To make sure their detective wasn't just guessing, they ran two tests:

Human Test: They hired 32 smart people (with Master's degrees or PhDs) to manually label 100 papers.
AI Test: They asked a powerful Large Language Model (like a super-smart chatbot) to label 1,000 papers.

The Verdict:
The rule-based detective performed just as well as the humans.

The humans agreed with each other about 61% of the time (which is actually pretty low, showing that even experts find some sections confusing!).
The detective agreed with the humans at almost the same rate (58%).
This means the computer is as good as a human expert at sorting these papers, but it can do it millions of times faster.

5. What's in the Box?

The authors didn't just keep this to themselves. They released the entire map of the library for free.

They didn't give away the full text of the papers (because that's already available elsewhere).
Instead, they gave a "map" that tells you exactly where the "Introduction" starts and ends in every single one of those 13.5 million papers.

Summary

This paper is about creating a massive, free map of scientific writing. By using simple, fast rules instead of complex AI, they successfully organized millions of messy scientific papers into neat chapters. This allows other researchers to study how scientists write, spot trends in different fields, or analyze scientific communication on a scale that was previously impossible.

The Bottom Line: They built a fast, cheap, and reliable robot librarian that can sort the world's scientific papers better than a human could, and they handed the keys to everyone.

Technical Summary: Large-scale dataset of automatically classified rhetorical sections in scientific papers

Problem Statement
Scientific papers adhere to rhetorical structures (e.g., Introduction, Methods, Results, Discussion) that organize content for specific communicative purposes, often following the IMRaD format. While analyzing these structures at scale could enable granular studies of scientific writing patterns and discourse, manually annotating millions of papers is impractical. Existing automated approaches face significant limitations: they are often evaluated on small datasets (hundreds of papers), lack generalizability to million-scale corpora, and frequently rely on non-public code, preventing replication. Furthermore, prior methods struggle with the variability of section headers across disciplines, publishers, and document structures.

Methodology
The authors developed a rule-based classification algorithm designed for the scale and characteristics of the Semantic Scholar Open Research Corpus (S2ORC), which contains 15.6 million open-access scientific papers. The pipeline processes structured XML documents parsed via GROBID.

The core methodology involves a two-pass labeling strategy:

Exact Matching: The first pass uses exact string matching against a dictionary of common, generic section titles (e.g., "methods," "results") to capture headers with standard terminology.
Regular Expressions: The second pass employs regex patterns to identify broader semantic variations of section headers (e.g., "study design" mapped to "methods") while excluding overly general terms to prevent false positives.

Following the initial passes, the system applies label propagation to assign labels to unlabeled rows between two labeled headers, assuming they belong to the preceding section. Special handling is included for non-core headers (e.g., figure captions) and a "development" label for major top-level sections that do not match standard dictionaries but are identified via structural cues (e.g., numbered headers).

To ensure quality, the dataset undergoes filtering:

Papers must have at least two identified sections to be retained.
A threshold is applied to the maximum proportion of a paper a single section can occupy (excluding references), utilizing an Interquartile Range (IQR) rule ( $k=0.50$ ) to filter out papers where label propagation likely failed (e.g., where the bulk of the text is mislabeled as an introduction).

The final dataset aggregates text from consecutive rows with identical labels, producing section-level entries. The authors explicitly chose rule-based methods over machine learning or Large Language Models (LLMs) to ensure transparency, controllability, and computational efficiency, noting that the method's performance appears to approach the ceiling achievable using only header text without full-context semantic analysis.

Key Contributions

Dataset Scale: The authors present a dataset of section-level annotations for 13.5 million papers (86% of the initially classified S2ORC corpus), representing two orders of magnitude more annotated papers than prior datasets.
Validation: The study provides comprehensive validation using both human annotators (100 papers) and LLM-based annotation (1,000 papers).
Open Science: Unlike previous efforts, the code for the classification pipeline and the full dataset are publicly available, enabling replication and extension.
Discipline Focus: The dataset primarily covers STEM disciplines (approx. 75%), with strong representation in medicine (25.6%), biology, and engineering.

Results

Agreement Scores: The rule-based classifier achieved a Krippendorff's Alpha agreement of 0.5848 with human annotators. This is remarkably comparable to the agreement between independent human experts (0.6092) and between humans and the LLM (0.6293).
Performance by Section:
- High Performance: The algorithm performs strongly on standard IMRaD sections. For "Methods" (F1 = 0.79) and "Conclusion" (F1 = 0.91), the rule-based approach aligns closely with human experts. "Introduction" shows high recall (0.89), while "Methods" and "Results" show high precision (0.86).
- Challenges: Performance decreases for "Lit review" (F1 = 0.17) and "Development" categories. The "Lit review" label suffers from low recall (0.09) due to non-standard, descriptive titles. The "Development" label shows high recall (0.79) but low precision (0.12), indicating it captures structural boundaries but often over-captures sections that humans might classify differently (e.g., as methods or results).
Validation Metrics: The LLM inter-run agreement was 0.9261, while human inter-annotator agreement was 0.61, highlighting the inherent subjectivity and complexity of rhetorical classification even among highly educated native speakers.

Significance and Claims
The paper claims that this resource addresses key gaps in existing tools for analyzing the rhetorical structure of scientific literature by combining scale, validation, and public availability. The authors assert that the dataset enables large-scale computational studies of scientific discourse, the examination of discipline-specific writing conventions, and the tracking of changes in scientific communication over time at the section level.

The authors modestly note that the dataset is best suited for empirical papers with IMRaD-like structures and may not fully capture the organizational nuances of humanities and social sciences papers or non-standard document types. They emphasize that the rule-based approach offers an optimal balance between accuracy, transparency, and resource efficiency for corpus-scale annotation, achieving performance levels comparable to human experts for core empirical sections.