Technical Summary: Characterizing Narrative Content in Web-scale LLM Pretraining Data

Problem Statement

While narratives are a fundamental mode of human communication and a primary use case for Large Language Models (LLMs), the narrative composition of web-scale pretraining corpora remains largely unexplored. Existing research has examined pretraining data along dimensions such as quality, toxicity, and topic distribution, but has not systematically analyzed narrative structure. This gap is critical because current data curation practices treat sources (e.g., books, Reddit, news) as narratively homogeneous. Consequently, training mixtures may disproportionately reduce a model's narrative exposure or skew its event structures and perspectives without the curation team's knowledge. The authors posit that narrative qualities are likely unequally distributed across sources and topics in ways that current practices neither measure nor account for.

Methodology

The authors present a comprehensive framework to measure narrative structure at scale using DOLMA, a 3-trillion-token open pretraining corpus. Their approach involves four main stages:

1. Theoretical Framework and Annotation Dimensions

Drawing on narrative theory (Herman, 2011; Piper et al., 2021), the authors define narrative not as a binary classification but as a continuous, multidimensional structure. They operationalize this into 11 interpretable dimensions grouped into three core elements:

• Agency (5 dimensions): Focalization (perspective), Internal Emotion, Internal Cognition, Change of State, and Conflict. These are rated on a 1–5 Likert scale.
• Setting (4 dimensions): Concreteness, Temporal Grounding, Spatial Grounding, and Sensory Detail. These are also rated on a 1–5 Likert scale.
• Event Relations (3 dimensions): Temporal ordering, Causal relations, and Event Density (events per token). These are derived from event trigger detection and relation extraction.

2. Sampling and Data Construction

To address the heterogeneity of web text, the authors developed a four-step sampling pipeline from DOLMA:

1. Extraction: Drawing ~17M three-sentence passages from ~5M documents.
2. Scoring: Using a DeBERTa-based binary classifier to filter for coherent narrative content.
3. Topic Classification: Applying the WEBORGANIZER classifier to Common Crawl sources to balance subject domains.
4. Stratified Sampling: Creating two datasets:
   • Gold Dataset: 400 passages manually annotated by human experts for validation.
   • NARRADOLMA: A large-scale dataset of ~3 million passages (spanning ~785K documents) labeled via a distillation pipeline.

3. Model Development: NARRABERT

To scale annotation beyond human capacity, the authors utilized a "validation-first" workflow:

• LLM Labeling: They compared three LLMs (Claude Sonnet 4.6, Qwen3, Gemma4) against human annotations. Gemma4 was selected for large-scale labeling due to cost-effectiveness and open-source availability.
• Knowledge Distillation: The 5,000 passages labeled by Gemma4 were used to train NARRABERT, a pair of RoBERTa-based classifiers. One encoder handles the 9 Likert-scale dimensions (Agency and Setting) as regression tasks, while a dedicated encoder handles Event Relations as binary classification tasks.
• Validation: NARRABERT was validated against a held-out human-annotated set, achieving agreement scores comparable to the LLM teachers for Agency and Setting, though with slightly lower performance on Event Relations due to class imbalance.

4. Analysis

The authors applied NARRABERT to the full NARRADOLMA corpus to generate 12-dimensional narrative feature vectors for each document. They analyzed these vectors using Principal Component Analysis (PCA) to identify dominant modes of variation and examined the distribution of these features across different sources and topics.

Key Contributions

1. Annotation Framework: A theory-grounded framework operationalizing 11 fine-grained narrative dimensions across Agency, Setting, and Events.
2. NARRABERT: An efficient classifier capable of predicting all 11 narrative dimensions, validated against both human and LLM annotations.
3. NARRADOLMA: A new dataset comprising ~3 million labeled passages sampled from the DOLMA corpus using a principled stratified pipeline.
4. Large-Scale Analysis: The first empirical mapping of narrative structure in pretraining data, revealing that narrativity forms a continuous, multidimensional structure rather than a single scalar quantity.

Results

The analysis of NARRADOLMA yielded three core empirical findings:

1. Measurability: Narrative structure is measurable at scale across extremely heterogeneous data.
2. Multidimensionality: Narrative qualities covary into three interpretable principal components (axes):
   • Interiority (PC1): Driven by focalization, emotion, and cognition.
   • Grounded Eventfulness (PC2): Driven by change of state, conflict, event density, and temporal/spatial grounding.
   • Storyworld Texture (PC3): Driven by concreteness, sensory detail, and spatial grounding.
3. Unequal Distribution: These narrative qualities are not uniformly distributed across sources or topics.
   • Source Variation: Sources like Reddit and Gutenberg are high in Interiority but low in other dimensions, while Wikipedia and Crime/Law are high in Grounded Eventfulness but nearly absent in Interiority.
   • Topic Variation: Topics like Food & Dining and Travel score high on Storyworld Texture, while Software and Finance score low.
   • Internal Diversity: Narrative-rich categories (e.g., Reddit, Literature) show high internal variance, indicating that source-level labels are too coarse to capture narrative diversity. Upweighting a "narrative" source does not uniformly increase all narrative qualities.

Significance and Claims

The paper claims that narrative structure should be treated as a measurable, multidimensional property of pretraining data, complementing existing axes like quality and topic distribution. The authors argue that current data curation practices are too coarse; decisions made at the source level (e.g., which corpora to include) fail to account for the narrative diversity present within those sources.

By providing a framework, dataset, and models to characterize narrative distribution, this work lays the foundation for:

• Understanding how data composition affects narrative reasoning tasks.
• Investigating the "creativity deficit" in LLM storytelling by linking it to specific training mixtures.
• Future controlled experiments (e.g., using RegMix) to establish causal links between pretraining narrative composition and downstream model capabilities.

The authors remain modest regarding causal claims, noting that they characterize and map narrative structure but do not yet establish direct causal links to model behavior, identifying this as a direction for future work. They also acknowledge limitations, including the stratified nature of the sample (over-representing narrative content relative to raw DOLMA) and the intrinsic difficulty of annotating certain dimensions like temporal ordering.