Technical Summary: Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation

Problem Statement

Pretraining data curation is a fundamental component of Large Language Model (LLM) development. Current research focuses heavily on quality filtering of large web corpora, often reducing datasets to a binary classification: texts that pass quality filters are deemed valuable, while others are discarded as useless or detrimental. However, a detailed understanding of how specific types of texts contribute to model performance remains lacking. Existing curation methods rely on simple heuristics, quality signals, or LLM-based labeling that may overlook the nuanced effects of linguistic variation. This paper addresses the gap in understanding how different registers (or genres)—a standard concept in corpus linguistics used to model linguistic variation across situations—affect LLM performance when used as pretraining data.

Methodology

The authors conducted an ablation study training small generative models (1.71 billion parameters, Llama architecture) on datasets curated to contain texts from specific register classes.

• Data Source: The study utilized the HPLT version 2.01 dataset, specifically the deduplicated English subset.
• Register Classification: Documents were assigned register labels using a multi-label classifier based on XLM-RoBERTa-Large, trained on the CORE register scheme. This scheme includes 9 main registers and 25 subregisters.
• Experimental Design:
  • Individual Registers: Models were trained on 100 billion tokens (or all available data for smaller classes) from specific register classes, including main registers (e.g., How-to-Instructions, Narrative, Opinion), subregisters (e.g., News, Description), and a hybrid class (Instructive-Informational).
  • Combinations: The authors created combined datasets by sampling equally from top-performing individual registers to test if mixing registers could yield improvements over single-register or full-dataset training.
  • Baselines: Comparisons were made against a model trained on a random sample of the full HPLT v2 deduplicated dataset and a model trained on a sample of the FineWeb dataset.
• Evaluation: Models were evaluated using the LightEval harness in a zero-shot setting across eight standard benchmarks: HellaSwag, WinoGrande, PIQA, SIQA, OpenBookQA, ARC Easy, ARC Challenge, CommonsenseQA, and MMLU.

Key Results

The study reveals that the register of pretraining data has a substantial and non-uniform impact on model capabilities.

• Register Impact on Performance:

  • High Performers: Models trained on How-to-Instructions (HI), Informational Description (dtp), and Opinion (OP) registers achieved the highest average accuracies among single-register models. Surprisingly, the Opinion register (covering reviews and opinion blogs) yielded the 4th best overall performance, outperforming Narrative and Informational Description.
  • Low Performers: Registers such as Lyrical (LY), Machine Translation (MT), and Informational Persuasion (IP) resulted in the poorest performance. Notably, the News (ne) subregister performed among the worst, despite being a common source for pretraining.
  • Hybrids: The Instructive-Informational (HI-IN) hybrid model outperformed the model trained on the full HPLT v2 dataset, suggesting that hybrid documents possess qualities that yield better performance than either constituent register alone.

• Task-Specific Strengths:

  • How-to-Instructions excelled at physical reasoning (PIQA) and sentence completion (HellaSwag) but performed poorly on world-knowledge benchmarks (MMLU, CommonsenseQA).
  • Narrative boosted performance on social interaction tasks (SIQA) and pronoun resolution (WinoGrande) but struggled with scientific questions (ARC Challenge).
  • Opinion significantly improved performance on reasoning tasks involving social situations and commonsense (WinoGrande, SIQA, CommonsenseQA).

• Combination Effects:

  • While a model trained on the entire unfiltered dataset outperformed those trained on single registers, combining specific high-performing registers led to further gains.
  • The combination of Instructive-Informational, How-to-Instructions, Description, and Opinion (HI-IN-HI-dtp-OP) achieved the highest average accuracy, surpassing both the full HPLT v2 and FineWeb baselines.
  • Conversely, adding registers like Interactive Discussion or Spoken to high-performing combinations resulted in performance drops, suggesting these registers may contain qualities detrimental to the specific benchmarks used or dilute the benefits of more effective registers.

Significance and Claims

The paper claims to be the first study to utilize register classification to curate pretraining datasets and investigate the effect of register on LLM performance. The authors assert that:

1. Register is a Critical Variable: Register is an important explainer of model variation, challenging the notion that all "high-quality" text is equally beneficial for all tasks.
2. Beyond Binary Filtering: The findings suggest that data curation should move beyond simple binary quality flags. Specific registers, such as Opinion, are highly beneficial and should be considered for inclusion, while others, like News, may be less effective than assumed.
3. Strategic Data Selection: Combining registers with complementary strengths (e.g., How-to-Instructions for physical reasoning and Opinion for social reasoning) can lead to major improvements in model performance, potentially outperforming models trained on massive, uncurated web corpora.
4. Transparency: The study advocates for using register classification as a tool to increase transparency in foundation models, allowing researchers and users to understand the linguistic composition of training data and its specific impact on model capabilities.

The authors maintain modesty regarding their claims, noting that their experiments were limited to English and that the benchmarks used may not fully capture the capabilities of models trained on specific registers (e.g., Lyrical). They position this work as an initial step in bridging corpus linguistics frameworks with LLM pretraining data curation.