Technical Summary: Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning

Problem Statement

Large Language Models (LLMs) have achieved significant progress but at immense computational and data costs. While recent research has focused on improving data quality through filtering, reweighting, and optimizing data mixtures, the ordering of training data remains largely overlooked. Unlike human learning, which progresses from simple to complex concepts, LLMs are typically trained on randomly sampled data. This paper addresses the underexplored question: How can we build effective curriculum learning (CL) strategies for LLM pretraining to improve efficiency and performance?

Methodology

The authors conduct the first systematic investigation of curriculum learning in LLM pretraining, evaluating three distinct strategies across six difficulty metrics under three realistic training scenarios.

1. Difficulty Metrics

From an initial pool of 15 candidates, the authors selected six minimally correlated metrics representing distinct linguistic and information-theoretic properties:

• Compression Ratio: Measures information density and redundancy.
• Fertility: Measures tokenization complexity (subword tokens per word).
• Flesch Reading Ease: Captures text readability and comprehensibility.
• MTLD (Measure of Textual Lexical Diversity): Captures lexical richness, robust to text length.
• Number of Tokens: Sequence length at the token level.
• Perplexity: Model-centric metric reflecting linguistic uncertainty.

2. Curriculum Strategies

The study evaluates three ordering approaches:

• Vanilla Curriculum Learning: A strict ordering of a fixed dataset from easy to hard (Scenario 1: Limited Data).
• Pacing-Based Sampling: Sampling from a large data pool using pacing functions (Linear, Quadratic, Inverse Quadratic) to control the difficulty distribution over time (Scenario 2: Unlimited Data).
• Interleaved Curricula: Mixing difficulty levels within each training segment while maintaining a progressive structure (Scenario 2).

3. Experimental Setup

• Models: Decoder-only LMs (0.5B, 1B, and 3B parameters) based on the LLaMA3.2 architecture.
• Dataset: English subset of CulturaX (pre-cleaned, deduplicated).
• Scenarios:
  1. Limited Data: Training on a fixed 10B token set.
  2. Unlimited Data: Sampling up to 10B tokens from a larger pool using pacing.
  3. Continual Training: Using CL as a warmup phase followed by random sampling on additional data (up to 100B tokens).
• Evaluation: 8 benchmarks covering commonsense reasoning, language understanding, reading comprehension, and world knowledge (0-shot evaluation).

Key Results

1. Accelerated Convergence

Curriculum learning consistently accelerates convergence in early and mid-training phases. Models trained with curricula reached baseline performance 18–45% faster (in terms of training steps) compared to random sampling baselines.

• Effective Metrics: MTLD, Number of Tokens, and Compression Ratio were the most effective signals.
• Pacing: Linear pacing worked best for linguistic richness metrics (Compression Ratio, MTLD, Fertility), while Quadratic pacing yielded the best results for Flesch Reading Ease and Number of Tokens.

2. Sustained Performance Gains via Warmup

A critical finding is that using curriculum learning as a warmup strategy before standard random sampling yields sustained improvements.

• When a model trained with CL (e.g., using MTLD or Flesch Reading Ease) is continued with random sampling, it maintains a performance advantage over a purely random baseline.
• This "warmup" approach resulted in final performance improvements of up to 3.5% over the baseline.
• This benefit held true even when scaling up to 100B tokens, where the warmup model achieved a ~3.1% improvement.

3. Metric-Specific Observations

• Perplexity: Surprisingly, ordering by perplexity showed strong early gains but degraded in later phases, likely due to high-perplexity samples being noisy or low-quality.
• Flesch Reading Ease: Showed a unique pattern of constant improvement without signs of convergence, particularly with quadratic pacing.
• Interleaved Learning: Only showed consistent advantages for Compression Ratio and Number of Tokens; other metrics performed similarly to or slightly worse than the baseline.

4. Robustness and Scalability

• Ablation: The benefits of CL were confirmed to be robust across multiple runs and not due to chance.
• Ordering vs. Filtering: Experiments disentangling data selection from data ordering showed that while early-stage CL data acts as a filter, the ordering itself provides additional benefits beyond just selecting "good" data.
• Scale: The findings generalized to larger models (1B and 3B parameters) and larger datasets (up to 100B tokens).

Significance and Claims

The paper claims that data ordering is a practical, orthogonal mechanism to existing data selection methods (like filtering or pruning) for improving LLM pretraining efficiency.

• Low Overhead: The approach introduces negligible computational overhead. Difficulty metrics are lightweight, model-independent, and can be precomputed efficiently (costing only a few dollars and minutes for 10B tokens).
• Practical Mechanism: The "warmup" strategy is highlighted as the most robust and practical setting, offering a simple way to achieve both faster training and improved final performance without altering the model architecture or training algorithm.
• Generalizability: The results suggest that structured learning from easy to hard is beneficial even for large-scale LLM pretraining, challenging the assumption that random sampling is optimal for massive datasets.

The authors conclude that curriculum learning offers a "simple yet powerful mechanism" to enhance data- and compute-efficient LLM development, paving the way for more scalable model training. They note limitations regarding the focus on decoder-only English models and static difficulty scores, suggesting future work could explore adaptive, model-aware curricula.