Technical Summary: Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics

Problem Statement

Pretraining large language models (LLMs) typically involves a single pass over massive corpora, where data ordering is often randomized. While curriculum learning (ordering data from easy to hard) is a established concept, its efficacy and mechanistic impact on LLM pretraining remain unclear. Existing literature presents mixed results: some studies suggest gains at fixed compute budgets, while others find diminishing returns at scale. A critical ambiguity exists: do curricula alter the sequence of latent learning phases a model traverses, or do they primarily change which data the model encounters within a shared sequence of phases? Furthermore, it is unknown how data ordering interacts with model capacity constraints, specifically the "softmax bottleneck," where small models may suffer from output-head spectral saturation (degeneration of the singular-value structure) late in training.

Methodology

The authors conduct a controlled empirical study using the Pythia model suite (14M to 1B parameters) trained on 300 billion tokens from The Pile dataset. The core experimental design holds the set of 2048-token training samples fixed, varying only the ordering of these samples.

Curricula Design:
Three linguistically motivated curricula are tested, where samples are sorted in ascending order based on specific scores:

1. Age-of-Acquisition (AoA): Based on the age at which words are typically learned.
2. Word Frequency: Based on SUBTLEX-US frequency using the Zipf scale.
3. Verb Variation (VV): Measures the diversity of verb types within a sample.
   These are compared against a Random ordering baseline. A Reverse-Order VV control is also used to test the importance of directionality.

Analysis Framework:
The study employs three primary diagnostic lenses to analyze learning dynamics:

1. Latent Phase Analysis: Hidden Markov Models (HMMs) are fitted to training trajectories (using 14 checkpoint-level metrics like weight norms and singular values) to identify discrete learning phases. A joint HMM is trained across all orderings to ensure phases are directly comparable.
2. Gradient Noise Scale (GNS): Measures the signal-to-noise ratio of stochastic gradients. The authors hypothesize that curricula delaying high-variance examples may reduce GNS, improving stability.
3. Singular Entropy: A diagnostic for the "softmax bottleneck." It measures the concentration of the singular-value spectrum of the output head. High entropy indicates a saturated, degenerate spectrum, while low entropy suggests a more uniform distribution.

Scope:
Experiments span five model sizes (14M, 31M, 70M, 160M, 410M) and a 1B parameter run (Random vs. VV only due to compute constraints). Models are trained for up to 300B tokens to observe late-stage dynamics often missed in shorter training regimes.

Key Results

1. Shared Latent Phase Structure
The HMM analysis reveals that training trajectories under different data orderings share a consistent sequence of latent phases (P0 through P4). Curricula do not create new macro-phases or alter the fundamental transition structure. Instead, they primarily change phase occupancy (the time spent in each phase) and the specific data exposure within those phases. For instance, the Frequency curriculum spends more time in phase P2, while AoA and VV spend more time in P3.

2. Stability Diagnostics in Small Models
In smaller models (14M–70M), Random ordering exhibits significantly higher Gradient Noise Scale (GNS) and sharper late-stage spikes in singular entropy compared to the linguistic curricula.

• GNS: Random ordering produces noisier gradients relative to the signal, particularly in later training phases.
• Singular Entropy: Random ordering leads to a "spiked" spectrum (high entropy) in models up to 160M parameters, indicative of output-head saturation. Linguistic curricula maintain lower entropy, avoiding this saturation.
• Scale Dependence: These stability differences narrow significantly at 410M and are minimal at 1B, suggesting the effects are most pronounced when model capacity is constrained.

3. Capability-Specific Effects
Ordering effects on downstream accuracy are sparse and capability-specific rather than globally uniform:

• Verb Variation (VV): Improves accuracy on wh-questions (object-gap) and causatives, particularly in smaller models.
• Frequency: Underperforms Random on superlative quantifiers and NPI scope probes.
• AoA: Shows no consistent cross-probe advantages or regressions.
• Directionality: Reversing the VV order (descending) removes the late-training accuracy advantage of the ascending curriculum, confirming that the direction of ordering matters.

4. Domain Mixture Entanglement
Sorting by linguistic scores inadvertently alters the source-domain mixture over time (e.g., Frequency places GitHub content early and Pile-CC late). While the scores correlate with empirical sample loss, the authors note that the observed effects are a conflation of linguistic difficulty, source-domain exposure, and their interaction, rather than a pure isolation of linguistic difficulty.

Significance and Claims

The paper claims to resolve the mechanistic ambiguity of curriculum learning in LLM pretraining by demonstrating that:

• Curricula modify exposure, not phases: The primary effect of data ordering is changing which data is seen during specific, shared learning phases and how long the model remains in them, rather than altering the sequence of phases themselves.
• Stability over new capabilities: In capacity-constrained models, linguistic curricula improve training stability (lower GNS, reduced spectral saturation) rather than fundamentally changing the learning trajectory. This stability helps preserve specific capabilities (like argument structure) that might otherwise degrade in small models trained with random ordering.
• Scale sensitivity: The benefits of these specific linguistic curricula are scale-dependent, being most critical for smaller models where the softmax bottleneck is a limiting factor. At larger scales, the ordering effects on stability and performance diminish.

The authors conclude that while these curricula offer measurable stability benefits for smaller models, they do not create new developmental stages. The results suggest that for small models, managing the order of data exposure can mitigate late-stage saturation, but the effect is a modulation of existing dynamics rather than a transformation of the learning process.