Technical Summary: Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention

Problem Statement

While empirical evidence suggests that larger models excel at tasks where smaller models perform at random chance, the precise mechanism driving this phenomenon remains debated. Prior work often attributes this to "emergent abilities" or superior sample efficiency. However, this paper posits a more fundamental limitation: even under asymptotic training regimes (infinite data), smaller models may fail to learn specific parts of a data distribution that larger models can. The core question is whether this gap is due to a lack of representational capacity (expressivity) or a data-centric bottleneck arising from learning dynamics, specifically competition for resources and gradient interference.

Methodology

The authors employ a multi-faceted approach combining phenomenological theory, synthetic toy experiments, and large-scale pretraining validation.

1. Phenomenological Argument

The paper begins by analyzing power-law scaling laws ($L(N, D) = L_0 + A N^{-\alpha} + B D^{-\beta}$). By comparing the compute-optimal loss ($L_C$) with the asymptotic loss ($L_\infty$), the authors define two regimes:

• Learnable via data scaling: A smaller model can eventually match a larger model's loss if given infinite data.
• Learnable via model scaling: A smaller model fails to reach the larger model's loss even with infinite data. The authors argue that the latter regime exists in practice, implying a fundamental limitation in smaller models regarding specific data distributions.

2. Synthetic Toy Setup (Multi-Task Linear Regression)

To isolate mechanisms, the authors construct a synthetic multi-task learning environment:

• Data: A mixture of $K$ linear regression tasks with power-law distributed frequencies ($\pi_k \propto k^{-\beta}$) and power-law task spectra (complexity).
• Model: A shared encoder with width $N$ and task-specific decoders.
• Analysis: The authors derive theoretical bounds on which features are learned based on "utility" (frequency $\times$ spectral magnitude). They prove that an optimal width-$N$ encoder spans the top-$N$ eigenspace of the weighted covariance matrix.
• Key Mechanism: They analyze gradient dynamics to show that frequent tasks dominate the gradient updates. In smaller models, updates for frequent tasks overwrite the features of rare tasks before the next rare observation arrives ("update-and-forget" loop). Larger models, having more parameters, reduce this interference, allowing them to retain and accumulate signals from rare tasks.

3. Large-Scale Validation (OLMo Pretraining)

To verify these claims in a realistic setting, the authors pretrain OLMo models (ranging from 4M to 4B parameters) on the Dolma v1.7 corpus.

• Task Injection: They inject novel, non-natural tasks (Comparison and Modular Addition) at controlled, low frequencies.
• Metrics: They evaluate behavioral performance (loss/accuracy), representational evidence (localizing task features via Distributed Alignment Search and Fourier analysis), and gradient evidence (measuring interference between task and non-task gradients).

Key Contributions and Results

1. Scaling Enables Learning of Rare and Complex Tasks

Theoretical analysis and synthetic experiments demonstrate that larger models learn tasks with low frequency or high complexity (slow spectral decay) that smaller models fail to learn, even asymptotically.

• Result: In synthetic settings, increasing model width preferentially improves performance on low-frequency tasks because it allows the model to retain lower-utility features that smaller models must discard to accommodate high-utility (frequent) features.

2. Reduced Gradient Interference and Memory Retention

The paper identifies gradient interference as the primary bottleneck for smaller models.

• Mechanism: In small models, the gradient updates from frequent tasks are strong enough to overwrite the weak updates from rare tasks. This creates a "update-and-forget" dynamic where rare task features are never consolidated.
• Result: Larger models possess sufficient capacity such that the gradient updates for common tasks become weak (low norm) once those features are learned. This leaves "spare" capacity to retain rare-task features.
• Evidence: Experiments with matched-frequency injection (varying the gap between rare task observations) show that small models lose rare-task signals rapidly between injections, while larger models retain and accumulate these signals, eventually leading to successful learning.

3. Empirical Validation in Real LLMs

The findings from the toy setting hold true in OLMo pretraining:

• Behavioral: Only larger OLMo models (1B and 4B) successfully learn the injected rare tasks (Comparison and Modular Addition) at low frequencies; smaller models (4M, 20M, 300M) fail to generalize.
• Representational: Larger models embed more task-relevant features (e.g., global token order for comparison, Fourier modes for addition) in their representations.
• Gradient: Analysis of gradient cosine similarity reveals that in larger models, the gradient direction for the injected task is less interfered with by non-task (language modeling) gradients. In smaller models, non-task gradients frequently collide with the task direction, causing interference.

Significance and Claims

The paper offers a data-centric account of why larger models learn more, shifting the focus from static model expressivity to dynamic learning processes.

• Beyond Expressivity: The authors argue that the advantage of larger models is not merely that they can represent a task (as smaller models might theoretically be able to), but that they can learn it from a given data mixture under gradient-based optimization.
• Role of Memorization: The work reframes "memorization" not as a failure mode but as a necessary mechanism for learning rare tasks. By retaining specific task instances longer (due to reduced interference), larger models can accumulate signals across batches to eventually form abstract, generalizable task structures. This aligns with observations of "grokking" phenomena.
• Practical Implications: The findings suggest that simply increasing model size is not the only path to capability; understanding the interplay between data frequency, task complexity, and model capacity is crucial. It implies that data mixtures could be intentionally designed (e.g., adjusting task frequency) to elicit specific capabilities more efficiently than scaling parameters alone.

The authors maintain modesty, acknowledging that their explanation complements rather than replaces other theories (like sample efficiency or pure expressivity) and that their empirical validation, while strong, leaves some analytic gaps regarding extreme scaling regimes.