Technical Summary: Scaling Sparse Language Models with Repeated Training

Problem Statement

Recent advancements in Large Language Models (LLMs) have relied heavily on scaling laws that assume an abundance of high-quality training data. However, the supply of unique, high-quality human data is growing at a modest rate (1.03× per year), while pre-training data consumption has accelerated (4× per year). This mismatch has ushered in a data-constrained regime, where models must be trained on limited unique tokens, necessitating multi-epoch training (data repetition).

In this regime, standard dense scaling laws exhibit diminishing returns; repeating the same data leads to rapid saturation and overfitting. While prior work has explored sparse scaling (e.g., Mixture-of-Experts or Dynamic Sparse Training) in data-sufficient regimes, and data-constrained scaling in dense regimes, the interaction between sparsity and data repetition remains unexplored. Specifically, it is unclear whether sparse optimization can alter the saturation dynamics of repeated data or improve the trade-offs between model capacity, data reuse, and computational cost when unique data is scarce.

Methodology

The authors conduct a systematic empirical study of Dynamic Sparse Training (DST) under data-constrained conditions. Their methodology involves:

1. Experimental Setup:

   • Models: LLaMA-2 family architectures trained on the C4 dataset.
   • Scale: Experiments cover active parameter counts ($N$) from 15M to 960M, with dense-equivalent sizes up to 1.92B parameters (validated via extrapolation up to 7.68B).
   • Sparsity: Levels ranging from 0% (dense) to 93.75% (16× compression).
   • Data Budgets: Unique token budgets ($U_d$) of 0.52B, 1.3B, and 2.6B tokens.
   • Repetition: Training for 1, 2, 4, 8, and 16 epochs, resulting in total training tokens up to 41.6B.
   • Algorithm: Dynamic Sparse Training using the Sparse Evolutionary Training (SET) algorithm with magnitude pruning and random regrowth.

2. Scaling Law Formulation:
   The authors propose a new Sparse Data-Constrained Scaling Law that extends the Chinchilla and Muennighoff et al. (2023) formulations. The loss function $L$ is modeled as:
   $$L(N', D', S) = \frac{A}{(N')^\alpha} + \frac{B}{(D')^\beta} + E$$
   Where:

   • $N'$ and $D'$ are effective parameter and data counts, respectively, which account for diminishing returns.
   • $S$ is the sparsity level.
   • The effective data size $D'$ incorporates a sparsity-dependent saturation threshold $R^*_d(S)$:
     $$D' = U_d + U_d R^*_d(S) \left(1 - e^{-R_d/R^*_d(S)}\right)$$
   • The saturation threshold $R^*_d(S)$ is modeled as a function of sparsity: $R^*_d(S) = R^*_d (1 + \lambda_1 S + \sigma_1 S^2)$.
   • Similarly, the effective parameter count $N'$ includes a sparsity-dependent factor to model the trade-off between active parameters and dense-equivalent capacity.

3. Analysis:
   The study analyzes Iso-Loss and Iso-FLOPs curves to determine optimal resource allocation (model size vs. training epochs) under fixed data and compute budgets. They distinguish between two definitions of compute: Sparse FLOPs (operations on non-zero parameters) and Dense FLOPs (cost of training an equivalent dense model).

Key Contributions and Findings

1. A New Scaling Law for Sparse, Data-Constrained Training

The paper introduces a scaling law that accurately predicts performance across compute and data budgets by modeling loss as a function of active parameters, unique tokens, data repetition, and sparsity.

• Validation: The model achieves a validation $R^2$ of 0.914 on held-out configurations, demonstrating that the formulation captures the underlying structure of sparse scaling without overfitting.

2. Delayed Data Saturation

The most significant empirical finding is that sparsity delays the saturation of repeated data.

• Mechanism: In dense training, repeated exposure to limited data quickly leads to diminishing returns. Sparse training distributes updates across a larger pool of weights, reducing the frequency with which any single parameter is updated on the same data.
• Quantitative Impact: The data saturation threshold $R^*_d$ increases with sparsity, peaking around $S \approx 0.5$. At $S=0$, $R^*_d \approx 4.4$; at $S=0.5$, $R^*_d \approx 6.9$. This implies sparse models can tolerate more data repetition before performance plateaus, making multi-epoch training more effective.

3. Resource Allocation and Trade-offs

The study identifies distinct optimal strategies for sparse versus dense training under data constraints:

• Allocation Shift: In dense training, additional compute is best allocated to increasing model size. In sparse training (specifically at moderate sparsity), additional compute is better allocated to longer training (more epochs) rather than increasing the parameter count.
• Parameter Efficiency: Sparse training achieves better parameter-to-token ratios. For a 1.3B token budget, sparse training can achieve a 3× smaller parameter-to-token ratio compared to dense training, improving inference efficiency.

4. Optimal Sparsity Levels

The authors define two notions of optimal sparsity:

• Loss-Optimal Sparsity: The sparsity that minimizes validation loss for a fixed data budget. This remains relatively stable and moderate ($S \approx 0.45 - 0.5$) across different data scales.
• Compute-Optimal Sparsity: The sparsity that matches the performance of the optimal dense model while minimizing training compute. This value increases with data scale, shifting from $S \approx 0.6$ at 1.3B tokens to $S \approx 0.7 - 0.75$ at larger scales (13B–130B tokens).
• Efficiency Gains: At compute-optimal sparsity, sparse models match dense-model accuracy while using approximately 8× to 10× fewer training FLOPs.

Significance and Claims

The paper claims to provide the first systematic empirical characterization of Dynamic Sparse Training under data-constrained scaling regimes. Its primary significance lies in reframing sparsity not merely as a tool for inference efficiency or model compression, but as a mechanism for improving scaling trade-offs when data is the primary bottleneck.

Key claims include:

• Sparsity fundamentally alters the interaction between model capacity and data repetition, delaying the point of diminishing returns.
• Under data scarcity, the optimal strategy shifts from "bigger models" to "longer training on sparse models."
• The proposed scaling law offers a predictive framework for allocating compute and data in future data-constrained pre-training scenarios.

The authors acknowledge limitations, noting that their experiments do not yet reach the scale of frontier LLMs (beyond 7.68B dense-equivalent parameters) and that hardware-level inefficiencies in sparse implementations may reduce realized speedups. However, they assert that their findings provide a foundational understanding for training efficient LLMs in an era of limited data.