Technical Summary: Reusing Overtrained Language Models Saturates Scaling

Problem Statement

Training Large Language Models (LLMs) from scratch is computationally expensive and time-consuming. To mitigate this, researchers increasingly employ model reuse strategies, such as Continual Pretraining (CPT) (adapting base models to new domains) and Model Growth (increasing model capacity by reusing base parameters). While promising, the effectiveness of reusing overtrained base models (those trained on massive token counts) remains unclear. Intuitively, highly optimized parameters in overtrained models may hinder effective exploration when adapting to new data distributions or expanding capacity, leading to a phenomenon known as "loss of plasticity." This paper investigates the scaling properties of these reuse strategies to determine if and how overtraining limits the benefits of second-stage pretraining.

Methodology

The authors conducted extensive controlled experiments involving over 450 runs across various model sizes (15M to 1B parameters) and datasets.

• Experimental Setup:

  • Architecture: Decoder-only transformers (LLaMA-style) with SwiGLU activations and rotary position embeddings.
  • Two-Stage Training:
    1. First Stage (Base Model): Pretrained on generic internet data (Slimpajama-DC, CommonCrawl portion) for $D_1$ tokens.
    2. Second Stage (Reuse):
       • CPT: Further training on domain-specific data (Code: Stack/StarCoder; Math: OpenWebMath).
       • Model Growth: Increasing capacity via Width Expansion (function-preserving initialization) or Depth-wise Stacking (copying layers).
  • Variables: The study systematically varied the number of first-stage tokens ($D_1$), second-stage tokens ($D_2$), and model size ($N$).
  • Training Configuration: Mixed precision (bfloat16), AdamW optimizer, and a Warmup-Stable-Decay (WSD) learning rate schedule.

• Scaling Law Formulation:
  The authors sought a functional form $L(D_1, D_2)$ describing the validation loss after the second stage. They tested several hypotheses against the empirical data:

  1. Multiplicative with Interaction: $L(D_1, D_2) = A D_1^{-\alpha_1} D_2^{-(\alpha_2 + \alpha_3 \log D_1)} + E$
  2. Additive: $L(D_1, D_2) = A D_1^{-\alpha_1} + F D_2^{-\alpha_2} + E$
  3. Hybrid: $L(D_1, D_2) = (A D_1^{-\alpha_1} + F) D_2^{-\alpha_2} + E$
  4. Continuous Scaling: $L(D_1, D_2) = A(D_1 + D_2)^{-\alpha} + E$ (assuming data continuity).

  The authors fitted these forms using Huber loss optimization and evaluated them via leave-one-out Root Mean Square Error (RMS).

Key Contributions

1. Empirical Scaling Law for Model Reuse: The paper establishes that the scaling exponent with respect to second-stage tokens ($D_2$) decreases logarithmically as the first-stage token count ($D_1$) increases. This is captured by the interaction term $\alpha_3 \log D_1$ in the scaling exponent.
2. Quantification of Saturation: The authors demonstrate that reusing overtrained base models leads to a predictable saturation in scaling efficiency. As $D_1$ grows, the marginal benefit of additional $D_2$ tokens diminishes significantly.
3. Joint Scaling with Model Size: The scaling law is extended to jointly incorporate model size ($N$), allowing for the extrapolation of performance to larger models and datasets.
4. Mechanistic and Theoretical Analysis:
   • Mechanistic: Analysis of gradient norms reveals that overtrained models exhibit larger gradient norms during second-stage training, suggesting increased optimization difficulty and a steeper loss landscape.
   • Theoretical: A toy model based on learning curve theory and maximum entropy principles is proposed to explain the logarithmic saturation of the scaling exponent, modeling feature discovery as a tree search where the effective branching factor is modified by first-stage training.

Key Results

• Best-Fit Model: The multiplicative scaling law with an interaction term (Equation 1) consistently achieved the lowest RMS error across all datasets (Code, Math) and methods (CPT, Model Growth) compared to additive, hybrid, or continuous scaling formulations.
• The Saturation Effect:
  • For a fixed $D_1$, loss follows a power law with respect to $D_2$.
  • However, the exponent $\alpha_{eff} = \alpha_2 - \alpha_3 \log D_1$ decreases as $D_1$ increases.
  • This implies that for highly overtrained base models (large $D_1$), the returns on additional second-stage training tokens become increasingly marginal.
• Crossover Point: By comparing the scaling laws of model reuse against training from scratch, the authors identify a "crossover" point. For small $D_2$, model growth is more efficient. However, as $D_2$ increases, the saturation of the reused model causes the "from-scratch" approach to eventually outperform it. This threshold depends on the initial investment ($D_1$) and model size.
• Gradient Norms: Overtrained models show larger gradient norms during reuse, providing a mechanistic explanation for the observed saturation (i.e., the model is in a highly sensitive region of parameter space).
• Limitations: The scaling law predicts a theoretical crossover where from-scratch training becomes superior. However, in experiments with extremely large $D_2$ (up to 2000x token-to-parameter ratio), the data entered a noise floor, preventing definitive empirical confirmation of the crossover for larger models.

Significance and Claims

The paper claims to provide practical insights for efficient language model training by quantifying the trade-offs in multi-stage pretraining.

• Guidance for Practitioners: The derived scaling laws offer a data-driven strategy to decide when to train from scratch versus reusing a base model. If the target training budget ($D_2$) exceeds a certain threshold relative to the base model's overtraining ($D_1$), training from scratch may be more computationally optimal.
• Fundamental Trade-off: The work highlights a fundamental trade-off: the more extensively a base model is pretrained, the less benefit additional pretraining provides. This challenges the assumption that reusing any pretrained model is always superior to training from scratch.
• Scope: The authors explicitly state that their findings are robust for current scalable training methodologies (up to 1B parameters) but acknowledge that the behavior at extremely large scales (post-crossover) remains an open problem due to computational constraints and noise. They do not claim to have eliminated saturation effects but rather to have modeled and understood them to mitigate negative consequences.

The paper concludes that while model reuse is a promising strategy, its efficacy is bounded by the overtraining of the base model, and this saturation is a predictable, quantifiable phenomenon governed by specific scaling laws.