Technical Summary: When is Warmstarting Effective for Scaling Language Models?

Problem Statement

Training state-of-the-art deep learning models requires immense computational resources. As models scale from millions to billions of parameters, a critical question arises: can the compute invested in training smaller models be reused to accelerate the training of larger models? This technique, known as Warmstarting (WS) or Model Growth, involves initializing a larger model from a trained smaller checkpoint. Despite significant research interest, WS has seen limited practical adoption in production pipelines.

The authors attribute this lack of adoption to three primary barriers:

1. Implementation Complexity: Existing methods often require architecture-specific adaptations (e.g., specific cloning rules for function preservation) that do not generalize to modern components.
2. Unclear Growth Limits: Practitioners lack guidance on how much a checkpoint can be grown before WS becomes counterproductive compared to training from scratch.
3. Hyperparameter Opacity: The interaction between WS methods, hyperparameters, and scaling budgets (model size vs. data tokens) is under-explored, creating tuning overhead that negates efficiency gains.

Furthermore, prior literature often emphasizes preserving the base model's exact output at initialization (Function Preservation, or FP), which may unnecessarily constrain operator design.

Methodology

1. Theoretical Framework: Shrink-Zero-Perturb (SZP)

The authors propose a generalized, architecture-agnostic growth operation called Shrink-Zero-Perturb (SZP). They extend the "Shrink-and-Perturb" (SnP) formulation from continual learning to model growth. The target weights $\theta_{target}$ are derived as:
$$\theta_{target} = \lambda_{shrink} \cdot G(\theta_{base}) + N(0, \sigma_{perturb}^2)$$
Where:

• $G$ is a growth function (e.g., zero-padding or cloning) that maps base weights to the larger target shape.
• $\lambda_{shrink}$ scales down the inherited weights to reduce the dominance of the base representation.
• $\sigma_{perturb}$ adds Gaussian noise to break symmetry and activate newly added neurons.

The authors argue that strict Function Preservation (FP)—where the grown model's initial output exactly matches the base model—is not required for effective warmstarting. Instead, SZP offers a principled baseline that balances signal preservation with the flexibility to utilize new capacity.

2. Experimental Setup

The study employs two primary experimental setups:

• Synthetic Regression (MLPs): A controlled testbed using width-scaled Multi-Layer Perceptides (MLPs) on a synthetic regression task with a power-law Fourier spectrum. This allows for dense hyperparameter sweeps to analyze tuning strategies and growth limits.
• Language Modeling (LLMs): Decoder-only GPT-2 style models (and LLaMA-style in ablations) trained on the SLIMPAJAMA corpus. Experiments cover model sizes from 14M to 1.2B parameters.

Key Experimental Controls:

• Hyperparameter Transfer: The authors utilize $\mu$P (Maximal Update Parameterization) to transfer hyperparameters (learning rate, batch size) from the base model to the target model, avoiding the confounder of re-tuning costs.
• Training Budgets: Models are trained for fixed token-per-parameter budgets (10, 20, and 30 tokens/parameter), with 20 tokens/parameter representing the Chinchilla compute-optimal frontier.
• Comparisons: SZP is compared against training from Scratch and the standard Net2Net (a function-preserving baseline).

Key Contributions

1. Generalization to SZP: The authors introduce SZP as a simple, architecture-agnostic growth strategy. They demonstrate that exact function preservation at initialization is not necessary for strong final performance; in fact, SZP often outperforms complex FP-based operators like Net2Net, particularly in language models.
2. Empirical Upper Bound on Growth Factor ($g_{upper}$): The paper establishes an empirical upper bound on the growth factor $g$ (ratio of target to base parameters). Beyond $g_{upper}$, training the grown model from scratch converges as fast as (or faster than) warmstarting it. This bound is consistent across MLPs and LLMs.
3. Scaling Law Predictions: By fitting scaling laws over the observations, the authors provide predictive guidance on when WS is effective. They characterize the "efficiency window" where WS yields convergence speedups over training from scratch, showing that these gains are most pronounced under specific token-per-parameter budgets.

Key Results

1. SZP vs. Net2Net vs. Scratch

• Convergence Speed: SZP consistently converges faster than training from scratch and outperforms Net2Net across various model widths and growth factors.
• Function Preservation is Not Required: While Net2Net (FP-based) recovers the base model's loss quickly, it often struggles to escape the initial subspace in larger growth scenarios, leading to lower effective ranks and slower long-term convergence compared to SZP.
• Robustness: SZP is more robust to growth factors and does not require architecture-specific rules for weight cloning or noise injection.

2. The Growth Factor Limit ($g_{upper}$)

• Existence of a Bound: The authors identify a clear $g_{upper}$ beyond which WS loses its advantage. For many setups, this bound is approximately $g = 2$.
• Diminishing Returns: As the growth factor increases, the base model occupies a diminishing fraction of the target architecture. The initialization increasingly resembles a random one, and the constraints of the inherited structure outweigh the benefits.
• Budget Dependency: The effectiveness of WS is tightly coupled with the training budget. WS gains are most significant for budgets $\le$ 20 tokens/parameter. As the budget increases (approaching or exceeding the Chinchilla optimal frontier), the advantage of WS diminishes, and training from scratch becomes more efficient.

3. Scaling Law Analysis

• The authors fit power-law models $L(N, D) = E + A N^{-\alpha} + B D^{-\beta}$ to predict loss as a function of parameters ($N$) and tokens ($D$).
• Iso-Loss Contours: Visualizing the difference in loss between WS and Scratch reveals a boundary. For a fixed model size, training with fewer tokens/parameter favors WS, while training with more tokens/parameter favors Scratch.
• Predictive Guidance: The fitted laws allow practitioners to predict the crossover point where WS ceases to be beneficial for a given growth factor and budget.

Significance and Practical Guidelines

The paper provides a systematic, first-principles investigation into the efficacy of warmstarting, moving beyond anecdotal evidence to provide empirical limits and guidelines. The authors claim the following practical takeaways:

• Model Scaling: If only one model run is feasible, a 2$\times$ growth factor is the most reliable scaling strategy for warmstarting.
• Efficiency Window: Warmstarting should be prioritized over training from scratch when the training budget is at or below the compute-optimal scaling (approximately 20 tokens/parameter).
• Hyperparameter Transfer: $\mu$P serves as a principled strategy for transferring hyperparameters in warmstarting, particularly when tuning budgets for larger scales are constrained.
• Standardized Reporting: The authors propose that future literature should explicitly report the empirical $g_{upper}$ (the growth factor where WS matches Scratch performance) to facilitate fair comparisons and prevent the adoption of ineffective growth strategies.

The study concludes that while warmstarting offers resource savings, its utility is bounded. The proposed SZP method and the identified limits on growth factors provide a framework for practitioners to decide when and how much to grow a model to ensure genuine efficiency gains.