🤖 machine learning

Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts

This paper introduces an "orthogonal growth" strategy that recycles existing converged Mixture-of-Experts checkpoints by expanding their depth and width, demonstrating that this approach significantly outperforms training from scratch under identical compute budgets by leveraging prior "sunk costs" to achieve higher accuracy.

Original authors: Ruizhe Wang, Yucheng Ding, Xiao Liu, Yaoxiang Wang, Peng Cheng, Baining Guo, Zhengjun Zha, Yeyun Gong

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruizhe Wang, Yucheng Ding, Xiao Liu, Yaoxiang Wang, Peng Cheng, Baining Guo, Zhengjun Zha, Yeyun Gong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Sunk Cost" Trap

Imagine you have spent years and millions of dollars building a massive, highly skilled library of books (a pre-trained AI model). You've trained it on a huge amount of data, and it's very good at what it does. But then, you realize you need a bigger library to handle even more complex questions.

The old way of doing this is to tear down your existing library and start building a new, bigger one from scratch. This is incredibly expensive and wasteful because you throw away all the work you already did. In business terms, this is a "sunk cost"—money and effort already spent that you can't get back.

This paper asks: Can we recycle that old library instead of throwing it away? Can we take the existing, well-trained model and "grow" it into a bigger, better one without starting over?

The Solution: "Orthogonal Growth"

The authors propose a method called Orthogonal Growth. Think of "orthogonal" as meaning "at right angles" or "independent." They found two distinct ways to make the model bigger that don't interfere with each other. They can be done in any order, like putting on socks before shoes or shoes before socks—the result is the same.

1. Depth Growth: Adding More Floors (The "Interposition" Trick)

Imagine your model is a skyscraper with 20 floors. To make it taller, you could just stack another 20-floor building on top of the first one.

The Old Way (Stacking): If you just stack a new building on top, the elevator (the data flow) has to jump from the top floor of the old building directly to the bottom floor of the new one. This causes a jarring disconnect. The "vibe" of the building changes abruptly.
The New Way (Interposition): Instead of stacking, the authors suggest inserting copies of the existing floors between the original ones.
- Analogy: Imagine a musical scale. If you have notes C, D, E, F, G... and you want to make the song longer, you don't just repeat the whole song at the end. Instead, you insert a second "C" right after the first "C," a second "D" after the first "D," and so on. This keeps the melody smooth and continuous.
- Why it works: The paper found that well-trained models have a specific "rhythm" in how their layers work. Inserting copies preserves this rhythm, whereas stacking breaks it. This method works best when the model is already fully trained (converged).

2. Width Growth: Adding More Experts (The "Specialist" Trick)

Now imagine the model is a hospital. Inside each room (layer), there are a few doctors (experts) who specialize in different things. The model has a "router" that decides which doctor to call for a specific patient.

The Old Way: If you just copy the doctors exactly, you have two identical doctors in the room. They will do the exact same thing, which is redundant and confusing.
The New Way: The authors suggest copying the doctors but adding a tiny bit of "static" or "noise" to the new ones.
- Analogy: Imagine you have a master chef. You hire a clone of that chef, but you give the clone a slightly different spice rack or a tiny scratch on their apron. This small difference forces the clone to develop their own unique style and specialize in slightly different dishes, rather than just copying the master perfectly.
- Why it works: This tiny bit of noise helps the new experts learn to do different jobs (specialization) without destroying the knowledge the original experts already have.

The Results: Getting More Bang for Your Buck

The researchers tested this on models ranging from 3 billion to 70 billion parameters. Here is what they found:

Recycling Works: You can take a smaller, fully trained model and grow it into a massive one.
Better than Starting Over: When they compared growing a model vs. training a new, big model from scratch using the same amount of extra computing power, the "grown" model was 10.6% more accurate.
More Investment = Better Results: The more "sunk cost" (training time and data) you put into the original small model before growing it, the better the final big model performs. It's like investing in a strong foundation; the taller you build on it, the better the building stands.
Order Doesn't Matter: You can add floors first and then add experts, or add experts first and then floors. The final result is the same.

The Bottom Line

This paper provides a blueprint for "sustainable" AI development. Instead of constantly burning money to build new models from scratch, we can take the models we already have, carefully expand them using these two tricks (inserting layers and adding noisy experts), and get a significantly smarter, larger model for less money. It turns the "waste" of sunk costs into a valuable asset.

Technical Summary: Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts

Problem Statement

The computational demands for pre-training Large Language Models (LLMs) are surging, yet existing pre-trained checkpoints often remain under-leveraged due to architectural limitations. Current model growth strategies typically focus on expanding models during early training stages, failing to capitalize on the substantial "sunk costs" (prior computational investment) of well-converged checkpoints. Furthermore, the application of model growth to Mixture-of-Experts (MoE) architectures, which are increasingly dominant in large-scale LLMs, has not been systematically studied. The core challenge is determining how to effectively "recycle" converged MoE models to create larger, more capable models without discarding the learned knowledge and optimized parameters of the base model.

Methodology

The authors propose an "orthogonal growth" framework designed to expand converged MoE models along two independent dimensions: Depth (increasing layers) and Width (increasing the number of experts).

1. Depth Growth: Interpositional vs. Stacking

The paper contrasts two methods for expanding model depth:

Stacking: Concatenating the original sequence of layers $k$ times ( $l_1, \dots, l_n, l_1, \dots, l_n$ ).
Interpositional Growth: Duplicating each layer in place ( $l_1, l_1, \dots, l_2, l_2, \dots$ ).

Key Insight: Analysis of pre-trained LLMs reveals a characteristic layer-wise weight norm distribution: norms are small in initial layers, increase monotonically through the middle, and slightly decrease near the end. This pattern is a structural signature of healthy convergence.

Stacking disrupts this monotonic trend by placing early-layer norms after late-layer norms, creating sharp discontinuities that require the model to "repair" during continued training.
Interpositional Growth preserves the learned weight-norm trajectory by duplicating layers in place, maintaining the smooth structural progression.
Threshold: The superiority of interpositional growth becomes critical once the base model has been trained beyond the compute-optimal FLOPs defined by Chinchilla scaling laws (approximately $1\times F_c$ ).

2. Width Growth: Noisy Expert Duplication

To expand the model width, the authors double both the total number of experts ( $E \to 2E$ ) and the number of activated experts ( $k \to 2k$ ).

Initialization Strategy: New experts are initialized by copying the weights of existing experts and adding a small amount of Gaussian noise ( $\epsilon \sim \mathcal{N}(0, (\alpha\sigma_{orig})^2)$ with $\alpha \approx 0.01$ ).
Rationale: Exact duplication creates perfect symmetry, preventing functional divergence. A small perturbation breaks this symmetry just enough to allow the router to differentiate experts during continued training, while preserving the learned representations. This contrasts with "upcycling" from dense models, which often requires high noise levels ( $\ge 50\%$ ) and random router initialization.

3. Orthogonality

The paper demonstrates that depth and width growth are orthogonal.

Structural Disjointness: Depth growth adds new layers; width growth adds new experts within existing layers. They operate on non-overlapping parameter sets.
Optimization Dynamics: Gradient analysis shows that the optimization directions for depth-grown and width-grown models are nearly orthogonal to the pre-growth direction and to each other.
Order Independence: The final performance is invariant to the sequence of expansion (depth-then-width vs. width-then-depth), confirming the strategies can be flexibly integrated.

Key Results

The framework was validated through extensive experiments on models ranging from 3B to 70B parameters using a 1-trillion-token dataset.

Depth Growth Efficacy: On a 3B MoE model grown to 6B, the interpositional method consistently outperformed stacking in both training loss and downstream accuracy. The performance gap widened as the base model became more converged.
Sunk Cost Correlation: There is a strong positive correlation between the amount of "sunk FLOPs" (training steps prior to growth) and the final accuracy of the grown model. Models grown from later, more converged checkpoints achieved higher final performance than those grown from earlier checkpoints, even with identical additional compute budgets.
Comparison to Scratch Training:
- Under a fixed additional compute budget, model growth significantly outperformed training a large model from scratch (e.g., a 10.6% accuracy improvement in the 70B experiment).
- Under a fixed total compute budget (sunk + additional), model growth yielded results comparable to or slightly superior to training from scratch, provided the growth was initiated from a checkpoint with sufficient sunk cost.
Scalability: The method successfully scaled a 17B MoE model (trained on ~600B tokens) to 70B parameters (trained on a total of 1T tokens). The final 70B model achieved a 64.17 average accuracy, surpassing the 17B baseline by 5.62 points and the from-scratch 70B baseline by 10.6% relative gain under the same extra FLOPs budget.

Significance and Claims

The paper claims to provide a cost-effective blueprint for sustainable large-scale LLM development. Its primary significance lies in:

Recycling Sunk Costs: Demonstrating that pre-trained checkpoints, often viewed as disposable intermediate assets, can be systematically leveraged to create larger models, thereby preserving the value of prior computational investments.
Optimal Growth Strategies: Establishing that for well-converged models, interpositional layer copying and minimal-noise expert duplication are superior to traditional stacking and random initialization.
Orthogonal Scaling: Proving that depth and width can be expanded independently and in any order without interference, offering flexibility in resource-constrained scaling scenarios.
Efficiency: Showing that growth can achieve performance gains that exceed training from scratch under identical extra compute budgets, effectively lowering the barrier to frontier-scale model pre-training.

The authors emphasize that this work does not introduce new societal risks beyond those inherent in LLMs but aims to reduce the environmental footprint of model development by improving training efficiency.