💬 NLP

Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

This paper demonstrates that Mixture-of-Experts (MoE) language models can outperform dense counterparts under strictly equal resource constraints (total parameters, compute, and data) by identifying an optimal activation rate that remains consistent across model sizes, a finding validated through extensive experiments training nearly 250 models and processing 50 trillion tokens.

Original authors: Houyi Li, Ka Man Lo, Shijie Xuyang, Ziqi Wang, Wenzhen Zheng, Haocheng Zhang, Zhao Li, Shuigeng Zhou, Xiangyu Zhang, Daxin Jiang

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Houyi Li, Ka Man Lo, Shijie Xuyang, Ziqi Wang, Wenzhen Zheng, Haocheng Zhang, Zhao Li, Shuigeng Zhou, Xiangyu Zhang, Daxin Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a giant library of knowledge. You have a strict budget: you can only buy a specific number of books (parameters), you can only spend a specific amount of time reading them (compute), and you only have access to a specific number of unique stories to read (data).

For a long time, the standard way to build this library was to hire one massive, super-smart librarian who read every single book in the library for every single question asked. This is what the paper calls a Dense Model. It's reliable, but it's slow and expensive because that one librarian has to do all the heavy lifting.

Recently, a new idea called Mixture-of-Experts (MoE) became popular. Instead of one giant librarian, you hire a team of 100 specialized experts. When a question comes in, a "manager" (the gate) quickly picks just a few of the best experts to answer it, while the rest take a coffee break. This is faster and allows you to have a much larger library (more total books) without slowing down the reading speed.

The Big Question
The paper asks a very specific, tricky question: If we give the "One Giant Librarian" and the "Team of Experts" the exact same budget for books, time, and unique stories, can the Team of Experts actually win?

Previous studies often cheated by giving the Team of Experts more books or more time. This paper wanted to see if the Team could win when the rules were strictly equal.

The Experiment: Finding the Sweet Spot
The researchers didn't just throw money at the problem; they acted like master architects. They built nearly 200 different versions of these libraries to find the perfect design.

The Architecture: They figured out the best way to arrange the experts. They found that having one "generalist" layer (like a standard dense layer) at the start, followed by the expert layers, worked best.
The Activation Rate (The "Coffee Break" Ratio): This is the most important discovery. In an MoE team, the "activation rate" is the percentage of experts who actually wake up to work on a problem.
- If you wake up too few experts (too low a rate), the team doesn't have enough brainpower.
- If you wake up too many experts (too high a rate), the team gets confused and loses its special focus.
- The Golden Rule: The researchers found a "sweet spot" where exactly 20% of the experts wake up. No matter if the library was small (2 billion books) or medium-sized (7 billion books), this 20% rule always produced the best results.

The Data Problem: Reusing Stories
There was a catch. Because the Team of Experts is so efficient, they needed to read more stories to learn as well as the Giant Librarian. If you gave them the same unique stories as the Giant Librarian, they would fall behind.

To solve this, the researchers tried a strategy called Data Reuse. Imagine the Giant Librarian reads a story once. The Team of Experts reads the same story, but they read it twice or three times (reusing the data).

The Result: Even when the Team of Experts was forced to read the same unique stories as the Giant Librarian (by re-reading them), they still managed to outperform the Giant Librarian, provided they stuck to that 20% activation rate.

The Conclusion
The paper concludes that yes, the Team of Experts can beat the Giant Librarian even when they have the exact same total resources (books, time, and unique stories).

However, this only works if you:

Design the team layout perfectly.
Keep the "wake-up" rate of the experts at that magic 20%.
Allow the team to read the same stories a few extra times to make up for the efficiency gap.

In short, the paper proves that with the right design and a little bit of "re-reading," a specialized team of experts is a more powerful way to build intelligent systems than a single, massive worker, even when the budget is identical.

Technical Summary: Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

Problem Statement
While Mixture-of-Experts (MoE) language models are widely recognized for expanding model capacity without increasing per-token compute, a critical question remains unresolved: Can MoE architectures outperform dense architectures when strictly constrained by identical total parameters ( $N$ ), total training compute ( $C$ ), and total data budget ( $D$ )? Previous comparisons often favor MoE by relaxing these constraints (e.g., allowing MoE to use more data or comparing models with vastly different parameter counts). This paper addresses the gap in understanding whether architectural advantages alone, under strict resource parity, can yield superior performance.

Methodology
The authors propose a novel perspective and a rigorous three-step experimental methodology to ensure a fair comparison under fixed $N$ , $C$ , and $D$ :

Optimized Architecture Search: Before comparing activation rates, the authors systematically optimize the MoE backbone. They investigate layer arrangements (full MoE, interleaved, or 1-dense + MoE), gate score normalization, parameter allocation (Top- $K$ settings), and structural hyperparameters (model aspect ratio $\zeta$ and FFN expansion ratios). They identify a "1dense + Shared Expert" configuration with specific structural ratios ( $\zeta \approx 88$ , $\mu \approx 22$ ) as the optimal backbone.
Activation Rate (AR) Analysis under Fixed $N$ and $C$ : Using the optimized backbone, the authors train MoE models with varying activation rates ( $r_a$ , the ratio of activated parameters to total parameters) while matching the total training compute of a dense baseline. To maintain equal $C$ , MoE models (which have lower per-token compute) are trained on proportionally more tokens initially.
Data Reuse Strategy: To strictly equate the unique data budget ( $D$ ), the authors introduce a data reuse strategy. They train models on a smaller, fixed unique dataset $\hat{D}$ for multiple epochs. They compare "strict" reuse (matching $N, C, D$ exactly) and "loose" reuse (fixing epochs to 2) against dense baselines trained on unique data.

Key Results
The study involved training nearly 200 models at the 2B scale and over 50 at the 7B scale, cumulatively processing 50 trillion tokens.

Existence of an Optimal Activation Rate: The experiments reveal a stable "optimal activation rate" region ( $R^*_a$ ) where MoE models consistently outperform their dense counterparts. This optimal rate is approximately 20% ( $r_a \approx 0.2$ ).
Performance Superiority: Under fixed $N$ and $C$ , MoE models with $r_a \approx 20\%$ achieve lower loss (better performance) than dense models. Specifically, a 2B MoE model with optimal AR outperforms a dense model trained with the same compute, approaching the performance of a dense model trained with double the compute.
Consistency Across Scales: The optimal activation rate of ~20% remains consistent across different model sizes (2B, 3B, and 7B), suggesting the finding is independent of scale within the tested range. This contrasts with some prior work suggesting optimal sparsity scales with model size.
Data Reuse Efficacy: The data reuse strategy successfully offsets the additional data requirements of MoE models. While reusing data causes a slight performance reduction compared to training on unique data, MoE models with optimal AR still surpass dense baselines even under strict data constraints.
Downstream Generalization: The superiority of MoE models with $r_a \approx 20\%$ generalizes to downstream tasks (SFT) across reasoning, knowledge, math, and code benchmarks. Notably, data reuse has minimal negative impact on reasoning tasks but degrades knowledge performance; however, at the optimal AR, SFT-ed MoEs with reused data still outperform dense models trained on unique data.

Significance and Claims
The paper claims to provide the first definitive evidence that MoE architectures can surpass dense LLMs under strictly equal resource constraints ( $N, C, D$ ), provided the architecture is optimized and the activation rate falls within the identified optimal region.

Architectural Advantage: The performance gains are attributed solely to architectural advantages rather than disparities in parameter count or compute budget.
Resource Efficiency: The findings challenge the conventional wisdom that dense models are superior when resources are fixed, paving the way for more resource-efficient architectures in next-generation NLP systems.
Practical Strategy: The introduction of a data reuse strategy demonstrates that the additional data demand of MoE models can be mitigated, making the superior performance achievable without requiring unique data budgets significantly larger than those of dense models.

The authors acknowledge limitations, including the constraint of not training models larger than 7B due to computational costs and the focus on specific MoE components while fixing others. They suggest future work could explore how optimal activation rates enhance capabilities in other training paradigms like upcycling and MoEfication.

More like this