🤖 machine learning

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

This paper proposes a training-free method to halve the computational cost of fine-grained Mixture-of-Experts models at inference by decoupling expert selection from probability renormalization, thereby preserving performance by normalizing against a larger reference set of experts rather than the reduced active set.

Original authors: Xing Chen, Hengshuai Yao

Published 2026-09-07✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xing Chen, Hengshuai Yao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems that generate text often rely on a design that separates the total amount of knowledge they hold from the amount of thinking they do for each word. Imagine a massive library containing thousands of specialized books, but for every sentence written, the system only opens a handful of them to find the right information. This approach, known as a Mixture-of-Experts model, allows computers to store enormous amounts of data without slowing down every time they speak. The system uses a small internal guide to decide which specific books to open for each word, ensuring that only a tiny fraction of the total library is consulted at any moment. This creates a unique situation: the computer must keep all the books on its shelves to be ready for any topic, but it only reads a few pages at a time.

Because the computer must keep all these specialized sections in its memory, researchers have long looked for ways to make the system faster by forcing it to read even fewer pages. The standard method for doing this is simple: tell the system to open fewer books per sentence. However, this straightforward approach has historically caused the system to become confused and make more mistakes. The confusion arises not because the system lacks the right books, but because the way it weighs the information from those books changes unexpectedly when the number of books is reduced. A new study by Xing Chen and Hengshuai Yao reveals that this confusion is caused by a hidden mathematical adjustment the system makes automatically, and they found a way to turn that adjustment off without needing to retrain the system or add any new components.

The researchers focused on two very large language models, one with 35 billion parameters and another with nearly 400 billion. These models are designed with hundreds of tiny specialized sections, or experts, for every layer of processing. In their standard operation, the system selects eight of these experts for every word it processes. When the researchers tried to speed up the system by cutting this number in half, forcing it to use only four experts, the performance dropped significantly. On a standard test of general knowledge, the accuracy fell by nearly five points. This loss was so severe that many had assumed reducing the number of active experts was simply too costly to be useful. The researchers suspected that the problem was not the loss of the experts themselves, but rather how the system recalibrated the importance of the remaining ones.

In the standard setup, when the system picks its top experts, it adjusts their influence so that their combined importance always adds up to a fixed total. This is a safety mechanism to keep the system stable. However, in these fine-grained models, the top eight experts usually hold only a small fraction of the total possible importance. When the system is forced to pick only four experts, the standard adjustment multiplies their influence by a large factor to make up the difference. This sudden boost pushes the system into a state it was never trained to handle, causing it to overreact and make errors. The researchers realized that the system was not actually missing information; it was simply being told to shout louder than it was supposed to.

To fix this, the team introduced a simple change that requires no training and no extra computing power. They separated the decision of which experts to use from the decision of how to weigh them. They allowed the system to pick only four experts to do the work, but they told it to calculate the importance weights based on the top sixteen experts instead of just the top four. By doing this, the system kept the same volume of information it was used to hearing, even though it was only using half the number of active sections. This small adjustment, which involves changing a single integer setting, completely changed the outcome. On the 35-billion-parameter model, reducing the active experts from eight to four with this new setting caused the accuracy to drop by only 0.35 points, a difference so small it was statistically indistinguishable from the original, slower system.

The study also tested this method on the much larger 400-billion-parameter model, where the results were similarly dramatic. Cutting the active experts in half with the standard method caused a significant loss in performance, but using the new weighting method reduced the loss to just 0.55 points. The researchers found that the best setting for this new weighting depended on the specific task. For example, a setting that made the system sound best when reading text did not always make it perform best on logic puzzles or knowledge tests. This finding suggests that simply measuring how well a compressed model reads text is not enough to guarantee it will work well on other tasks. The researchers concluded that to safely reduce the computing power needed for these models, one must carefully tune this weighting setting using actual performance tests rather than relying on text fluency alone.

The paper also explored why simply removing the extra experts permanently, rather than just using fewer of them temporarily, is difficult. They found that these models are highly specialized, with different experts handling different types of content like code or general writing. Because the experts are so specialized and the system is so well-balanced, there is very little room to permanently delete them without losing the ability to handle diverse topics. The most effective path forward, therefore, is not to throw away parts of the model, but to simply tell it to use fewer parts at a time while keeping the overall volume of information it processes consistent. This approach offers a way to make these powerful systems faster and more efficient without sacrificing their intelligence, simply by correcting a hidden calibration error that occurs when the rules of engagement change.

Technical Summary: Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

Problem Statement

Modern fine-grained Mixture-of-Experts (MoE) models, such as the Qwen3 series, utilize hundreds of small expert sub-networks per layer, routing each token to a small subset ( $k$ ) of them. While this architecture decouples parameter count from per-token compute, it creates a memory-compute asymmetry: all experts must reside in memory, but only a fraction are active.

A common approach to reduce inference compute is to lower the number of activated experts ( $k$ ) at runtime. However, standard implementations implicitly renormalize the router probabilities of the selected experts so they sum to one. In fine-grained routers, the top- $k$ probabilities often sum to a small fraction of the total probability mass (e.g., 0.182 for top-8 out of 256). Consequently, renormalization applies a significant gain (amplification factor) to the selected experts' outputs.

The paper identifies that reducing $k$ at inference without adjusting this renormalization process conflates two effects:

Capacity Loss: The removal of experts.
Gain Miscalibration: The remaining experts are amplified by a factor larger than what the model was trained on, pushing the expert branch's contribution to the residual stream beyond its calibrated regime.

This miscalibration leads to severe performance degradation (e.g., a 4.65-point drop in MMLU accuracy when halving experts from 8 to 4 on a 35B model), making simple $k$ -reduction appear unaffordable.

Methodology

The authors propose a training-free method to decouple the selection of experts from the normalization (gain calibration) of their outputs.

The Decoupled Renormalization

Instead of normalizing by the mass of the top- $k_1$ selected experts, the method normalizes by the mass of a separate reference set of the top- $k_2$ experts:

$w_i = \frac{p_i}{\sum_{j \in T_{k_2}} p_j}, \quad i \in T_{k_1}$

Where:

$k_1$ : The number of experts actually activated (controls compute).
$k_2$ : The size of the reference set used for the denominator (controls gain).
$T_{k_1}$ : The set of indices for the top- $k_1$ experts.
$T_{k_2}$ : The set of indices for the top- $k_2$ experts.

Key Properties

Zero Training/Parameters: This requires only changing one integer ( $k_2$ ) at inference. No weights are modified, and no fine-tuning is performed.
Negligible Overhead: The router already computes probabilities for all $E$ experts. Selecting a larger $k_2$ for the denominator involves a slightly wider top- $k$ selection on already-computed logits, which is negligible compared to the cost of evaluating the expert FFNs.
Adaptive Scaling: When $k_2 > k_1$ , the weights sum to less than 1, resulting in an input-adaptive down-scaling of the expert branch, effectively anchoring the gain to the training distribution.

Key Contributions and Findings

1. The Reference Set is a Free Variable

The paper demonstrates that the renormalization denominator need not match the activation count. By setting $k_2$ to the model's trained $k$ (or a value in $[k, 2k]$ ) while reducing $k_1$ , the gain is preserved.

Result: On Qwen3.6-35B-A3B, reducing activated experts from 8 to 4 ( $k_1=4$ ) with standard renormalization ( $k_2=4$ ) drops MMLU by 4.65 points. With the proposed method ( $k_2=16$ ), the drop is only 0.35 points, which is statistically indistinguishable from the baseline.
Result: On the 11 $\times$ larger Qwen3.5-397B-A17B, reducing from 10 to 5 experts ( $k_1=5$ ) loses only 0.55 points with $k_2=10$ , compared to a 2.10-point loss with standard renormalization.

2. Expert Identity vs. Weighting

The study analyzes whether the model is more sensitive to which experts are chosen or how much they are weighted.

Finding: Expert identity is substantially more important than weighting. Replacing router weights with uniform weights (keeping the same top-8 selection) increases perplexity by 49.7%. Randomly selecting experts increases it by 1667%.
Implication: This validates the proposed method: as long as the correct experts are selected ( $k_1$ ), the model can tolerate significant distortion in the magnitude of their weights (via the $k_2$ correction).

3. Perplexity vs. Downstream Accuracy

The paper cautions against using perplexity alone to select compression hyperparameters.

Finding: On the 35B model, perplexity is minimized at $k_2=8$ (the trained $k$ ), but MMLU accuracy is maximized at $k_2=16$ . The perplexity-optimal setting results in a statistically significant 1.10-point drop in MMLU compared to the accuracy-optimal setting.
Implication: Selecting $k_2$ based on unlabeled text (perplexity) can lead to suboptimal configurations for downstream tasks. Validation should be performed on downstream tasks using paired significance testing.

4. Limits of Expert Pruning

The authors analyze the feasibility of removing experts entirely (pruning) versus just reducing activation.

Finding: The router is well-balanced with almost no "dead" experts. Furthermore, expert specialization is highly domain-dependent (e.g., WikiText vs. Code have low expert overlap).
Implication: Pruning experts for a general-purpose model offers little headroom, as a union of domain-specific top experts covers nearly the entire set. The compute-side lever (reducing $k_1$ with corrected $k_2$ ) is more effective than structural pruning for these fine-grained models.

Significance and Claims

The paper claims that the perceived trade-off between compute reduction and accuracy in fine-grained MoE models is largely an artifact of implicit gain miscalibration caused by standard renormalization.

Training-Free Efficiency: The method enables halving the activated expert compute with negligible loss in downstream accuracy, requiring no training, distillation, or parameter modification.
Re-evaluation of Prior Work: The authors suggest that prior reports of "precipitous" degradation when reducing $k$ in fine-grained models may be overly pessimistic because they did not decouple the reference set.
Methodological Shift: The paper advocates for scanning $k_2 \in \{k_1, k, 2k\}$ and validating via paired testing on downstream tasks rather than relying on perplexity or default renormalization settings.

The work is limited to specific Qwen series models and English/Python domains, and the authors note that the effect scales with router flatness, suggesting coarse-grained MoEs (e.g., 8 experts, top-2) may benefit less.