💬 NLP

Post-Trained MoE Can Skip Half Experts via Self-Distillation

This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that converts post-trained static Mixture-of-Experts models into efficient dynamic ones by injecting parameter-free zero-output experts and applying two-stage self-distillation, thereby reducing expert FLOPs by over 50% with marginal accuracy loss and delivering significant inference speedups.

Original authors: Xingtai Lv, Li Sheng, Kaiyan Zhang, Yichen You, Siyan Gao, Xueheng Luo, Yuxin Zuo, Yuchen Fan, Junlin Yang, Ganqu Cui, Bingning Wang, Fan Yang, Youbang Sun, Ning Ding, Bowen Zhou

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xingtai Lv, Li Sheng, Kaiyan Zhang, Yichen You, Siyan Gao, Xueheng Luo, Yuxin Zuo, Yuchen Fan, Junlin Yang, Ganqu Cui, Bingning Wang, Fan Yang, Youbang Sun, Ning Ding, Bowen Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The Over-Worked Kitchen

Imagine a high-end restaurant (a Large Language Model) that has a massive kitchen with 128 expert chefs (these are the "Experts" in the MoE architecture).

In a standard setup, no matter what order comes in—whether it's a simple "Hello" or a complex request to write a symphony—the kitchen always sends the order to the same 8 chefs. This ensures the food is always perfect, but it's incredibly expensive and slow because you are paying 8 chefs to chop a single carrot when one would do.

The goal of this paper is to make the kitchen smarter: Let the order decide how many chefs are needed. If it's a simple order, use fewer chefs. If it's hard, use more. This is called a "Dynamic" kitchen.

The Challenge: Don't Fire Anyone Yet!

Usually, to build a smart, dynamic kitchen, you have to hire a whole new team and train them from scratch. That takes years and costs a fortune.

But this paper asks: Can we take an existing, fully trained kitchen (one that already knows how to cook perfectly) and turn it into a dynamic one without firing anyone or retraining them from zero?

The answer is ZEDA (Zero-Expert Self-Distillation Adaptation).

The ZEDA Solution: The "Ghost Chefs"

ZEDA uses a clever trick involving Ghost Chefs (Zero Experts).

Injecting Ghosts: The researchers take the existing kitchen and secretly add a bunch of new "chefs" who have no hands and carry no knives. They are Zero Experts. They do absolutely nothing. Their output is always zero.
The New Rule: The kitchen manager (the Router) is now told: "You still have to pick 8 people for every order, but now you can pick from the 128 real chefs plus these new Ghost Chefs."
The Magic: If the order is simple (like "What's the weather?"), the manager learns to pick 4 real chefs and 4 Ghost Chefs. Since the Ghosts do no work, the kitchen only does the work of 4 chefs. If the order is hard (like "Solve this math problem"), the manager picks 8 real chefs and 0 Ghosts.

How Do We Teach the Manager? (Self-Distillation)

The problem is that the manager doesn't know when to pick a Ghost Chef yet. If they pick a Ghost for a hard math problem, the food will be burnt.

To fix this, ZEDA uses Self-Distillation, which is like a "Shadow Training" program:

The Teacher: The original, fully trained kitchen (with 128 real chefs) acts as the strict Teacher. It knows exactly what the perfect answer looks like.
The Student: The new kitchen (with Ghosts) tries to copy the Teacher.
The Process:
1. Stage 1 (SFT): The Student watches the Teacher cook. The Teacher says, "For this simple order, I would have used 8 chefs, but you can try using 4 real chefs and 4 Ghosts. Just make sure the taste is the same."
2. Stage 2 (OPD): The Student starts cooking on its own. If it makes a mistake, the Teacher steps in and says, "You picked a Ghost for a hard math problem! That's why the answer is wrong. Next time, pick more real chefs for this type of question."

The Secret Sauce: The "Group Balance"

There was a risk that the manager might get lazy and pick only Ghosts to save energy, or pick no Ghosts and waste energy.

To stop this, the researchers added a Group Balance Rule. They told the manager: "You must keep a healthy ratio. For every 2 real chefs you use, you should try to use 1 Ghost Chef, but never mess up the balance between the real chefs themselves."

This ensures the kitchen stays efficient (using Ghosts when possible) but doesn't ruin the quality of the food.

The Results: Faster, Cheaper, Same Taste

After this training (which took less than 2 days on powerful computers, compared to years for original training):

Speed: The kitchen became 20% faster because it stopped paying for unnecessary work.
Efficiency: It skipped over 50% of the expert work (using Ghosts for about half the tokens).
Quality: The food tasted almost exactly the same as before. On some tests, it even got slightly better because the manager learned to focus its energy better.

Summary

ZEDA is a way to take a "static" AI model (one that always does the same amount of work) and turn it into a "dynamic" one (one that does less work for easy tasks) by:

Adding invisible "Ghost" experts that do nothing.
Teaching the model to use these Ghosts for easy tasks by copying a "Teacher" version of itself.
Using a special rule to ensure the Ghosts are used just enough to save money, but not so much that the answers get bad.

It's like turning a bus that always carries 50 passengers (even if only 5 show up) into a bus that automatically shrinks to a van when only 5 people are on board, saving fuel without leaving anyone behind.

Technical Summary: Zero-Expert Self-Distillation Adaptation (ZEDA)

1. Problem Statement

Mixture-of-Experts (MoE) architectures have become standard for scaling Large Language Models (LLMs) by increasing capacity while bounding per-token computation. While dynamic MoE variants offer further efficiency by adjusting the number of activated experts based on input difficulty, existing methods typically require pre-training from scratch or task-specific adaptation. Consequently, the practical conversion of fully post-trained static MoE models (which have undergone extensive pre-training, Supervised Fine-Tuning (SFT), and Reinforcement Learning (RL)) into efficient dynamic counterparts remains underexplored. Directly applying existing dynamic routing methods to these mature models risks disrupting the carefully calibrated routing distributions and capabilities established during their full training pipelines. The core challenge is to migrate a post-trained static MoE into a dynamic one that skips unnecessary experts for "easy" tokens without sacrificing established performance or incurring prohibitive adaptation costs.

2. Methodology: ZEDA

The paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into dynamic ones. The method consists of three primary components:

2.1 Zero-Expert Injection

ZEDA architecturally converts a static MoE into a dynamic one by injecting parameter-free zero experts ( $Z = \{Z_1, ..., Z_{N_Z}\}$ ) into the existing expert pool.

Mechanism: These zero experts output identically zero ( $Z_j(h) = 0$ ) for any input hidden state $h$ .
Routing: The router selects a top- $K$ subset from the expanded pool of $N + N_Z$ experts. If a zero expert is selected, it contributes no computation, effectively reducing the number of active normal experts for that token.
Initialization: Original router parameters for normal experts are preserved. New router parameters for zero experts are initialized from a Gaussian distribution matching the mean and variance of the original parameters to maintain the scale of router logits.

2.2 Two-Stage Self-Distillation

To adapt the augmented model without retraining from scratch, ZEDA employs a two-stage self-distillation process using the original post-trained MoE as a frozen teacher:

Supervised Fine-Tuning (SFT): The student model is trained on responses sampled from the teacher. The loss combines standard cross-entropy with the Group Auxiliary Loss.
On-Policy Distillation (OPD): The student generates its own trajectories, and the teacher evaluates these to provide token-level targets. The objective minimizes the reverse KL divergence between the student's on-policy distribution and the teacher's distribution.

2.3 Group Auxiliary Loss ( $L_{GA}$ )

Standard load-balancing losses (which enforce uniformity across all experts) are unsuitable for post-trained MoEs as they would disrupt learned, non-uniform routing patterns among normal experts. ZEDA introduces a Group Auxiliary Loss that balances activation at the group level rather than the individual expert level:

Groups: Normal experts form group $\mathcal{E}$ ; zero experts form group $\mathcal{Z}$ .
Objective: The loss regulates the relative activation frequency between the two groups, controlled by a weight parameter $w$ . It encourages the ratio of activated normal experts to zero experts to converge toward a target ratio ( $N : N_Z \cdot w$ ) without flattening the routing distribution within the normal expert group.
Result: This allows the model to learn when to skip computation (select zero experts) while preserving the internal routing logic of the original model.

3. Key Contributions

Post-Training Adaptation Framework: ZEDA is the first method to effectively convert fully post-trained static MoE models into dynamic ones via zero-expert injection and self-distillation, avoiding the need for expensive re-pretraining.
Stable Architectural Conversion: The introduction of the Group Auxiliary Loss stabilizes the conversion by regulating zero-expert utilization without disrupting the learned routing structures of normal experts.
Efficiency-Performance Trade-off: The method achieves a significant reduction in computation (skipping >50% of experts) with only marginal accuracy loss, outperforming existing dynamic MoE baselines and variants (e.g., Naive Expert Truncation).
Insight into Activation Dynamics: The paper provides analysis showing that zero-expert activation correlates with teacher-student distribution gaps (logp-diff) and model uncertainty, rather than overall task difficulty. Structured segments (like code or math expressions) tend to have higher zero-expert activation ratios (less computation) compared to natural text.

4. Experimental Results

Experiments were conducted on Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks covering math, code, and instruction following.

Performance: ZEDA eliminates over 50% of expert FLOPs (achieving $r_{ZE} \approx 51.2\%$ $r_{Z E} \approx 51.2%$ on Qwen and $53.0\%$ $53.0%$ on GLM) while maintaining performance comparable to the original static model.
- On Qwen3-30B-A3B, ZEDA outperforms the strongest dynamic baseline (AdaMoE) by 6.1 points on average.
- On GLM-4.7-Flash, it outperforms the baseline by 4.0 points.
- Unlike baselines that collapse on hard reasoning or code tasks, ZEDA maintains competitive performance uniformly across all domains.
Efficiency: The adaptation yields an end-to-end inference speedup of approximately 1.20× (roughly 20% speedup in both prefill and decode phases) on 8 NVIDIA H200 GPUs.
Adaptation Cost: The entire adaptation process takes less than 31 hours for Qwen and 62 hours for GLM, which is negligible compared to the costs of pre-training or full post-training.
Generalization: ZEDA demonstrates strong Out-of-Distribution (OOD) generalization on MMLU-Redux and GPQA-Diamond, preserving accuracy while maintaining high zero-expert utilization.

5. Significance and Claims

The paper claims that ZEDA offers a practical, cost-effective solution for enhancing the deployment efficiency of large-scale MoE systems. By leveraging the existing capabilities of post-trained models and using self-distillation to learn dynamic routing, ZEDA enables "easy" tokens to bypass unnecessary expert computation without compromising the model's established knowledge or reasoning abilities.

The authors emphasize that this approach avoids the substantial architectural modifications and retraining costs associated with previous dynamic MoE methods. The results suggest that post-trained MoE models can be successfully migrated to efficient dynamic architectures, making high-capacity models more viable for real-world serving where inference costs are a primary constraint. The work highlights that dynamic computation allocation is intrinsically linked to the model's internal uncertainty and the alignment between student and teacher distributions, rather than just external task difficulty.