Technical Summary: SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training

Problem Statement

Mixture-of-Experts (MoE) architectures have become dominant for scaling Large Language Models (LLMs), yet they remain computationally expensive to pretrain and serve. While structured pruning and knowledge distillation (KD) are established techniques for compressing dense models, their application to MoE models at the pretraining scale remains unclear. Specifically, there is a lack of systematic understanding regarding:

1. Whether pruning a pretrained MoE provides a superior initialization compared to training a target architecture from scratch.
2. How different expert compression strategies (pruning vs. merging) affect final performance after large-scale continual pretraining.
3. What constitutes the optimal training recipe (loss functions and scheduling) for recovering performance post-compression.

Methodology

The paper proposes SlimQwen, a systematic framework for compressing MoE models during pretraining. The methodology encompasses three primary dimensions:

1. Structured Pruning and Initialization

The authors investigate compressing the Qwen3-Next-80A3B model (a hybrid-attention MoE) into a smaller 23A2B configuration. The compression operates across three dimensions:

• Depth Pruning: Removing the last 25% of transformer layers.
• Width Pruning: Reducing the hidden dimension (from 2048 to 1536) based on activation statistics (Mean Absolute Activation) computed on a calibration dataset.
• Expert Compression: Reducing the number of experts per MoE module (from 512 to 256) and the number of activated experts (from 10 to 8 routed + 1 shared).

2. Expert Merging Strategy

To address the trade-off between preserving specialized knowledge and consolidating discarded experts, the authors propose a Partial-Preservation Expert Merging strategy. Instead of purely pruning or aggressively merging all experts:

• The top 50% of experts (ranked by importance scores derived from routing frequency, soft logits, or REAP) are retained intact.
• The remaining 50% of target experts are constructed by merging discarded experts into selected "base" experts from the retained set, weighted by their relative importance scores.
• This approach aims to prevent representation homogenization while maintaining the capacity to absorb knowledge from pruned experts.

3. Training Recipes and Distillation

The paper evaluates several training objectives and scheduling strategies for the compressed model:

• Loss Functions: The authors compare pure Knowledge Distillation (KD) against a hybrid approach combining KD with standard Language Modeling (LM) loss. They further introduce Multi-Token Prediction (MTP) Distillation, which extends the distillation objective to predict multiple future tokens simultaneously, not just the next token.
• Progressive Pruning: Instead of "one-shot" compression, the authors explore progressive schedules (Depth-first, Width-first, and Joint) where structural reduction occurs in stages (e.g., 40B tokens at an intermediate stage, followed by 360B tokens at the final target configuration).

Key Findings and Results

Initialization Superiority

Training a target architecture from scratch (Random Init) significantly underperforms compared to initializing from a pruned MoE model. Under the same 120B token training budget, the pruned initialization achieved an average benchmark score of 73.45, compared to 61.66 for random initialization. The pruned model recovered 86.5% of the teacher's performance despite being 3.4× smaller.

Expert Compression Equivalence and Partial Preservation

• One-shot Methods: After large-scale continual pretraining (400B tokens), different one-shot expert pruning or merging methods (based on frequency, soft logits, or REAP) converged to similar final performance levels. No single metric dominated across all tasks.
• Partial Preservation: The proposed partial-preservation strategy (retaining half the experts intact) consistently outperformed other merging strategies across major benchmarks (MMLU, MMLU-Pro, GSM8K), suggesting that preserving a subset of experts intact helps maintain specialized knowledge.

Training Objectives

• Hybrid Loss: Combining Next-Token Prediction (NTP) KD with standard LM loss outperformed pure KD, particularly on knowledge-intensive tasks (e.g., MMLU improved from 74.16 to 74.93).
• MTP Distillation: Incorporating MTP KD yielded consistent gains across benchmarks. Furthermore, MTP KD significantly improved the multi-token acceptance rate in speculative decoding (e.g., increasing acceptance from acc 1 to acc 4), enhancing inference efficiency.

Progressive vs. One-shot Compression

Progressive pruning schedules consistently outperformed one-shot compression under the same total token budget (400B). The Depth-first progressive strategy (SlimQwen) achieved the best results, improving MMLU scores from 75.86 (one-shot) to 77.39. This suggests that gradual architectural transitions facilitate smoother optimization trajectories for knowledge transfer.

Significance and Contributions

The paper claims to offer practical guidance for efficient MoE compression at the pretraining scale. Its main contributions are:

1. Systematic Study of MoE Compression: It establishes that structured pruning provides a strong initialization for continual pretraining and demonstrates that different one-shot expert compression methods yield similar results after extensive training, provided a proper recovery strategy is used.
2. Partial-Preservation Strategy: It introduces a simple yet effective expert merging technique that balances knowledge preservation and consolidation, leading to consistent downstream performance improvements.
3. Advanced Distillation Techniques: It validates the efficacy of combining LM loss with KD and proposes MTP Distillation, which improves both backbone training quality and speculative decoding efficiency.
4. Progressive Scheduling: It demonstrates that progressive pruning schedules (specifically Depth-first) are superior to one-shot compression, confirming that staged capacity reduction optimizes the transfer of pretrained knowledge.

Empirically, the work successfully compresses the Qwen3-Next-80A3B model into a 23A2B model (approx. 4× compression) that retains competitive performance across general reasoning, mathematics, coding, and Chinese language benchmarks.