💬 NLP

Pruning and Distilling Mixture-of-Experts into Dense Language Models

This paper introduces a novel framework that converts trained Mixture-of-Experts (MoE) models into standard dense architectures by scoring, selecting, and grouping experts followed by knowledge distillation, demonstrating that this approach significantly outperforms traditional dense-to-dense pruning in accuracy and training efficiency.

Original authors: Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "All-Hands-On-Deck" Bottleneck

Imagine a massive, high-end restaurant kitchen (the MoE model) designed to cook incredibly complex meals. This kitchen has 128 different specialist chefs (experts) on staff. However, for any single order, the head chef (the router) only calls on 8 of them to work. The other 120 chefs stand idle, waiting for their turn.

This setup is great for cooking speed and variety, but it has a huge problem: To open the kitchen, you must hire and pay all 128 chefs. Even if you only use 8 at a time, you need enough space (memory) and money (compute) to keep all 128 on the payroll. This makes it impossible to open a small branch of this restaurant in a tiny shop (like a phone or a single computer) because there isn't enough room for all the staff.

Current solutions try to fire some chefs to make the kitchen smaller, but you still have to keep the remaining chefs on separate, specialized stations. You still need to load everyone into the building.

The Solution: The "Master Chef" Transformation

The authors of this paper propose a radical new recipe. Instead of just firing chefs, they want to merge the best 8 chefs into one super-chef who can do everything the team did, but without needing the extra space.

They take the trained team of 128 specialists and turn them into a standard, dense kitchen (a Dense model) that fits in a small shop but still cooks like a master.

How They Did It: The Three-Step Process

The paper outlines a three-step process to transform the "MoE Kitchen" into a "Dense Kitchen."

1. Scoring: Finding the "Star" Chefs

First, they need to decide which chefs to keep. They can't just pick the ones who show up the most often (frequency), because those might be the "generalists" who do everything okay but nothing great.

The Old Way: Pick the chefs who are called the most. (Like picking the most popular employees).
The New Way (Diversity-Aware Scoring): The authors invented a new metric called DO-ACP. Imagine you are picking a sports team. You don't just pick the 8 players who play the most games; you pick the 8 players who have the most different skills and are the best at those specific skills.
- If Chef A is great at baking and Chef B is great at grilling, you want both.
- If Chef C and Chef D are both great at grilling, you only need one of them.
- The new method mathematically ensures the selected chefs cover the widest range of "flavors" (knowledge) without redundancy.

2. Grouping & Merging: Building the Super-Chef

Once they pick the top 8 chefs (or a few more to be safe), they need to combine them.

Pure Pruning (The Winner): In most cases, the best result came from simply copying the top 8 chefs directly into the new kitchen without mixing their recipes. It turns out, having 8 distinct, high-quality specialists working side-by-side is better than trying to blend their recipes into one average recipe.
Merging: If they had to pick more than 8, they would blend the recipes of similar chefs together, weighted by how good they were.

3. Knowledge Distillation: The "Taste Test" Training

Now they have a new kitchen with 8 chefs (the Student), but it's not perfect yet. It's like a new restaurant that has the right staff but hasn't learned the secret family recipes.

They put the new kitchen to work alongside the original 128-chef kitchen (the Teacher).

The Teacher cooks a dish.
The Student tries to cook the exact same dish.
The Teacher corrects the Student: "No, you added too much salt," or "You missed the subtle spice."
The Student learns from these corrections. This process is called Knowledge Distillation.

The Results: Why This Matters

The authors tested this on several massive AI models (like Qwen3, DeepSeek, and GPT-OSS). Here is what they found:

Picking the Right Chefs is Everything: The method used to score the chefs mattered more than how they were grouped. Their new "Diversity-Aware" scoring was the clear winner, beating all previous methods by a significant margin.
Don't Blend, Just Select: Surprisingly, the best strategy was to pick exactly 8 chefs and not blend them at all. Just having the 8 best, most diverse experts working together was better than averaging them out.
Beating the Competition: They compared their "MoE-to-Dense" method against the old way of making small models (taking a big dense model and cutting it down).
- The Result: Their new method produced a student model that was 6.3% more accurate on average tasks.
- Speed: It was also 1.6 times faster to train because the original "Teacher" kitchen was more efficient to run during the training process.

The Bottom Line

Think of this paper as a blueprint for downsizing a massive corporate headquarters into a small, efficient startup office without losing any of the company's brainpower.

Instead of just firing people to save rent, they carefully selected the most diverse and talented 8 employees, gave them a new, compact office, and had them learn the company's secrets from the original big team. The result? A small office that performs just as well as the big one, but fits in a much smaller space.

Key Takeaway: You don't need to keep the whole massive team to get the results; you just need to pick the right 8, keep them distinct, and teach them well.

Technical Summary: Pruning and Distilling Mixture-of-Experts into Dense Language Models

Problem Statement

Mixture-of-Experts (MoE) architectures have become the dominant paradigm for frontier large language models (LLMs) due to their ability to scale total parameter counts while maintaining training and serving efficiency. However, a fundamental limitation persists: despite activating only a fraction of parameters per token, MoE models require all expert parameters to be loaded into memory. This memory footprint makes them unsuitable for memory-constrained deployment scenarios, such as on-device inference or single-GPU serving.

Current solutions involve releasing separate small dense models trained from scratch or via cascaded pruning of large dense teachers. These approaches are computationally expensive and fail to leverage the rich capabilities already present in the trained MoE model. Existing MoE compression methods (e.g., MC-SMoE, HC-SMoE) reduce the number of experts but retain the sparse MoE structure, preserving the memory inefficiency. There is a lack of systematic frameworks for converting a trained MoE model into a standard, fully dense architecture.

Methodology

The authors present the first systematic framework for converting a trained MoE language model into a fully dense architecture through a three-stage pipeline: Expert Scoring & Selection, Grouping & Concatenation, and Knowledge Distillation.

1. Expert Scoring and Selection

The core of the method involves scoring all $E$ experts in a layer to select a subset of size $K$ ( $K \ge k$ , where $k$ is the number of active experts per token). The paper evaluates seven scoring methods:

Frequency-based: Selection Frequency (SF), Pre-selection Probability (PP), Post-selection Probability (PS). These favor generalist experts.
Conditional Probability (CP): Focuses on routing confidence, favoring specialists.
Activation-weighted CP (ACP): Combines routing confidence with output magnitude.
D-Optimal Selection (DO): A novel criterion that jointly maximizes expert importance and mutual diversity. It selects a subset $S$ to maximize the log-determinant of an importance-weighted Gram matrix:
$S^* = \arg \max_{|S|=K} \log \det(\mathcal{K}_S + \lambda_{reg} I)$
where $\mathcal{K}_{ij} = \sqrt{I_i I_j} \cdot \mathbb{E}_t[\langle f_i(t), f_j(t) \rangle]$ . This is solved greedily. The authors propose DO-ACP (D-Optimal selection based on ACP scores) as the optimal strategy.

2. Grouping and Concatenation

Selected experts are assigned to $k$ groups.

Pure Pruning ( $K=k$ ): Each group contains exactly one expert; no weight averaging occurs.
Merging ( $K>k$ ): Experts in a group are merged via score-weighted averaging of their parameters.
The resulting $k$ groups are concatenated into a single dense Feed-Forward Network (FFN) with intermediate dimension $d_{dense} = k \times d_{expert}$ . Down-projection matrices are scaled by factors $\alpha_g$ (uniform or proportional) to approximate the average routing behavior of the original MoE.

3. Knowledge Distillation

The initialized dense model is refined via knowledge distillation from the MoE teacher. The objective minimizes the forward KL divergence between the teacher and student output distributions. The authors also explore expanding the teacher's routing during distillation (activating $k' > k$ experts) to expose the student to knowledge from non-selected experts.

Key Contributions

First MoE-to-Dense Framework: The paper introduces the first end-to-end system for converting a trained MoE into a fully dense architecture, moving beyond the limitation of producing smaller sparse MoE models.
Diversity-Aware Expert Selection: The authors introduce a D-optimal selection criterion (DO-ACP) that maximizes the log-determinant of an importance-weighted Gram matrix. This method explicitly penalizes redundancy, selecting experts that are both important and diverse.
Comprehensive Evaluation: The study evaluates 350 configurations (7 scoring $\times$ 5 grouping $\times$ 2 scaling $\times$ 5 values of $K$ ) on Qwen3-30B-A3B and validates findings on DeepSeek-V2-Lite and GPT-OSS-20B.

Results

Performance on Qwen3-30B-A3B

Scoring Dominance: The choice of scoring method is the most impactful factor, creating a 5.7 percentage point (pp) gap in average downstream accuracy between the best and worst methods. Grouping strategies contribute only $\sim$ 1 pp.
Superiority of DO-ACP: The DO-ACP method consistently outperforms all other configurations. At $K=8$ (pure pruning), it achieves 43.41% average accuracy on downstream benchmarks (Winogrande, HellaSwag, ARC-E/C, MMLU).
Pure Pruning vs. Merging: For diversity-aware scoring (DO-ACP, ACP), pure pruning ( $K=k$ ) outperforms merging ( $K>k$ ). Frequency-based methods (SF) benefit from merging, but DO-ACP experts are individually strong enough to stand alone.
Comparison to Baselines:
- vs. MoE Compression: Outperforms repurposed MoE compression metrics (e.g., MC-SMoE, HC-SMoE) by 5.9–7.1 pp.
- vs. Dense-to-Dense (D2D) Pruning: In a controlled comparison with a matched-parameter dense teacher (Qwen3-32B $\to$ 3.4B student), MoE-to-dense (DO-ACP) outperforms D2D pruning by +6.3 pp (58.10% vs. 51.84%) after $\sim$ 4B-token distillation.
- vs. Random Initialization: Outperforms random FFN initialization by +12.7 pp, demonstrating that the expert structure provides a powerful initialization.

Extended Training and Cross-Model Validation

Scaling: The performance advantage of DO-ACP persists and widens with extended training ( $\sim$ 4B tokens).
Generalization: The findings generalize to DeepSeek-V2-Lite (64 experts) and GPT-OSS-20B (32 experts). While the absolute gap between scoring methods narrows as the expert pool size decreases, DO-ACP remains the top performer.
Efficiency: MoE-to-dense distillation is 1.6 $\times$ faster in wall-clock time than D2D pruning because the MoE teacher activates fewer parameters per token during the distillation phase.

Qualitative Analysis

Error analysis on MMLU chain-of-thought tasks reveals that DO-ACP models have the lowest rates of catastrophic failures (incoherence, repetitive loops) and knowledge errors compared to D2D and frequency-based methods. The diversity-aware selection appears to preserve factual and cultural knowledge more effectively, particularly in humanities and social sciences.

Significance and Claims

The paper claims to provide a simple, effective recipe for deploying MoE capabilities in memory-constrained environments: Score with DO-ACP, retain exactly the top- $k$ experts per layer (pure pruning), and distill with forward KL.

The authors emphasize that this approach transforms the "memory inefficiency" of MoE into a "dense efficiency" without sacrificing the performance gains derived from the MoE's specialized expert structure. They assert that MoE models provide a stronger starting point for distillation than dense teachers of matched parameter count, challenging the necessity of training dense models from scratch or pruning dense teachers. The work establishes that expert diversity, rather than just frequency or magnitude, is the critical factor in compressing MoE models into dense architectures.