💬 NLP

Less is MoE: Trimming Experts in Domain-Specialist Language Models

This paper introduces Fisher-MoE, a compression method that identifies and removes task-critical intermediate dimensions within FFN layers using Fisher importance, enabling significant memory and throughput gains in Mixture-of-Experts models while preserving their performance on general-purpose benchmarks.

Original authors: Haoze He, Xinkai Zou, Xuan Jiang, Xingyuan Ding, Ao Qu, Juncheng Billy Li, Heather Miller

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoze He, Xinkai Zou, Xuan Jiang, Xingyuan Ding, Ao Qu, Juncheng Billy Li, Heather Miller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, high-powered language model as a giant kitchen designed to cook any kind of meal, from simple toast to complex five-course feasts.

The Problem: A Kitchen Too Big to Fit in Your House

This kitchen (called a Mixture-of-Experts or MoE model) is incredibly smart because it has hundreds of specialized chefs (called "experts"). When you ask a question, a smart manager (the "router") picks the best few chefs to work on your specific task.

However, this kitchen is huge. It takes up so much space (memory) and requires so many chefs to be on standby that it's impossible to fit into a normal home (a standard computer or phone). You need to shrink the kitchen without losing the ability to cook good meals.

The Old Way: Firing Entire Chefs

Previous attempts to shrink this kitchen were like firing entire chefs.

If a chef didn't cook often, or if their apron was small, the managers would fire them.
The Result: This was a disaster. Even though you kept the "best" chefs, you accidentally fired the one person who knew the secret ingredient for making perfect math problems. Suddenly, the kitchen could still tell you a joke (knowledge), but it couldn't do basic arithmetic anymore. The whole system collapsed.

The New Discovery: It's Not the Chef, It's the Knife

The authors of this paper discovered something surprising: The problem isn't which chefs you keep; it's which specific tools they use.

Inside every chef's station, there are thousands of tiny tools (called intermediate dimensions).

Most tools are just sitting there, gathering dust.
But a tiny handful of specific tools are absolutely critical. For example, one specific "knife" might be the only thing that allows the kitchen to solve math problems, while another "spoon" is vital for writing code.

The old methods were like throwing away a whole chef's station because they used a "dusty spoon," not realizing that same chef also held the "critical math knife."

The Solution: "Fisher-MoE" (The Precision Scalpel)

The authors propose a new method called Fisher-MoE. Instead of firing chefs, they use a special scanner (called Fisher Importance) to look at every single tool in the kitchen.

The Scanner: It measures how much the kitchen's performance drops if you remove a specific tool.
- Analogy: Imagine testing every knife in the kitchen. If you remove "Knife A," the kitchen can still make toast. But if you remove "Knife B," the kitchen can no longer cut a tomato. The scanner identifies "Knife B" as critical.
The Trim: They don't fire anyone. Instead, they physically shrink the workstations. They remove the "dusty spoons" and "unused spatulas" (the low-scoring tools) but keep the "critical knives" (the high-scoring tools).
The Result: They can shrink the kitchen by 50% (removing half the tools) and the kitchen still works almost perfectly.
- It can still solve hard math problems.
- It can still write code.
- It can still answer trivia.
- Bonus: Because the kitchen is smaller, it cooks faster (21% faster) and fits in a much smaller space (45% less memory).

Why This Matters

Precision over Bluntness: The old way was a sledgehammer (firing whole people). This new way is a scalpel (removing only the useless parts of the tools).
The "Math" Mystery: They found that if you remove just 12 specific tools out of 1.35 million, the kitchen instantly forgets how to do math, even though it remembers everything else. This proves that complex skills are hidden in tiny, specific corners of the model, not spread out evenly.
Compatibility: This shrinking method works perfectly with other space-saving tricks (like "quantization"), meaning you can shrink the model even further without breaking it.

In a Nutshell

The paper says: Don't fire the experts; just trim their toolkits. By using a smart scanner to identify and remove only the useless "tools" inside the experts, we can make these giant AI models half the size, twice as fast, and just as smart as before.

Technical Summary: Less is MoE: Trimming Experts in Domain-Specialist Language Models

Problem Statement

Mixture-of-Experts (MoE) models have become a dominant paradigm for scaling language model capacity while maintaining efficient inference through conditional computation. However, their massive total parameter footprint creates significant barriers to deployment regarding memory, storage, and serving costs.

Prior approaches to compressing MoE models typically operate at the expert level, removing or merging entire experts based on heuristic importance metrics such as activation frequency, router scores, or weight magnitudes. The authors observe that these methods suffer catastrophic performance collapse when evaluated on general-purpose benchmarks beyond commonsense reasoning (e.g., mathematical reasoning, code generation, and complex reasoning tasks). The paper hypothesizes that this failure stems from two factors:

Inaccurate Importance Metrics: Existing heuristics fail to correctly estimate parameter importance.
Coarse Granularity: Compression at the expert level assumes capabilities are localized to specific experts. In reality, the paper argues that capabilities are distributed across experts but concentrated in a small subset of intermediate dimensions within the Feed-Forward Networks (FFNs). Removing an entire expert discards both redundant and critical dimensions simultaneously.

Methodology

1. Importance Attribution: Empirical Fisher Information

The authors propose using Empirical Fisher Information as a superior metric for ranking parameters. Unlike activation ratios, router scores, or weight magnitudes, Fisher information measures the sensitivity of the model's predictive distribution to perturbations in specific parameters.

Derivation: The method approximates the Fisher Information Matrix ( $F_\theta$ ) using ground-truth data from a calibration set. For an MoE expert $E_i$ with weights $W^{gate}_i, W^{up}_i, W^{down}_i$ , the Fisher score is computed for each intermediate dimension $j$ .
Granularity: The authors extend this scoring to the intermediate dimension level (rows of $W^{gate}/W^{up}$ and columns of $W^{down}$ ), rather than the entire expert or router level.
Validation: Empirical evidence shows that Fisher importance outperforms all baselines. Crucially, zeroing out just 12 of the ~1.35 million intermediate dimensions identified as "top" by Fisher scores collapses mathematical reasoning performance (GSM8K/MATH) to near-zero, while largely preserving factual knowledge. This confirms that Fisher successfully identifies a tiny set of task-critical dimensions.

2. Compression Strategy: Fisher-MoE

Building on the attribution analysis, the authors propose Fisher-MoE, a fine-grained compression method that operates within the FFN of each expert rather than removing experts entirely.

Mechanism: For each expert, the intermediate dimensions are ranked by their Fisher scores. The lowest-scoring fraction (e.g., 50%) is physically removed by resizing the weight matrices ( $W^{gate}, W^{up}, W^{down}$ ).
Allocation Rules: The paper explores three budget allocation strategies:
- IntDim-E: Keeps the top $(1-p)$ dimensions within each expert.
- IntDim-L: Pools dimensions across all experts in a layer and keeps the top $(1-p)$ per layer.
- IntDim-G: Pools dimensions across the whole model and keeps the top $(1-p)$ globally.
Preservation: This approach preserves the model's routing behavior and the number of active experts per token, but reduces the internal dimensionality ( $d_{ff}$ ) of the active experts.

Key Contributions

Structural Insight: The paper identifies a structural property of MoE models where capabilities are not localized at the expert level but are instead concentrated in a small subset of intermediate dimensions distributed across all experts. This explains the failure of expert-level compression.
Fisher Importance as an Attribution Tool: The authors define the intermediate dimension as the attributable unit and demonstrate that Empirical Fisher Information is a robust tool for distinguishing critical dimensions from redundant ones, outperforming activation, router, and magnitude-based metrics.
Fisher-MoE Algorithm: They propose a fine-grained compression method that physically resizes FFN matrices based on Fisher rankings. This method achieves high compression ratios without the catastrophic failure seen in expert-level pruning.

Results

The authors evaluate Fisher-MoE on Qwen1.5-MoE-A2.7B, OLMoE-1B-7B, and larger frontier models (Qwen3-30B-A3B, Qwen3.5-35B-A3B) across eight general-purpose benchmarks (Knowledge, Code, Reasoning, Math).

Performance Preservation: At a 50% MoE compression ratio (removing 50% of routed FFN parameters):
- Expert-level methods (even those using Fisher scores) suffer catastrophic drops on GSM8K, HumanEval, and MATH.
- Fisher-MoE (IntDim-E) preserves downstream performance, retaining capabilities in math and code reasoning that expert-level methods lose. In some cases (e.g., CEval, MultiArith), it matches or slightly exceeds the uncompressed base model.
- Scaling: The advantage holds for larger models (30B+ parameters) and long Chain-of-Thought (CoT) reasoning tasks.
Post-Training Recovery: When followed by Supervised Fine-Tuning (SFT) on domain data, Fisher-MoE recovers performance to levels competitive with the uncompressed Base + SFT ceiling, whereas expert-level methods remain significantly degraded.
Deployment Efficiency:
- Memory: Reduces weight memory by ~45%.
- Throughput: Improves inference throughput by 21% (1.21x speedup) on H100 GPUs.
- KV Cache: Freed VRAM allows for a 33.7% increase in KV cache budget, increasing request capacity.
Compatibility: Fisher-MoE composes effectively with 4-bit quantization (AWQ), further reducing the deployment footprint to ~1/8th of the original size without amplifying quantization sensitivity.
Generalization: Models compressed using math-calibrated Fisher scores (GSM8K) maintain strong performance on out-of-domain tasks (Knowledge, Code, Reasoning), indicating the method does not over-specialize to the calibration domain.

Significance and Claims

The paper claims that intermediate dimension granularity is the effective unit for both compression and capability analysis in MoE models. The significance of the work lies in:

Redefining Compression Granularity: It challenges the prevailing assumption that MoE compression must happen at the expert level, demonstrating that fine-grained trimming within experts is necessary to preserve complex reasoning capabilities.
Superior Attribution: It establishes Empirical Fisher Information as a critical tool for understanding where capabilities "live" inside MoE models, revealing that critical parameters are sparse and distributed.
Practical Deployment: It offers a practical pathway to deploy large MoE models with significantly reduced memory and latency requirements without sacrificing the performance required for high-stakes tasks like math and code generation.

The authors modestly note limitations, specifically that they did not explore 100B+ parameter backbones due to computational constraints, nor did they conduct extensive post-training recovery studies on diverse corpora to fully characterize the residual gap to uncompressed models. These are identified as directions for future work.