💻 computer science

Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression

This paper proposes a structural pruning framework for Mixture-of-Experts (MoE) models that maximizes channel-score coverage via attribution-based approximation to achieve fine-grained redundancy removal, significantly reducing memory footprint while preserving accuracy under high compression ratios.

Original authors: Yifu Ding, Jiacheng Wang, Ge Yang, Yongcheng Jing, Jinyang Guo, Xianglong Liu, Dacheng Tao

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifu Ding, Jiacheng Wang, Ge Yang, Yongcheng Jing, Jinyang Guo, Xianglong Liu, Dacheng Tao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Too Many Chefs" Problem

Imagine a massive, high-end restaurant (a Mixture-of-Experts or MoE AI model). Instead of having one giant chef cook every dish, this kitchen has hundreds of specialized chefs (called Experts). For every order (a token of text), the head waiter (the Router) picks only a few chefs to work on that specific dish.

This system is brilliant because it's efficient: you only pay for the chefs you use. However, the restaurant is still huge, expensive to run, and takes up a lot of space (memory) because it employs hundreds of chefs, even if only a few are active at once.

The goal of this paper is to downsize the kitchen without ruining the food. They want to fire some chefs or shrink their workstations to save space and money, but they need to make sure the restaurant still serves 5-star meals.

The Problem with Old Methods: "The Coarse Knife"

Previous attempts to shrink these models were like using a coarse machete instead of a scalpel.

The Old Way: They would look at a whole chef and decide, "This chef is important, keep them," or "This chef is rarely called, fire them."
The Flaw: This is too blunt. Even an "important" chef might have a lot of wasted space in their kitchen. Maybe they have 100 cutting boards, but they only ever use the top 20. The other 80 are just gathering dust.
The Result: Old methods would keep the whole chef (wasting space on the 80 unused boards) or fire the whole chef (losing the 20 useful boards). They couldn't see the internal redundancy inside the chef's workspace.

The New Solution: A Three-Step "Smart Renovation"

The authors propose a new framework that acts like a precision architect. They don't just look at who is important; they look at where the value is inside each expert.

Step 1: The "Attribution" Detective (Finding the Real Value)

First, they need to know which parts of the model actually matter.

The Metaphor: Imagine trying to figure out which ingredients in a complex sauce actually make it taste good. You can't just guess based on who bought the ingredients (router stats) or how much they weigh (raw data).
The Innovation: They use a clever math trick called Attribution-Guided Loss Approximation. Instead of testing every single ingredient by removing it one by one (which takes forever), they use a "back-of-the-napkin" calculation to instantly estimate how much each part contributes to the final taste.
The Benefit: This is 20 times faster than previous methods. It's like having a super-fast taste-tester who can guess the impact of an ingredient without actually cooking the whole dish.

Step 2: The "Coverage" Map (Maximizing the Good Stuff)

Once they know which parts are valuable, they need to decide how much space to keep.

The Metaphor: Imagine you have a bucket of sand. Some grains are gold, some are dirt. You want to keep the gold but throw away the dirt.
The Old Way: "Keep 50% of the sand." This might accidentally keep a lot of dirt and throw away some gold.
The New Way (Coverage-Maximized): "Keep enough sand to cover 90% of the gold."
How it works: They realized that in these models, the "gold" (important information) is highly concentrated in just a few channels (like the top 20 cutting boards). So, they calculate exactly how many channels they need to keep to capture almost all the value. They stop cutting once they have "covered" the important information, even if that means keeping very few channels for some experts and more for others.

Step 3: The "Alignment" Tiler (Fitting the Puzzle Pieces)

Finally, they have a list of how many channels to keep, but there's a catch. Computer chips (hardware) are picky. They like numbers that are multiples of 64 or 128 (like fitting tiles perfectly into a grid). If you have 125 channels, the computer wastes space padding it up to 128, or it runs slowly.

The Metaphor: You have a pile of bricks of different sizes. You need to build a wall where every section must be exactly 128 bricks wide.
The Innovation: They use a fair redistribution method (called Hamilton's Largest Remainder) to shuffle the extra "leftover" space. If one expert is short by 3 bricks and another is short by 60, they give the extra space to the one who needs it most to get closest to the perfect 128-block size.
The Benefit: This ensures the shrunken model fits perfectly into the computer's memory, allowing it to run fast and use low-bit (compressed) storage without slowing down.

The Results: Smaller, Faster, Just as Smart

They tested this on famous models like Qwen and DeepSeek.

The Outcome: They managed to shrink the models by 5 times (5x compression) while keeping the accuracy almost exactly the same.
The Proof: On a model called Qwen3-30B, they reduced the memory footprint by 5.27 times. Even with aggressive cutting (50% pruning), the model still scored incredibly high on math and reasoning tests (like getting 94.5 on the MATH500 benchmark).

Summary

Think of this paper as the ultimate decluttering guide for AI.

Stop guessing which whole experts to fire.
Start looking inside to find the specific "golden channels" that hold the value.
Keep just enough to cover the gold, and trim the rest.
Reorganize the remaining pieces so they fit perfectly into the computer's hardware.

The result is a tiny, efficient AI that fits in your pocket but thinks like a giant.

Technical Summary: Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression

1. Problem Statement

Mixture-of-Experts (MoE) models offer high parameter capacity with manageable computation by activating only a subset of experts per token. However, deploying these models remains expensive due to substantial memory footprints and inference overhead. While structural pruning is a promising solution, existing methods face significant limitations:

Coarse Granularity: Prior approaches operate at the expert level, either removing entire experts or ranking them by importance. This fails to capture internal redundancy within "high-importance" experts, where information is often concentrated in only a few channels.
Inaccurate Importance Estimation: Existing metrics for expert importance (e.g., router weights, token usage, raw activation/gradient statistics) are often misaligned with the actual contribution of an expert to the final loss. Router statistics reflect selection frequency rather than utility, and raw statistics exhibit layer-dependent biases that make cross-layer comparison unreliable.
Inefficient Budget Allocation: Allocating pruning budgets based on coarse expert rankings leads to wasted capacity on low-contribution structures and fails to optimize the trade-off between compression and accuracy in large-scale MoEs with hundreds of experts.

2. Methodology

The authors propose a structural pruning framework tailored for MoEs, consisting of three core components: Attribution-Guided Loss Approximation (ALA), Coverage-Maximized Budget Allocation (CBA), and Alignment-Aware Redistribution (AAR).

2.1 Attribution-Guided Loss Approximation (ALA)

To address the high cost of expert-wise ablation (which scales linearly with the number of experts), the authors derive an efficient proxy for expert importance using a first-order Taylor expansion.

Derivation: The loss change induced by removing an expert $e$ is approximated as $\Delta L^{(e)} \approx -(\frac{\partial L}{\partial z_{\ell,e}})^\top z_{\ell,e}$ , where $z_{\ell,e}$ is the expert output and $\frac{\partial L}{\partial z_{\ell,e}}$ is the gradient with respect to that output.
Efficiency: This allows the estimation of importance for all experts in a layer via a single backward pass, reducing calibration time by 14–26 $\times$ compared to exhaustive ablation.

2.2 Coverage-Maximized Budget Allocation (CBA)

Motivated by the observation that information in MoE experts is highly concentrated in a small fraction of channels, the authors reformulate the pruning objective. Instead of allocating a fixed pruning ratio per expert, the goal is to maximize the cumulative channel score coverage under a global budget.

Mechanism: Channels within each expert are sorted by their scores (derived from ALA). The algorithm searches for a target coverage ratio $\rho$ for each expert such that the total number of retained channels meets the global pruning budget while maximizing the sum of retained channel scores.
Benefit: This approach prioritizes retaining high-contribution channels and aggressively prunes low-score tails, avoiding the waste of capacity on redundant structures that uniform or ratio-based methods incur.

2.3 Alignment-Aware Redistribution (AAR)

To ensure compatibility with low-bit quantization and hardware-efficient kernels (e.g., GEMM), channel dimensions must be multiples of a block size $a$ (e.g., 64 or 128).

Process: After CBA determines the optimal channel counts, AAR performs a downward alignment to the nearest multiple of $a$ . The "remainder" channels released by this rounding are redistributed to experts with the largest fractional remainders (using Hamilton's largest-remainder rule).
Goal: This ensures the pruned model satisfies kernel constraints without significant deviation from the optimal coverage-based allocation, preventing performance degradation due to padding or inefficient memory access.

3. Key Contributions

Observation of Internal Redundancy: The paper demonstrates that MoE information is highly concentrated in a small fraction of channels, rendering expert-level importance metrics too coarse for effective structural compression.
Novel Pruning Objective: The authors introduce channel score coverage as a pruning objective, reformulating capacity allocation to maximize coverage under a global budget. This is the first work to explicitly optimize for score coverage rather than simple pruning ratios.
Efficient Framework: They propose a scalable framework combining:
- ALA: An attribution-based approximation for expert importance estimation that reduces GPU hours by 20 $\times$ compared to ablation.
- CBA: A binary-search-based algorithm to allocate budgets that maximize score coverage.
- AAR: A redistribution strategy to satisfy low-bit kernel constraints.
Empirical Validation: Extensive experiments on DeepSeek and Qwen MoE architectures show that the method achieves over 5 $\times$ compression with negligible accuracy loss on general knowledge tasks and maintains strong performance on reasoning benchmarks.

4. Experimental Results

The framework was evaluated on DeepSeek-MoE-16B, DeepSeek-V2-Lite, Qwen1.5-MoE-A2.7B, and Qwen3-30B-A3B.

General Knowledge: Under a 25% pruning ratio combined with 4-bit quantization ( $P25\% Q4b$ ), the method achieved over 5 $\times$ storage reduction. On Qwen3-30B-A3B, the memory footprint was reduced by 5.27 $\times$ (from 61.06GB to 11.64GB) with an average accuracy drop of at most 1% on general benchmarks (e.g., MMLU, ARC).
Reasoning Tasks: On reasoning benchmarks (GSM8K, MATH500), the compressed models consistently approached or surpassed original counterparts. For instance, Qwen3-30B-A3B under 50% pruning retained a 94.5 score on MATH500 (compared to the baseline 92.8).
Efficiency: The method outperformed state-of-the-art baselines (including Wanda, MoNE, EAC-MoE, and PuzzleMoE) across diverse benchmarks. It also demonstrated robust performance under different routing strategies (top-1 to top-6) and calibration corpora.
Throughput: Alignment-aware redistribution significantly improved inference throughput. For Qwen3-30B-A3B, aligned dimensions (128) yielded 14.21 tokens/s compared to 10.23 tokens/s for non-aligned dimensions under 50% sparsity.

5. Significance and Claims

The paper claims to provide a practical path toward efficient MoE deployment by addressing the mismatch between expert-level importance and internal channel redundancy.

Scalability: By replacing exhaustive ablation with attribution-based approximation, the method scales to MoEs with hundreds of experts, a regime where previous loss-based methods were computationally prohibitive.
Fine-Grained Optimization: The shift from expert-level pruning to channel-level coverage maximization allows for more precise capacity allocation, preserving critical information while removing noise.
Hardware Compatibility: The integration of alignment-aware redistribution ensures that the compressed models are not only smaller but also compatible with low-bit quantization and high-throughput inference kernels, addressing a critical bottleneck in real-world deployment.

The authors position this work as a foundational step in moving MoE compression from coarse, expert-level decisions to fine-grained, channel-level optimization, enabling the deployment of large-scale MoE models with significantly reduced memory and compute requirements.