🤖 machine learning

Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

This paper introduces MoEXBench, a systematic benchmark that evaluates the complex interactions and deployment efficiency of combining expert pruning, weight quantization, and KV-cache compression across diverse Mixture-of-Experts LLMs, revealing that their joint performance cannot be predicted from isolated technique evaluations.

Original authors: Afsara Benazir, Chen Chen, Rongxiao Qu, Jiabo Huang, Jingtao Li, Lingjuan Lyu

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Afsara Benazir, Chen Chen, Rongxiao Qu, Jiabo Huang, Jingtao Li, Lingjuan Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are the engines behind many of today's most advanced artificial intelligence tools, capable of writing code, solving complex math problems, and holding conversations that feel surprisingly human. To achieve this level of intelligence, these models are built with billions of parameters, which are essentially the internal knobs and switches that determine how the system thinks. However, this massive size creates a significant problem: the models are so large that they require enormous amounts of computer memory to run, making them difficult to install on standard laptops or servers. To solve this, researchers have developed a specific type of model called a Mixture-of-Experts. Instead of using every single part of the brain for every question, these models route each piece of information to only a small, specialized group of experts, keeping the active work manageable while maintaining a huge total capacity. The challenge remains that even with this efficiency, the sheer volume of data these models need to store and the complex calculations they perform during long conversations still strain ordinary hardware.

A team of researchers set out to understand how to make these powerful models fit onto everyday computers without losing their intelligence. They focused on three main ways to shrink these models: removing unused experts, reducing the precision of the numbers the model uses, and compressing the memory needed for long conversations. While each of these methods had been studied on its own, no one had systematically tested what happens when they are stacked together. The researchers built a comprehensive testing framework to simulate the entire process of taking a massive model and compressing it for real-world use. They tested ten different models of varying sizes and designs, applying different combinations of these compression techniques to see how they interacted. Their goal was not just to see if the models got smaller, but to measure exactly how much their performance dropped and whether the smaller size actually translated to faster speeds on real hardware.

The study revealed a surprising truth: the amount of space saved does not predict how much intelligence is lost. The researchers found that cutting out unused experts, a process known as pruning, caused far more damage to the model's ability to reason than simply reducing the precision of its internal numbers. In fact, removing a relatively small portion of experts could degrade performance significantly, while aggressively reducing the bit-depth of the weights had a much gentler effect. This means that the specific method used to shrink the model matters far more than the total percentage of size reduction. Furthermore, the researchers discovered that the model's architecture, or its internal design, played a bigger role in how well it survived compression than its overall size. A larger model was not necessarily more robust; some smaller, differently designed models handled compression much better than their massive counterparts.

Another critical finding was that the average performance of a compressed model can be misleading. While a model might retain a high overall score, it could fail dramatically on specific types of tasks, such as coding or following complex instructions, while performing well on others. The researchers also examined how these compressed models behaved on actual computer chips, including high-end graphics cards and the processors found in modern laptops. They found that making a model smaller does not automatically make it faster. While compression successfully reduced the memory required to run the model, the time it took to process information did not always improve proportionally. In some cases, adding more compression layers actually slowed down the model because the computer had to spend extra time unpacking the compressed data. This was particularly true for long conversations, where the extra steps needed to manage the compressed memory offset the benefits of having less data to move.

The researchers concluded that there is no single "best" way to compress these models. Instead, the process must be viewed as a delicate balancing act where the choice of technique depends heavily on the specific hardware and the intended use. They found that expert pruning is the most aggressive and risky step, often dominating the loss in quality, while reducing the precision of numbers is a safer first step. Compressing the memory used for long contexts helps save space but does not guarantee a speed boost, and in some scenarios, it can even slow things down. The study suggests that developers should not simply aim for the smallest possible file size. Instead, they should test the entire pipeline of compression techniques together on their target hardware to find the sweet spot where the model remains accurate and responsive. By providing a clear map of how these different compression methods interact, the researchers have given the community a practical guide for deploying these powerful intelligence systems on the devices we use every day.

Technical Summary: Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

Problem Statement

Mixture-of-Experts (MoE) Large Language Models (LLMs) offer an efficient scaling mechanism by activating only a sparse subset of parameters per token. However, their deployment on commodity hardware is hindered by three primary factors: the large memory footprint of storing all expert parameters, routing imbalances, and the growth of the Key-Value (KV) cache during long-context inference.

While individual compression techniques—expert pruning, weight quantization, and KV-cache compression—are well-studied in isolation, their interactions when stacked in a realistic deployment pipeline remain poorly understood. Existing benchmarks fail to address:

Systematic Evaluation: Prior work focuses on dense LLMs or single compression axes, ignoring the unique tradeoffs introduced by sparse routing and expert redundancy in MoEs.
Composability: It is unclear whether combining pruning, quantization, and KV compression yields additive benefits, redundancy, or harmful interactions.
Real-World Efficiency: Standard metrics (e.g., parameter count, FLOPs) often fail to capture actual runtime behaviors such as dequantization overhead, memory layout constraints, and hardware-specific scheduling, leading to a disconnect between theoretical compression and actual throughput gains.

Methodology: MoE-XBench

To address these gaps, the authors introduce MoE-XBench, a systematic, end-to-end benchmark for evaluating composable MoE compression.

Workflow and Scope

MoE-XBench transforms baseline MoE checkpoints into deployable artifacts through a one-shot pipeline involving calibration, expert pruning, weight quantization, and KV-cache compression. The benchmark evaluates 10 MoE models (ranging from 30B to 235B parameters) across three architectural families:

Standard-attention (e.g., Qwen3, GLM-4.7)
Hybrid linear-attention (e.g., Kimi-Linear, Qwen3-Next)
Sliding-window attention (e.g., Step-3.5)

The evaluation spans seven workloads (knowledge, math reasoning, generative reasoning, instruction following, code generation, tool use, and long-context understanding) and tests 20%–50% expert pruning rates, 1–16 bit weight quantization, and various KV-cache precision settings.

Evaluation Suite

The framework employs an eight-module evaluation suite to measure both quality retention and deployment efficiency:

Stacked Compression Performance: Measures task quality retention after full compression.
Workload Robustness: Assesses consistency of quality across diverse tasks.
Architecture Reliability: Evaluates performance stability across different MoE families.
Quantization Sensitivity: Measures quality degradation as weight precision decreases.
KV-Cache Sensitivity: Measures quality retention as KV-cache precision decreases.
Expert Pruning Sensitivity: Measures quality retention as experts are removed.
Inference Efficiency: Reports peak memory, prefill latency, and decode latency.
Hardware Acceleration: Measures effective runtime speedup relative to the compression rate.

Experiments were conducted on commodity hardware (Apple M1 Max) and high-end accelerators (NVIDIA H100) using llama.cpp and GGUF quantization standards.

Key Results and Findings

The empirical study reveals non-trivial interactions among compression methods that contradict common assumptions:

Compression Ratio $\neq$ Quality Loss: The fraction of parameters or bits removed is not a reliable predictor of performance degradation. For instance, 19.35% expert pruning caused a 28.5% increase in perplexity (PPL), whereas 69.5% weight quantization (4-bit) increased PPL by only 0.7%. Expert pruning is the dominant source of degradation, far outweighing the impact of aggressive quantization.
Architecture Dominance: Pruning tolerance is governed more by model architecture design than by parameter scale. Hybrid linear-attention MoEs demonstrated greater robustness under extreme compression compared to standard attention MoEs. Larger models did not necessarily tolerate pruning better than smaller ones.
Average Accuracy Masks Failures: High average performance scores can hide significant workload-specific failures. For example, pruning caused severe drops in code generation and knowledge tasks while leaving math reasoning relatively stable.
KV-Cache Sensitivity: 8-bit KV compression (Q8) is robust across settings, but lower precisions (Q4, TurboQuant) cause significant quality loss, particularly in pruned models.
Non-Additive Interactions: Composable compression losses are not additive. At similar memory targets, different combinations (e.g., Prune+Quant vs. Prune+Quant+KV) yield vastly different PPLs.
Memory Reduction $\neq$ Throughput Gain: Smaller memory footprints do not guarantee faster inference.
- Adding KV compression reduced peak memory but often decreased decode throughput (e.g., dropping to 0.63x of the baseline) due to the overhead of on-the-fly dequantization, especially at long context lengths (32K).
- Aggressive low-bit quantization (e.g., IQ1_M) incurred high dequantization overheads that negated memory savings, resulting in lower throughput than moderate-bit quantization (Q4_K_M).
Pruning vs. Quantization for Hardware: Expert pruning provided more effective hardware acceleration than aggressive low-bit quantization. Pruning preserved the BF16 execution path, maintaining near-baseline speedups, whereas low-bit quantization introduced significant dequantization penalties that reduced effective speedup.

Significance and Claims

The paper positions MoE-XBench as the first systematic study of composable compression in MoE models. Its primary significance lies in shifting the evaluation paradigm from isolated accuracy-preserving steps to a holistic deployment pipeline perspective.

Key takeaways emphasized by the authors include:

Holistic Evaluation: MoE compression must be evaluated as a complete pipeline. Average accuracy alone is insufficient; practitioners must consider workload robustness, architecture reliability, and hardware-specific efficiency.
Pruning is Critical: Since expert pruning dominates quality degradation and interacts heavily with other techniques, it requires architecture-specific tuning rather than heuristic application.
KV Compression Trade-offs: KV-cache compression should be viewed primarily as a memory reduction technique for long contexts, not a guaranteed runtime optimization, as it can introduce latency penalties.
Hardware Dependency: The benefits of compression are decoupled from memory reduction in many cases; throughput gains depend heavily on the ability of the runtime to hide dequantization and cache-format overheads.

By releasing normalized module scores, compressed artifacts, and reproducible scripts, MoE-XBench enables practical comparisons of accuracy, memory, and latency across MoE model families and hardware backends, guiding the community toward more effective deployment strategies.