🤖 machine learning

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

This paper reveals that existing pre-trained Mixture-of-Experts (MoE) models inherently possess substantial intra-expert activation sparsity, which can be leveraged by modifying the vLLM execution pipeline to skip inactive neuron computations, achieving up to 2.5x speedup in MoE layer execution and 1.2x end-to-end speedup without any model retraining or parameter modification.

Original authors: Jongseok Park, Sunga Kim, Zhenyu Gu, Ion Stoica, Alvin Cheung

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jongseok Park, Sunga Kim, Zhenyu Gu, Ion Stoica, Alvin Cheung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Finding the "Sleeping Giants" in AI

Imagine a massive, high-end restaurant kitchen (the AI model) that has hundreds of specialized chefs (called Experts). In a standard "Mixture of Experts" (MoE) setup, when a customer orders a dish, the head chef (the Router) picks only a few specific experts to work on that order. For example, if the order is "spicy curry," the router might wake up the "Spice Expert" and the "Curry Expert," while the "Dessert Expert" and "Bread Expert" stay asleep.

This is already efficient because the kitchen doesn't run all 100 chefs for every single order. However, the researchers in this paper asked a new question: "Even when a chef is awake and working, are they using all their energy?"

They discovered that even the "awake" chefs are mostly just standing around doing nothing. They found that within a single expert, up to 90% of the neurons (the individual brain cells of the AI) are effectively "sleeping" or producing zero output for a given input.

The Problem: Why We Can't Just Add More Chefs

Previously, to make AI faster, researchers tried to make the kitchen more efficient by adding more chefs and waking up fewer of them (increasing "inter-expert sparsity"). But this is getting very hard.

The Training Struggle: If you have too many chefs and only wake up a few, the kitchen gets chaotic. Some chefs get all the work (load imbalance), while others never get trained and become useless (expert collapse).
The Limit: We are hitting a wall where we can't easily make the "who gets to work" selection more efficient without breaking the model.

The Solution: The "Lazy Chef" Strategy

Instead of trying to pick fewer chefs, the authors decided to make the individual chefs more efficient. They realized that even when a chef is working, they don't need to use every single tool in their toolbox.

The Analogy:
Imagine a master carpenter (an Expert) building a chair.

Old Way: The carpenter grabs every single hammer, saw, and screwdriver in the shed, even if they only need a hammer and a screwdriver. They carry the whole shed to the job site.
New Way (This Paper): The carpenter looks at the job, realizes they only need the hammer and the screwdriver, and leaves the saw and the other 90% of the tools in the shed. They only carry what they need.

The researchers found that in existing, pre-trained AI models (like Qwen, Llama, and DeepSeek), this "lazy" behavior is already happening naturally. The math inside the model naturally zeroes out most of the work. They just needed to teach the computer to skip that wasted work.

How They Did It: The "Skip-List" System

The team took a popular, high-speed AI engine called vLLM (think of it as the delivery service that gets the food from the kitchen to the customer) and upgraded it.

The Check: Before the "chef" starts cooking, the system quickly checks which tools (neurons) are actually needed.
The Skip: If a tool isn't needed (its value is too low), the system doesn't even load it into the memory or try to use it. It literally skips the calculation.
The Result: They built a special "fast lane" in the software. If the batch of orders is small (like a quiet lunch rush), the system uses this fast lane and skips the heavy lifting. If the kitchen is super busy (a huge dinner rush), it switches back to the standard, heavy-duty mode to keep things stable.

What They Found (The Results)

They tested this on eight different giant AI models, ranging from small ones (1 billion parameters) to massive ones (400 billion parameters).

Accuracy: They could skip up to 90% of the work inside an expert and the AI still got the answers right 95% of the time. It was like skipping 90% of the ingredients in a recipe and the dish still tasting 95% as good.
Speed:
- For the specific "cooking" part of the AI (the MoE layer), they saw speedups of up to 2.5 times.
- For the whole system (end-to-end), they saw a 1.2 times speedup.
Hardware: The speedup was most dramatic on smaller, less powerful graphics cards (like a consumer gaming card), proving this is a great way to run big AI models on cheaper hardware.

The Catch (Limitations)

The paper is honest about the limits:

The Gatekeeper: The system still has to check which tools are needed before it can skip them. This "check" takes some time and can't be skipped yet. It's like a manager still having to walk around the kitchen to tell the chefs which tools to grab, even if the chefs don't use them all.
Batch Size: The speedup is best when the AI is processing a few requests at a time. If you are processing thousands of requests simultaneously, the "checking" overhead becomes a bottleneck, and the speedup drops.

Summary

This paper didn't invent a new type of AI or train a new model from scratch. Instead, they looked at existing, powerful AI models and realized, "Hey, these models are already doing a lot of unnecessary work, even when they are 'active'."

By building a system that recognizes this wasted effort and simply skips it, they made these massive AI models run significantly faster and more efficiently, especially on hardware that isn't super expensive. It's a "smart skipping" technique that makes the current generation of AI models more practical to run.

Technical Summary: Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

Problem Statement
Mixture-of-Experts (MoE) architectures have become the standard for state-of-the-art Large Language Models (LLMs) due to their computational efficiency derived from sparse expert activation. However, further increasing sparsity through finer expert granularity (inter-expert sparsity) faces fundamental training challenges, including expert collapse, load imbalance, and representation collapse. These issues make it increasingly difficult to reduce the percentage of active experts without compromising model quality or training stability. Consequently, there is a need to explore complementary dimensions of sparsity that do not rely on architectural changes or retraining.

Methodology
The authors investigate intra-expert activation sparsity, a dimension of sparsity existing within the activations of individual experts in pre-trained MoE models. The methodology involves three primary components:

Empirical Analysis: The study analyzes eight off-the-shelf MoE models ranging from 1B to 400B parameters (including Granite, OLMoE, DeepSeek-V2, GPT-OSS, Qwen3.5, and Llama-4 variants). The authors measure benchmark accuracy (ARC-Challenge, ARC-Easy, HellaSwag, Winogrande, TruthfulQA) while progressively zeroing out the lowest-scoring neuron activations within active experts. They distinguish between "routed experts" (selected by the router) and "shared experts" (always active).
Observation of Sparsity Characteristics: The analysis reveals that activation outputs in MoE FFN layers follow a long-tailed distribution. A significant fraction of neurons produce near-zero activations, allowing for substantial sparsity (up to 90%) with minimal accuracy loss. The study finds that shared expert neurons have a higher per-neuron impact on accuracy compared to routed expert neurons; in large models like Qwen3.5-122B-A10B, keeping all shared expert neurons active while activating only ~2% of routed expert neurons suffices to maintain model capacity.
System Implementation in vLLM: The authors integrate this sparsity into the production-grade inference system vLLM. The implementation modifies the routed expert execution pipeline while preserving shared expert execution and router logic to maintain compatibility with expert parallelism and existing optimizations (HIP Graph, CUDA Graph, kernel fusion).
- Gate Projection: Executed densely using a batched matrix multiplication (BMM) kernel to determine activation scores.
- Activation Masking: A custom Triton kernel performs threshold-based masking in a single pass ( $O(n)$ ), avoiding the computational cost of top-k sorting ( $O(n \log n)$ ).
- Fused Up-Down Projection: A custom sparse kernel gathers weights for only the active neurons, packs them into a dense matrix in shared memory, and performs dense matrix-vector multiplication. This approach balances the irregularity of sparse access with the throughput of dense hardware accelerators.
- Dynamic Switching: To handle the trade-off between low-latency sparse execution and high-throughput dense execution, the system dynamically switches between the sparse and dense codepaths based on the batch size.

Key Contributions

Discovery of Intra-Expert Sparsity: The paper demonstrates that substantial intra-expert activation sparsity is readily available in existing pre-trained MoE models without any modification to model parameters or activation functions.
System Integration: The authors redesign the vLLM MoE execution pipeline to efficiently exploit both inter-expert and intra-expert sparsity on GPUs, maintaining full support for production-level features like expert parallelism and graph capture.
Performance Evaluation: The work provides a comprehensive evaluation across different model scales and hardware (AMD MI355X, NVIDIA H200, RTX4090), establishing the practical viability of this optimization.

Results

Sparsity Levels: The study identifies sparsity cutoffs where models retain 95% of their baseline accuracy. For example, the largest model tested (Llama-4-Maverick) retains accuracy with up to 90.8% intra-expert sparsity, while smaller models like Granite-1B-A400M retain it at 26.5%.
Layer Speedup: Intra-expert sparse execution achieves up to 2.5× speedup in MoE layer execution compared to the dense vLLM baseline. The speedup is most pronounced on resource-constrained hardware (e.g., RTX4090) and for larger models (Qwen3.5-122B-A10B), where memory and compute bottlenecks are more severe.
End-to-End Speedup: Due to Amdahl's Law (as the MoE layer constitutes roughly 45% of total execution time), the end-to-end inference speedup is more modest, achieving up to 1.2× on H200 and 1.1× on MI355X. The benefits are most significant for latency-bound workloads (e.g., decode-heavy scenarios with small batch sizes).
Accuracy Preservation: The threshold-based approach maintains 95% of the baseline average benchmark score up to the identified sparsity cutoffs (e.g., 85% target sparsity for Qwen3.5-35B-A3B).

Significance
The paper positions intra-expert activation sparsity as a practical and complementary optimization technique to inter-expert sparsity. It addresses the limitations of pushing inter-expert sparsity further by leveraging the inherent redundancy within the dense FFN of each selected expert. The authors claim this work lays the groundwork for future efficient LLM research, offering a method to reduce serving costs for large MoE models without requiring retraining or architectural changes. The integration into vLLM demonstrates that such sparsity can be effectively utilized in high-performance production systems, bridging the gap between theoretical sparsity and realized throughput.