🤖 machine learning

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

This paper introduces DECO, a sparse Mixture-of-Experts architecture that achieves dense-comparable performance with only 20% expert activation and a 3.00× inference speedup on end-side devices by leveraging ReLU-based routing, learnable expert-wise scaling, and a novel NormSiLU activation function.

Original authors: Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-end restaurant. You want to serve the best possible food (high performance) to as many customers as possible, but you have two strict rules:

Low Cost: You can't afford to have every chef cooking every single dish for every customer (low computation).
Small Kitchen: You don't have enough space to store every single recipe book and ingredient for every chef (small storage).

For a long time, the industry standard was to hire a huge team of chefs (a "Dense" model) where everyone tries to cook everything. This tastes great but is incredibly expensive and requires a giant kitchen.

Then, people tried a "Mixture of Experts" (MoE) approach. This is like having a huge team of 100 specialized chefs, but for each order, you only wake up 20 of them to cook. This saves money on cooking time. However, you still have to keep the recipes and ingredients for all 100 chefs in your kitchen. If your kitchen is small (like a phone or a laptop), you run out of space, and the time spent running back and forth to get the right chef's ingredients slows everything down.

Enter DECO.

The authors of this paper created a new restaurant design called DECO. Their goal was to build a "sparse" kitchen (where only a few chefs work at a time) that tastes just as good as the giant "dense" kitchen, but fits in a tiny space and runs fast.

Here is how they did it, using simple analogies:

1. The Smart Waiter (The Router)

In a normal MoE restaurant, the waiter (the router) is a bit rigid. They might say, "For every table, exactly 2 chefs must cook."

DECO's Innovation: They gave the waiter a flexible, differentiable tool (called ReLU-based routing). Now, the waiter can look at the specific order and decide, "This dish needs 3 chefs," or "That one only needs 1." It's a fluid decision based on the food, not a rigid rule.
The Scaling Factor: Sometimes, one chef is naturally louder or stronger than another. DECO gives the waiter a set of adjustable volume knobs (learnable scaling) for each chef. If Chef A is naturally quieter, the waiter turns up their volume so their contribution is balanced with the louder chefs.

2. The Specialized Chefs (The Experts)

The chefs in DECO are designed differently to be more stable.

The "NormSiLU" Apron: The authors noticed that when chefs use a standard apron (activation function), they sometimes get confused or stop working entirely (vanishing signals). They invented a new apron called NormSiLU. Think of this as a special uniform that calibrates the chef's energy before they start cooking. It keeps their output steady and prevents them from burning out or becoming too quiet.
No Gatekeepers: Most restaurants use a "gatekeeper" chef who decides if the main chef should even start cooking. DECO found that for their flexible waiter, it's actually better to remove the gatekeeper. The chefs work better when they can just start cooking based on the waiter's signal without an extra layer of "permission."

3. The Result: The "Ideal Triangle"

The paper claims DECO achieves a "holy grail" of three things simultaneously:

High Performance: It tastes just as good as the giant, expensive restaurant (Dense models).
Low Cost: It only uses 20% of the chefs at any given time, saving massive amounts of energy.
Small Storage: Because the design is so efficient, the total "kitchen footprint" (total parameters) is small enough to fit on end-side devices like phones, without needing to swap ingredients in and out of storage constantly.

4. The Proof

Taste Tests: When they tested DECO against other models, it matched the performance of the giant "Dense" models and beat other "Sparse" models, all while using the same amount of training data and total ingredients.
Speed: They built a custom engine (acceleration kernel) to run this restaurant. On real hardware (like a high-end graphics card and a Jetson device), DECO was 3 times faster than the standard way of running these models.

The Catch (Limitations)

The authors are honest about what they haven't tested yet. They haven't tried this restaurant design for "fine-tuning" (teaching the chefs specific new skills after the main training) or "reinforcement learning" (teaching them through trial and error). They suspect there might be some instability there, so they are currently building a bigger version to test those specific scenarios.

In summary: DECO is a clever re-imagining of how AI models work. It proves that you don't need a giant, bloated kitchen to get great food. With the right waiter, the right uniforms, and a flexible approach, you can get the same high-quality results with a tiny, fast, and efficient kitchen.

Technical Summary: DECO – Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

Problem Statement
While Mixture-of-Experts (MoE) architectures scale model capacity without proportionally increasing computation, they traditionally suffer from massive total parameter footprints. This creates significant storage and memory-access bottlenecks that hinder efficient deployment on end-side devices, which require high performance, low computational cost, and minimal storage overhead. Existing MoE research often prioritizes performance and low computation but neglects the "storage overhead" objective. Furthermore, many MoE models achieve their results by training on substantially more tokens than dense baselines under the same compute budget, rather than matching the exact token count. The paper poses the question: Can a sparse MoE model achieve performance comparable to a dense model given the same total parameter budget and the same number of training tokens?

Methodology
The authors propose DECO (DEnse-COmparable), a sparse MoE architecture designed to match dense Transformer performance under identical constraints. The architecture is built upon three core design pillars:

Router Design:
- ReLU-based Routing: Unlike conventional non-differentiable TopK routing, DECO employs ReLU-based routing. This allows for flexible, token-dependent activation ratios and smooth learnability.
- Learnable Expert-wise Router Scaling: To address output scale imbalances between shared and routed experts and accommodate expert heterogeneity, DECO replaces fixed scalar scaling factors with learnable vectorized scaling factors ( $\alpha$ ). This calibrates the contribution of individual routed experts.
Expert Design:
- NormSiLU Activation: The authors identify that coupling ReLU-based routing with vanilla SiLU-activated experts leads to surging activation ratios and vanishing output magnitudes. To resolve this, they introduce NormSiLU, which applies a dual-stage normalization prior to the SiLU operator:
  - Inter-expert mean normalization: Centers expert up-projection weights around zero.
  - Intra-expert RMS normalization: Maintains consistent activation magnitudes.
    This stabilizes activation trends, reduces the need for aggressive sparsity regularization, and ensures robust parameter utilization.
- Non-Gated MLP Experts: Contrary to the standard preference for gated MLPs (SwiGLU), DECO utilizes non-gated MLP experts. Empirical results show that non-gated experts exhibit more stable activation ratio trends when paired with ReLU-based routing, whereas gated variants suffer from unstable, surging activation dynamics that require heavy regularization.
Adaptive Sparsity Regularization:
- DECO employs an adaptive sparsity regularization based on router entropy. Unlike static coefficients, the regularization strength ( $\lambda$ ) is dynamically scaled based on the current sparsity level relative to the target, ensuring a stable activation ratio centered precisely at the desired level.

Key Results
Experiments were conducted across four parameter scales (0.11B to 1.18B) with a fixed training token count (approx. 40x parameter count) and a 20% routed-expert activation ratio.

Dense Comparability: DECO achieves performance parity with dense Transformer baselines (using SwiGLU FFNs) in terms of Perplexity (PPL) on the C4 validation set and average accuracy on commonsense reasoning benchmarks, despite activating only 20% of experts.
Superiority over MoE Baselines: DECO outperforms established MoE baselines (TopP, DeepSeek-V3, ReMoE, BlockFFN) of the same scale and activation ratio.
Ablation Insights:
- Learnable expert-wise scaling significantly improves performance over fixed or scalar scaling, addressing observed heterogeneity in expert output norms.
- NormSiLU is critical; removing its normalization steps leads to surging activation ratios and vanishing SiLU outputs, degrading performance.
- Non-gated experts are superior to gated experts specifically within the context of ReLU-based routing.
Hardware Acceleration: A specialized acceleration kernel leveraging CUTLASS and tensor cores was implemented. On real hardware (NVIDIA RTX 4090 and Jetson AGX), DECO achieved a 3.00× speedup over vanilla dense inference by exploiting sparse activation to reduce memory access overhead.

Significance and Claims
The paper claims that DECO successfully realizes the "ideal triangle" of end-side MoE deployment: high performance, low computation cost, and small storage overhead. Its significance lies in demonstrating that sparse MoE models can match dense performance without requiring excess training tokens or massive parameter footprints.

The authors attribute this success to a fundamental revision of MoE design that aligns the routing mechanism (ReLU-based), expert activation (NormSiLU), and expert structure (non-gated) to create a stable training dynamic. They note that dense-comparability is conditional, relying on factors such as activation ratio, expert granularity, and data heterogeneity. The paper suggests that high-entropy, multi-domain datasets are particularly well-suited for this sparse architecture, as the model can effectively activate specialized parameter subsets for different tasks.

Limitations
The authors acknowledge that they have not yet evaluated DECO in supervised fine-tuning (SFT) or reinforcement learning (RL) stages, where MoE architectures may face unique challenges like router instability. Additionally, the scaling laws for the activation ratio threshold required for dense-comparability across larger model sizes and diverse data distributions require further investigation.