🤖 machine learning

MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation

MAPLE is a plug-and-play framework that optimizes the efficiency of pretrained Mixture-of-Experts (MoE) models by analytically and genetically reallocating expert budgets heterogeneously across layers based on sensitivity, achieving superior accuracy and significant latency reductions without requiring weight modifications or retraining.

Original authors: Lie Li, Wen Li, Junxiao Shen, Gusheng Hu

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lie Li, Wen Li, Junxiao Shen, Gusheng Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence often relies on massive digital brains called Transformers, which process language by passing information through a long chain of layers. Think of these layers as a series of processing stations where the model learns to understand context, grammar, and meaning. In recent years, engineers have made these models more efficient by using a design called Mixture-of-Experts. Instead of every station using the same amount of computing power, the model activates only a few specialized sub-networks, or "experts," for each piece of text it reads. This allows the system to be very large and capable without requiring impossible amounts of energy to run. However, a standard rule has governed how these experts are used: every single layer in the chain is forced to activate the exact same number of experts, regardless of whether that layer actually needs that much help.

A team of researchers at the University of Bristol has challenged this uniform rule, proposing that different layers in an AI model have very different needs. Some layers are highly sensitive and require many experts to function correctly, while others are redundant and can operate with far fewer without losing accuracy. They developed a new method called MAPLE, which acts as a plug-and-play tool. This means it can be applied to existing, pre-trained models without needing to retrain them or change their internal weights. The system works by first testing how much each layer's performance drops if you reduce the number of experts it uses. It then uses these measurements to create a custom map, shifting the computing budget away from layers that can spare it and directing those resources to the layers that need them most.

The results of this approach are striking. When the researchers tested MAPLE on four different large language models, they found that simply redistributing the existing resources was more effective than using them all evenly. In one specific test using a model called DeepSeek-MoE-16B, the new method managed to outperform the original, fully-loaded model while using only 75% of the available experts. On several reasoning benchmarks, the accuracy improved significantly; for instance, on a science question-answering task, the score jumped from 65.09 to 71.40, and on a logical reasoning test, it rose from 48.49 to 51.50. This demonstrates that the old assumption of uniformity was holding the models back, and that a smarter, uneven distribution of effort yields better results.

Beyond just getting smarter answers, this method also makes the models faster and cheaper to run. Because fewer experts are being activated, the computer has less work to do for every question it answers. When the researchers implemented MAPLE in a real-world serving system, they measured a 32.2% reduction in the time it took to generate a response on a single graphics card. At the same time, the system could handle 47.4% more requests per second. These gains were achieved without altering the model's core structure or discarding any of its learned knowledge; the system simply learned to use its existing parts more efficiently. The researchers confirmed that this improvement was not a fluke of a specific test but a robust finding that held up across different types of reasoning tasks and even when the model was scaled up or down.

The study also explored how this method handles the complex task of multi-step reasoning, where the model must generate a long chain of thoughts to solve a problem. Even in these demanding scenarios, the redistributed model maintained its accuracy while cutting the total time required to solve the problems by more than half in some cases. The researchers showed that the method is flexible enough to work with different models and that the allocation plan it creates can be reused for various tasks without needing to be recalculated every time. By proving that a one-size-fits-all approach is suboptimal, this work offers a practical and principled way to make large artificial intelligence systems more efficient, powerful, and ready for real-world deployment.

Technical Summary: MAPLE – MoE Adaptive Plug-and-Play Layer-wise Expert Allocation

Problem Statement
Current Sparsely-activated Mixture-of-Experts (MoE) Transformers universally employ a uniform routing strategy, activating the same fixed number of experts across all layers. This convention ignores well-documented heterogeneity in layer-wise redundancy and expert importance. The authors argue that this uniformity is systematically suboptimal: it fails to direct computational capacity toward layers that are sensitive to expert count variations while wasting resources on redundant layers. Existing solutions to improve MoE efficiency, such as unstructured pruning, expert merging, or structured compression, typically require weight reconstruction, additional calibration data, retraining, or permanent architectural changes, making them unsuitable for "plug-and-play" deployment on pretrained models.

Methodology
The paper proposes MAPLE (MoE Adaptive Plug-and-Play Layer-wise Expert Allocation), a framework that reallocates the routed-expert budget heterogeneously across layers of a pretrained MoE Large Language Model (LLM) without modifying weights, retraining, or altering the model architecture. MAPLE operates in three distinct phases:

Layer-wise Sensitivity Modelling:
The framework probes each layer individually by varying the number of routed experts ( $e$ ) while keeping other layers fixed at the baseline count ( $k_0$ ). It records the validation accuracy $A_l(e)$ for each configuration.
- Single-layer optimum ( $k^*_l$ ): The expert count that maximizes accuracy for a specific layer.
- Sensitivity Measures: To quantify how strongly a layer responds to changes in expert count, MAPLE employs three lightweight metrics:
  - $s^{acc}_l$ : The range of accuracy variation ( $\max A_l(e) - \min A_l(e)$ ).
  - $s^w_l$ : The variance of parameter tensors within the layer.
  - $s^{out}_l$ : The Mean Squared Error (MSE) between the layer's output with $e$ experts and its output with the baseline $k_0$ experts.
- The authors default to the accuracy range ( $s^{acc}_l$ ) as the primary sensitivity metric.
Sensitivity-Guided Closed-Form Allocation:
MAPLE formulates expert reallocation as a constrained optimization problem. The goal is to find an allocation vector $\mathbf{k} = [k_1, \dots, k_L]$ that stays close to each layer's single-layer optimum ( $k^*_l$ ), weighted by sensitivity ( $s_l$ ), subject to a global expert budget $K$ .
- The objective minimizes the weighted squared deviation: $\min \sum w_l (k_l - k^*_l)^2$ where $w_l = s_l$ .
- A Lagrange multiplier approach yields a continuous solution ( $k^{cont}_l$ ). Surplus or deficit experts relative to the budget are assigned preferentially to low-sensitivity layers.
- The continuous solution is clipped to feasible bounds $[k_{min}, k_{max}]$ and converted to an integer allocation using the largest-remainder method.
Sensitivity-Constrained Genetic Refinement:
To address rounding artifacts and cross-layer interactions, a lightweight genetic algorithm refines the closed-form solution.
- Mutation: Candidates are generated via Gaussian mutation where the variance is inversely proportional to the layer's sensitivity ( $\sigma^2_l \propto 1/s_l$ ), ensuring high-sensitivity layers receive smaller perturbations.
- Selection: The top two candidates (elites) are retained per generation, and the population is regenerated without crossover.
- This process converges rapidly (often within a few generations) to a final allocation $\mathbf{k}_{best}$ .

Key Contributions

Plug-and-Play Framework: MAPLE is the first method to propose a fully post-hoc, weight-free framework for heterogeneous expert allocation in pretrained MoE models. It requires no retraining, weight updates, or architectural modifications.
Sensitivity-Guided Optimization: The paper introduces a closed-form analytical solution for expert reallocation based on layer-wise sensitivity, refined by a sensitivity-constrained genetic search. This avoids the computational cost of exhaustive search or reinforcement learning.
Performance with Reduced Budget: The method demonstrates that heterogeneous allocation can outperform uniform allocation even when the total routed-expert budget is reduced (e.g., to 75%), effectively surpassing the performance of the original full-budget uniform baseline.

Experimental Results
The authors evaluated MAPLE on four MoE models (DeepSeek-MoE-16B, DeepSeek-V2-Lite, Moonlight-16B-A3B, and OLMoE-1B-7B) across seven benchmarks (ARC-E, ARC-C, BoolQ, PIQA, RTE, GSM8K, BBH).

Accuracy Gains: Under a 75% routed-expert budget, MAPLE outperformed uniform baselines and pruning-based methods (SparseGPT, Wanda, EPP, MoNE, LExI) across all models.
- On DeepSeek-MoE-16B, using only 75% of experts, MAPLE improved accuracy on ARC-E (65.09% $\to$ 71.40%), ARC-C (48.49% $\to$ 51.50%), and BoolQ (80.03% $\to$ 82.38%) compared to the original 100% uniform baseline.
- On OLMoE-1B-7B, MAPLE achieved the best average accuracy, improving by 4.13 points over the full-budget baseline.
Long-Form Reasoning: On GSM8K and BBH, MAPLE maintained competitive accuracy (within 0.61 points on GSM8K) while significantly reducing evaluation time (53.0% reduction on GSM8K).
Deployment Efficiency: Implemented in SGLang, MAPLE reduced single-GPU end-to-end serving latency by 32.2% and improved throughput by 47.4%. This is achieved by bypassing multi-expert dispatch logic for layers assigned a single active expert ( $k=1$ ).
Robustness: Ablation studies confirmed that the accuracy range metric is the most effective sensitivity signal. The method showed stability across random seeds and generalizability across tasks (cross-task transfer), indicating that the identified sensitivity patterns are intrinsic to the model layers rather than overfitting to specific calibration sets.

Significance and Claims
The paper claims that MAPLE establishes a "principled and practical axis" for improving MoE efficiency. Its primary significance lies in demonstrating that well-designed heterogeneous allocation is more effective than simply activating more experts. By proving that capacity can be redistributed to match the heterogeneous demands of different layers without retraining, MAPLE offers a cost-effective path to deploying more efficient MoE models. The authors emphasize that their approach avoids the overhead of existing pruning or compression methods, making it immediately applicable to existing pretrained models in production environments.

Limitations and Future Work
The authors acknowledge that their current evaluation is limited to specific GPU environments (NVIDIA RTX A6000) and serving engines (SGLang, vLLM). Future work involves evaluating MAPLE on diverse hardware platforms (CPUs, edge devices) and under broader real-world inference scenarios to assess how reduced expert computation translates to actual latency, memory, and energy benefits in different system constraints.

Technical Summary: MAPLE – MoE Adaptive Plug-and-Play Layer-wise Expert Allocation

More like this