Technical Summary: MOE-PRISM

1. Problem Statement

Mixture-of-Experts (MoE) architectures have become central to large language models (LLMs) for scaling capacity without proportional computational increases. However, existing MoE serving systems typically execute all requests under a fixed routing configuration (a static top-$k$), failing to exploit the heterogeneous computation requirements across different requests. While some approaches achieve "request-level compute elasticity" by cascading multiple models or using early-exit mechanisms, these methods introduce significant memory/deployment costs or require specialized training and architectural changes.

The paper identifies two primary gaps preventing effective request-level elasticity within a single MoE instance:

1. Coarse-Grained Routing Units: Native MoE routing treats entire experts as indivisible units. Reducing the routing budget $k$ by one removes a whole expert, causing a sharp, discrete drop in model quality and limiting the available operating points in the computation-quality trade-off space.
2. Inefficient Heterogeneous Batching: Serving systems like vLLM use continuous batching. When requests with different routing budgets are mixed, the system typically executes the batch at the maximum requested $k$ to ensure quality, leading to "over-serving" and wasted computation. Conversely, splitting batches to preserve budgets fragments execution, reducing throughput.

2. Methodology: MOE-PRISM

MOE-PRISM is a model and system support framework designed to enable request-level compute elasticity within a single MoE serving instance. It addresses the problem through two co-designed components:

A. Model Refactoring Engine (Model-Side)

This component transforms the model to expose a denser spectrum of routing operating points without requiring end-to-end retraining.

• Expert Decomposition: The engine profiles intermediate neuron activations on calibration data. It observes that activation mass within an expert is concentrated in token-dependent subsets of neurons.
• Sub-Expert Partitioning: Using a partition optimization solver (greedy initialization followed by simulated annealing), each monolithic expert is sliced into $N$ finer-grained, disjoint sub-experts. The partitioning aims to group neurons that co-activate across tokens, ensuring that skipping a sub-expert minimizes quality loss.
• Gating Reconstruction: The system reconstructs the routing interface for the new sub-experts via two paths:
  1. Training-Free Proxy: A hierarchical selection mechanism that uses co-activation statistics to score sub-experts, allowing immediate deployment.
  2. Gate-Only Fine-Tuning: A low-cost adaptation where only the router weights are fine-tuned (with the rest of the model frozen) to align with the expanded sub-expert space, maximizing fidelity.

B. k-Aware Serving Runtime (System-Side)

This component manages the execution of heterogeneous routing budgets efficiently.

• k-Aware Serving Scheduler:
  • Offline (Throughput-Oriented): Uses temporal separation via quality-tiered batching. Requests are sorted by their routing budget ($k$) and grouped into tiers. The scheduler decides whether to dispatch a residual batch immediately or merge it with the next tier based on a local throughput heuristic, minimizing over-serving while maintaining batch fullness.
  • Online (Latency-Sensitive): Uses spatial separation via lane-based routing. Data-parallel (DP) ranks are partitioned into "low-$k$" and "high-$k$" lanes with a soft, adaptive boundary. This prevents mixing vastly different budgets in the same execution stream while avoiding rigid partitioning that could cause queue buildup.
• Segmented Graph Executor: To handle the increased routing states without exploding CUDA Graph memory usage, the executor splits the Transformer block at the Attention-MoE boundary.
  • Attention Segments: Shared across different $k$ values (routing-insensitive).
  • MoE Segments: Specialized by the routing budget $k$ (routing-sensitive).
  • This design allows the system to retain CUDA Graph efficiency while supporting dynamic $k$ execution.

3. Key Contributions

1. Model Refactoring Method: A technique to decompose monolithic experts into sub-experts and reconstruct the gating network, enabling a single model to expose denser routing operating points. It supports training-free deployment and optional gate-only fine-tuning.
2. k-Aware Serving Runtime: An extension to vLLM that combines budget-aware scheduling (temporal and spatial separation) with efficient dynamic-$k$ execution via a segmented graph executor.
3. System Integration and Evaluation: Full integration into vLLM and evaluation on three representative MoE models (OLMoE, DeepSeek-V2-Lite, Qwen3), demonstrating practical elasticity.

4. Experimental Results

The authors evaluated MOE-PRISM on three models using the vLLM framework across A100, RTX A6000, and H800 GPUs.

• Throughput (Offline): MOE-PRISM improved offline inference throughput by up to 33.9% compared to baselines. The gains stem from both the ability to execute lower $k$ (via refactoring) and the scheduler's ability to group similar requests to avoid over-serving.
• Latency (Online): Under heterogeneous workloads, MOE-PRISM reduced Time-to-First-Token (TTFT), particularly at high loads where baselines suffer from queueing delays and tail latency.
• Accuracy: The model refactoring largely preserved the accuracy of the original models. In some cases (e.g., OLMoE and Qwen3 at lower budgets), the refactored models achieved modestly better perplexity or downstream task accuracy, suggesting the new structure supports effective expert selection under constrained computation.
• Ablation Studies:
  • Refactoring: Confirmed that fine-grained decomposition significantly reduces routing-budget overshoot compared to coarse-grained rounding.
  • Segmented Graph: Showed that the executor reduces graph memory usage by ~84% (compared to a standard k-aware approach) with negligible throughput penalty (<0.2%).
  • Scheduling: Demonstrated that dynamic-$k$ execution alone is insufficient; explicit scheduling is required to prevent cross-$k$ interference and realize end-to-end performance gains.

5. Significance and Claims

The paper claims that MOE-PRISM demonstrates the practical feasibility of request-level compute elasticity in MoE serving. By treating the routing top-$k$ as a runtime control knob rather than a fixed architectural setting, the system can:

• Serve requests with varying quality/computation requirements within a single deployed model instance.
• Translate heterogeneous routing budgets into tangible end-to-end latency and throughput improvements.
• Avoid the memory and deployment overheads associated with cascading multiple models.

The authors note a limitation: the evaluation is currently restricted to two-GPU testbeds, which represents the minimal non-trivial environment for validating the core mechanism. They do not claim immediate scalability to massive clusters without further engineering, but the results establish the foundational model-system co-design principles for elastic MoE serving.