🤖 machine learning

WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

This paper introduces WiSP, a routing-aware working-set management system for Mixture-of-Experts models on low-resource hardware that dynamically pages expert weights and optimizes VRAM allocation between experts and the KV cache to significantly improve decode throughput without modifying the underlying serving engine.

Original authors: Jiamu Zhang, Liang Wu, Mayank Darbari, Liangjie Hong

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiamu Zhang, Liang Wu, Mayank Darbari, Liangjie Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Too Big to Fit" AI

Imagine you have a massive library of books (the AI model) that contains hundreds of billions of pages. You want to read a specific story on a tiny, portable e-reader (a standard computer graphics card, or GPU).

The problem is that the e-reader doesn't have enough memory to hold the whole library at once. However, for any single sentence you read, you only need a few specific pages from a few specific books. The rest of the library sits unused.

Current solutions try to solve this by keeping the whole library in a warehouse (the computer's main memory) and rushing to the warehouse to grab the pages you need every time you turn a page. This is slow because the trip to the warehouse (data transfer) takes a long time.

The WiSP Solution: The "Smart Librarian"

The authors created a system called WiSP (Working-Set Paging). Instead of rushing to the warehouse for every single page, WiSP acts like a smart librarian who watches what you are reading and keeps a small, curated stack of likely-to-be-needed books right on your desk (the GPU memory).

Here is how it works in three simple parts:

1. The "Smart Stack" (Expert Paging)

In AI models called "Mixture-of-Experts" (MoE), the model has many different "experts" (specialized sub-models), but it only uses a few for each word it generates.

Old Way: The system grabs all the experts for a layer, even if it only uses two of them. It wastes space and time.
WiSP Way: WiSP only keeps the specific experts you are actually using on your desk. If you need a new expert, it quickly swaps one out of the stack and grabs the new one from the warehouse.
The Result: Because it only moves the tiny pieces you actually need, it is much faster. The paper found that on small computers, this makes the AI run up to 2 times faster than the old method.

2. The "Shared Desk" (Memory Allocation)

Your desk (GPU memory) is very small. You have two things fighting for space:

The Expert Stack: The books you need to read right now.
The Context Notes (KV Cache): The notes you've written so far about the story so far, so you don't forget what happened.

If you fill your desk with too many books, you have no room for your notes. If you fill it with too many notes, you can't grab the books you need.

The WiSP Solution (MV-WSA): This is a smart rule that decides how to split your desk space. It constantly asks: "Is it more helpful to have more books on the desk, or more notes?"
It adjusts the split dynamically. If you are writing a long story, it gives you more room for notes. If you are switching topics quickly, it gives you more room for books. This ensures you never run out of space for either.

3. The "Crystal Ball" Myth (Why Prediction Didn't Help)

The researchers wondered: "What if we use a crystal ball (AI prediction) to guess which book you'll need next, and grab it before you ask for it?"

The Surprise: They found that in this specific setting (reading one sentence at a time), prediction doesn't make it faster.
Why? The "road" (data connection) between the warehouse and the desk is too narrow. Even if you guess perfectly, the truck can only carry so much at once. The bottleneck isn't guessing the right book; it's the speed of the truck.
The Real Value: The "crystal ball" is still useful, but not for speed. It helps the librarian know exactly how big the stack needs to be for you specifically. This allows the system to shrink the stack to the perfect size, freeing up more desk space for your notes.

The Bottom Line

The paper argues that running big AI models on small computers is a memory management problem, not just a computing problem.

WiSP is a tool that acts like a smart librarian, keeping only the necessary books on the desk and swapping them out efficiently.
It doesn't change the AI model itself; it just manages the memory better.
It makes the AI run significantly faster on small devices by stopping it from wasting time moving unnecessary data.
It teaches us that in this specific scenario, managing your workspace efficiently is more important than trying to predict the future.

The system works so well that it can even run massive AI models on a single computer card that previously couldn't handle them at all, without changing the quality of the answers the AI gives.

Technical Summary: WiSP – A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

Problem Statement
Modern Mixture-of-Experts (MoE) models, such as Qwen3-30B-A3B and Mixtral-8×7B, place the majority of their parameters in expert layers, yet only a small fraction of these experts are active for any given token. On commodity GPUs with limited VRAM, these models often cannot fit entirely in memory. The standard solution, static layer-level CPU offloading (e.g., --cpu-offload-gb in vLLM), keeps memory usage low but transfers all experts in a layer across the PCIe bus on every forward pass. This approach fails to exploit the sparsity of MoE models, effectively negating their efficiency benefits.

The authors frame low-resource MoE serving as a working-set management problem. In this regime, GPU memory is the limited fast store, and host memory is the backing store. Two distinct streams of memory demand compete for the same VRAM budget:

Expert Weights: A routing-dependent stream where only a subset of experts is needed per token.
KV Cache: A token-dependent stream required for autoregressive generation.

Existing systems often treat these as separate problems or target high-batch datacenter settings where computation can overlap with data transfer. However, in low-concurrency interactive serving (e.g., local coding assistants), computation is minimal per step, making PCIe bandwidth the primary bottleneck rather than compute or prediction accuracy.

Methodology: WiSP (Working-Set Paging)
The paper introduces WiSP, a system that treats the resident expert set as a cache over the model's routing behavior, managed jointly with the KV cache. WiSP consists of two main components:

Routing-Aware Expert Pager:
- Mechanism: WiSP plugs into an unmodified serving engine (e.g., vLLM) by leveraging an existing expert-map indirection hook (π). Instead of loading full layers, it maintains a small GPU scratch space containing only the "working set" of experts likely to be reused.
- Operation: At each step, the pager fetches only the missing routed experts from host memory (H2D) and evicts the least-recently-used (LRU) experts to make room.
- Guarantees: Because the underlying kernels and weights remain unchanged (only the mapping is redirected), WiSP produces byte-identical outputs to the baseline engine. It requires no new kernels, quantization changes, or model retraining.
MV-WSA (Marginal-Value Working-Set Allocation):
- The Allocation Problem: Since expert weights and the KV cache share the same VRAM budget, increasing the resident expert set reduces KV capacity, and vice versa.
- Strategy: MV-WSA determines the optimal split ( $f$ ) between the expert budget and the KV budget. It aims to equalize the marginal latency benefit per byte across both pools.
- Admission Floor: Crucially, the allocator enforces a minimum KV cache size ( $\kappa_{min}$ ) required to admit the current concurrency level. If the unconstrained optimal split would starve the KV cache, the system clamps the split to ensure the scheduler can initialize and run.
- Implementation Modes:
  - Offline Configurator: Analyzes a representative trace to set a fixed split at startup.
  - Online Controller: Dynamically resizes both pools during serving at "drained barriers" (points where no active requests hold memory blocks), physically moving bytes between the expert scratch and KV pool without exceeding the total budget.

Key Findings on Prefetching
A central empirical finding of the paper is that routing-based prefetching provides little to no benefit in single-stream, low-concurrency decode.

Bottleneck: Performance is limited by PCIe bandwidth, not prediction accuracy. In batch-1 decode, the computation per step is too small to hide PCIe transfers behind.
Result: Speculative prefetching often degrades performance (reducing throughput by ~50% in tests) because it competes for bandwidth and evicts experts needed for the immediate next step.
Reframing: The routing signal is valuable not for hiding latency, but for memory allocation. It helps identify the minimal "safe" resident set size, allowing the system to right-size the expert cache per user and allocate the freed VRAM to the KV cache.

Results
The authors evaluate WiSP on various MoE architectures (LLMs, VLMs, Hybrid SSM-MoE, and Diffusion models) using an NVIDIA H100 (with some experiments emulating smaller memory budgets).

Throughput Gains: Under matched VRAM budgets where the model does not fit, WiSP achieves up to 1.95× decode throughput compared to static CPU offloading. The gain is highest at intermediate memory budgets where WiSP can cache a significant portion of the reused expert set while the baseline still streams everything.
KV Capacity: By reducing the expert footprint, WiSP directly increases KV cache capacity. Experiments show that shrinking the expert cap can increase maximum concurrency by over 40% at the same total memory budget.
Allocation Performance:
- Offline: The MV-WSA configurator performs well on both prefill and decode, staying within a few percent of a per-workflow oracle in simulation, whereas fixed splits (e.g., 50/50) are ~20% worse.
- Online: The dynamic dual-resize controller improves throughput by 1.20× over the best static offline split by adapting to workload shifts (e.g., switching from long-context turns to tool-call bursts).
Generality: WiSP successfully serves models that cannot fit on a single GPU (e.g., Jamba-v0.1, MiniMax-M2) and works across modalities including Vision-Language Models (Kimi-VL) and Diffusion models (DiT-MoE, LLaDA-MoE), maintaining byte-identical outputs.

Significance and Claims
The paper claims that WiSP provides a unified framework for low-resource MoE serving by treating expert residency and KV caching as a single working-set allocation problem.

Shift in Design Paradigm: It argues that for low-concurrency serving, the focus should shift from "how to prefetch experts faster" to "how to allocate VRAM optimally between experts and KV cache."
Practical Impact: WiSP enables the serving of frontier-scale MoE models on commodity hardware without sacrificing output correctness or requiring specialized hardware (like Intel AMX).
Limitations: The authors explicitly note that WiSP is designed for the regime where the model does not fit in VRAM; if the model fits, static residency remains faster as paging adds unnecessary indirection. Additionally, the prefetching optimization is shown to be ineffective in single-stream decode, though it may be relevant for high-batch or multi-turn agent sessions where compute overlap is possible.

The work concludes that in memory-bound, single-stream decode, the routing signal is a memory lever (determining the size of the working set) rather than a speed lever (hiding latency), making intelligent allocation the primary driver of performance.