Which Expert Misses Are Worth Waiting For? Substitution and Caching for Mixture-of-Experts Inference on a Consumer GPU

Technical Summary: Which Expert Misses Are Worth Waiting For?

Problem Statement
Mixture-of-Experts (MoE) language models, such as Qwen3-Next-80B-A3B, offer high capacity but require storing all experts in memory, exceeding the VRAM of consumer GPUs (e.g., 24 GB). Offloading experts to CPU memory or SSDs allows these models to run on consumer hardware but introduces latency when "cache-miss" experts (those not currently in GPU memory) are needed. Traditional solutions include fetching every miss (high latency), computing on the CPU (slow arithmetic), or prefetching (risk of misprediction). Alternative approaches change the computation by substituting a missed expert with a resident one. However, it remains unclear which substitutions cause significant quality loss in models with hundreds of experts per layer, whether a single metric can guide cache policies, and when a cache should admit an expert that is repeatedly substituted.

Methodology
The study focuses on Qwen3-Next-80B-A3B (512 experts per layer, selecting 10) with approximately 25% of experts resident in GPU memory. The research employs three distinct tools to evaluate substitution policies:

1. Emulation: A PyTorch implementation that computes model outputs exactly for all policies on the same token sequences, idealizing fetch latency to isolate the impact of substitution on quality (measured by teacher-forced perplexity).
2. Router-Logit Simulator: A C++ tool that replays router logits to compare policies quickly without computing the full model, estimating time per token and cache states.
3. Engine: A modified llama.cpp implementation that performs real expert fetching, substitution, and caching on consumer GPUs (RTX 3090).

The study defines substituted gate mass ($m$) as the summed normalized gate weight of substituted experts. It tests several policies:

• Fetch every miss: The reference baseline.
• 0.15 Rule: Synchronously fetches missed experts if they are the router's first choice or have a gate weight $\ge 0.15$; substitutes the rest.
• First-choice Rule: Fetches only the first-choice miss; substitutes all others.
• Background Admission: A rule that accumulates the gate weight of substituted experts over time and admits them to the cache in the background once a threshold (e.g., 0.5) is reached.

Key Contributions and Findings

1. Source of Quality Loss: In emulation, most quality loss stems from substituting a few heavily weighted experts (first choices or those with high gate weights), rather than the sheer number of substitutions. For Qwen3-Next, the 0.15 rule substituted 7.5% of calls (vs. 8.5% when fetching every miss) but reduced perplexity increase from 5.67% to 0.35%. This suggests that protecting high-weight experts is critical, even if many low-weight experts are substituted.
2. Mass as a Comparative Metric: The study proposes the "Mass Hypothesis": among policies that always fetch the first choice, higher substituted gate mass correlates with higher perplexity increases on the same inputs. In the engine, this correlation was strong (Pearson $r \approx 0.92$) across different policies on the same data. However, mass did not predict which specific inputs would lose quality, nor did it hold when comparing policies that substituted first choices.
3. Composition of Substitutions: An experiment (Experiment B) varying the composition of substitutions at equal mass found that a few heavy substitutions caused more loss than many light ones, though the planned statistical decision rule did not yield a definitive judgment due to low realized mass in the primary condition.
4. Background Admission Policy: A rule that admits an expert to the cache in the background once its accumulated substituted mass reaches a threshold of 0.5 reduced perplexity by 0.3–1.5%, with the most significant gains occurring immediately after topic changes.
5. Performance of Selected Policies: Two policies were selected based on these results:
   • Quality-First: 0.15 Rule + Frequency-based eviction + Background admission ($\beta=0.5$).
   • Speed-First: First-choice Rule + Frequency-based eviction + Background admission ($\beta=0.5$).
     On two hosts, these policies were 2.5–3.0× faster per token than fetching every miss, with a perplexity increase of only 0.7–1.5%. Compared to the 0.15 rule alone, the Quality-First policy reduced perplexity further (by ~0.5–0.8%) at a slight time cost, while Speed-First reduced time by ~5% with no statistically significant increase in perplexity.
6. Comparison with Stock llama.cpp: On a machine with ~16 GB of available RAM, the engine using the 0.15 rule decoded 2.8–3.0× faster than stock llama.cpp configured to use the same VRAM, with a perplexity increase of only 1.6%.

Significance and Claims
The paper claims to provide the first direct measurement of which expert substitutions account for quality loss in MoEs with hundreds of experts per layer. It establishes that:

• Substitution Strategy: Protecting high-weight experts (via the 0.15 rule) is more effective than simply reducing the total number of substitutions.
• Metric Utility: Substituted gate mass is a useful metric for comparing policies on the same inputs but is not a predictor of loss across different inputs.
• Practical Efficiency: It is possible to run large MoEs on consumer GPUs with minimal quality degradation (0.7–1.5% perplexity increase) while achieving 2.5–3.0× speedups over standard offloading strategies.

Limitations and Modesty
The author explicitly states that their conclusions are modest and conditional:

• Scope: Results are primarily based on Qwen3-Next-80B-A3B and teacher-forced perplexity. The generation test (GSM8K and HumanEval) supported predictions only partially; for instance, the correlation between mass and agreement on HumanEval failed.
• Causality: The study does not separate the effects of expert rank from weight, nor does it claim that gate weight itself causes the loss.
• Comparisons: The policies were not compared against prefetching or lower-precision methods.
• Generalizability: Speed ratios and specific perplexity values are reported for the specific hardware hosts measured and may not hold on other machines.
• Experimental Status: Apart from the generation test and Experiment B, the analyses are exploratory. The paper reports failed predictions (e.g., regarding problem-level mass correlations) and conclusions that reversed as sample sizes increased (e.g., the correlation between mass and loss).

In summary, the work demonstrates that intelligent substitution and background admission policies can significantly mitigate the latency of offloading large MoEs on consumer hardware, provided that high-weight experts are prioritized for fetching.