💻 computer science

Predatory MoE: Serving Mixture-of-Experts with a Sequence-Local Expert Roster

This paper proposes a sequence-local expert roster strategy that fixes a small set of experts per sequence to drastically reduce memory footprint and expert transfers during MoE model serving, achieving near-full-residence throughput with significantly fewer resident weights by applying this constraint consistently during both training and inference.

Original authors: ChaeWoo Son

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: ChaeWoo Son

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems that generate text often rely on a design called a mixture of experts. Imagine a vast library of specialized knowledge workers, where each worker is an expert in a specific, narrow topic. When the system needs to answer a question, it does not activate every single worker at once. Instead, it selects only a tiny handful of the most relevant experts for that specific moment. This approach allows the system to be incredibly large and capable while using relatively little computing power for each word it produces. However, there is a significant bottleneck in how these systems are run on standard computers. Even though only a few experts are used at any given moment, the computer's memory must hold the entire library of experts ready to go, because the system does not know in advance which ones will be needed next. This requirement forces the total memory size to match the size of the entire library, rather than just the small group being used, making it difficult to run these powerful models on devices with limited memory.

Researchers have tried to solve this by storing the unused experts on a hard drive and loading them into memory only when needed. But this creates a new problem: because the system picks experts in a scattered, unpredictable way, it ends up fetching new experts constantly, slowing down the process. A team led by independent researcher ChaeWoo Son asked a different question: instead of trying to make the computer faster at fetching scattered experts, could they change the system so that it naturally sticks to a small, consistent group of experts for the duration of a single conversation? They wanted to see if they could train the system to become "sequence-local," meaning that once a conversation starts, the same small team of experts handles the entire thread, allowing the computer to keep just that team in memory.

The researchers first tried to teach the system this behavior directly. They attempted to train the model to favor experts it had already used in the current conversation, hoping this habit would become a permanent part of the system's intelligence. They found that while this trick worked while the training was happening, the behavior did not stick. As soon as the training pressure was removed, the system reverted to its old, scattered habits. Even when they tried to reinforce the behavior by having the system learn from its own biased choices, the process became unstable and the quality of the text suffered. The study concluded that the system was not learning a new skill, but was simply reacting to a temporary rule. The researchers realized that trying to force the system to internalize this behavior was the wrong approach.

Instead of trying to change the system's brain, the researchers decided to change the rules of the game. They imposed a strict, permanent rule that applied both during training and when the system was actually being used. Under this new rule, every conversation is assigned a small, fixed team of experts based on the first few words of the text. Once this team is selected, the system is not allowed to switch to anyone else for the rest of that conversation. This rule acts like a budget: the computer only needs to keep that specific small team in its fast memory, while the rest of the library can stay on the slower storage drive. Because the team is fixed for the whole conversation, the computer does not need to constantly swap experts in and out.

The results of this approach were striking. When the researchers applied this rule to a test model and gave it a short period of retraining to adapt to the new constraints, the system performed almost as well as it did without the rule, but with a massive reduction in memory usage. In their tests, they were able to keep only one-quarter of the total experts in memory and still achieve nearly the same speed as if they had kept all of them. The system reduced the number of times it had to fetch data from the slow storage drive by a factor of thirty-two compared to the old method. While the text quality did drop slightly—by about five percent in the most extreme memory-saving scenario—the trade-off allowed the model to run on much smaller, more accessible hardware.

The study also revealed an unexpected insight about how these systems learn. The researchers found that the longer a model is trained without these memory constraints, the harder it becomes to force it into this efficient pattern later. The system's natural tendency to scatter its attention seems to grow stronger over time, making it more expensive to adapt to memory limits after the fact. This suggests that if we want these models to be efficient on everyday devices, we might need to teach them to be efficient from the very beginning, rather than trying to fix the problem after they have already learned to be scattered.

This work does not claim to have solved the problem for every possible situation, and the researchers were careful to note that their tests were done on synthetic data and smaller models. They observed that if a conversation changes topics abruptly in the middle, the fixed team of experts might struggle, though their tests showed that even with a topic switch, the system could adapt with a small penalty in quality. The study serves as a proof of concept that by treating the memory constraint as a rule rather than a learning goal, we can make these massive models much more practical to run. It shifts the focus from building faster hardware to designing smarter rules for how the system uses its resources, opening the door for powerful artificial intelligence to run on devices that are currently too small to hold them.

Technical Summary: Predatory MoE: Serving Mixture-of-Experts with a Sequence-Local Expert Roster

Problem Statement

Open-weight Mixture-of-Experts (MoE) models achieve high capability by activating only a small fraction of their total parameters per token. However, in memory-constrained, single-accelerator settings, serving these models typically requires keeping all experts resident in memory, meaning the memory footprint scales with the total parameter count rather than the active count.

Standard solutions involve offloading inactive experts to host memory or disk and fetching them on demand. Existing systems adapt to the routing demand as it occurs (e.g., using LRU caches or predictive prefetching). However, unconstrained routing is often "hostile" to bounded caches: token-level routers may select any expert at any token, causing frequent weight transfers. In the author's 32-expert testbed, an unconstrained model accessed an average of 28 out of 32 experts per 256-token sequence. The paper asks whether the routing demand pattern itself can be changed to be sequence-local, such that each sequence is served by a small, fixed roster of experts, allowing resident memory to follow a roster budget rather than the total expert count.

Methodology

The paper investigates two primary approaches to achieving sequence-locality: training it into the model weights versus imposing it as a routing rule.

1. Attempting to Train Sequence-Locality

The author first explored whether sequence-locality could be instilled as a training objective using a "rich-get-richer" selection bias (an amplifier bias). This bias favored experts already used within a sequence for the remainder of that sequence.

Findings: While the bias reduced the working set size while active, the resulting locality was not persistent.
- Timing Sensitivity: The bias only reshaped routing durably when applied from the start of training; applying it late yielded only immediate selection effects without further adaptation.
- Reversion: If the bias was removed mid-training, continued unbiased training caused the routing to revert to a scattered pattern, restoring the original working set size.
- Distillation Failure: Attempts to internalize the bias via self-distillation (training the router to imitate its own biased selections) resulted in a moving target that destabilized routing and degraded model quality.
Conclusion: Sequence-locality behaved as a property of the routing procedure, not the learned weights.

2. The Roster Routing Rule

Based on the failure of training-based approaches, the author imposed sequence-locality as a routing rule applied identically during training and serving.

Mechanism:
- Each sequence maintains a running count of expert selections per layer.
- A roster $R$ is formed by the top- $m$ most-used experts.
- Exploration Phase: Until $m$ distinct experts are used, no bias is applied.
- Barrier Phase: Once the roster is full, members receive a selection bonus $+\delta$ , and non-members receive $-\delta$ . Non-members must overcome a $2\delta$ score deficit to enter the roster, displacing the least-used member.
- Pinning (Serving): At deployment, the roster is frozen after a short prefix (e.g., 20% of the sequence). Subsequent decoding is restricted to this pinned roster.
Retrofitting: For pretrained models, the author proposes a 600-step fine-tuning phase under the deployed rule (with pinning active). This allows the experts to adapt to the traffic patterns created by the rule without requiring the router to learn locality itself.

Key Contributions

Empirical Analysis of Training-Based Locality: A controlled analysis demonstrating that sequence-locality resists being trained as a persistent objective due to timing sensitivity, reversion under continued training, and the self-referential instability of router distillation.
Roster-Budget Routing Rule: A mechanism that treats locality as a procedural constraint with an explicit equilibrium (the budget $m$ ), rather than an emergent behavior. This rule maintains stable training dynamics and keeps per-sequence working sets close to the budget $m$ .
Retrofit Recipe: A method for adapting pretrained models to the roster rule via a brief fine-tune (600 steps). This process matches the performance of models trained from scratch under the rule, effectively removing the quality cost of attaching the rule to a pretrained model.
Serving Analysis: A trace-driven simulation showing that the resulting demand pattern drastically reduces expert transfers compared to standard caching strategies under matched memory budgets.

Results

Quality and Budget Trade-offs

From-Scratch Training: Training under the roster rule with $m=4$ (out of 8 experts) resulted in a working set of $\approx 4.18$ with only a $+0.27\%$ perplexity increase over the baseline.
Retrofitting: Applying the rule to a pretrained model without fine-tuning caused a large perplexity jump ( $+32\%$ ). However, a 600-step fine-tune under the pinned rule reduced this cost to $+0.9\%$ (at $m=4$ ) and $+5.0\%$ (at $m=8$ for a 32-expert model).
Scale: In the 32-expert testbed, fine-tuning reduced the cost of pinning $m=8$ (75% memory reduction) to $+5.0\%$ perplexity, compared to a $\sim 9\times$ perplexity multiplier for zero-shot attachment.
Erosion of Serveability: The author observed that unconstrained training gradually erodes a model's suitability for roster serving. Even after perplexity converges, continued training increases the cost of serving under a pinned roster, suggesting serving constraints should be exposed during training.

Serving Performance (Trace-Driven Simulation)

Under matched memory budgets (25% of expert weights resident):

Transfer Reduction: The roster-served model reduced expert transfers by up to 32 $\times$ compared to an LRU cache serving an unconstrained model.
Throughput: The roster-served model recovered 94% of the full-residence decode throughput, whereas the LRU-served unconstrained model achieved only 33%.
Latency: The roster approach reduced stall time to near zero (0.01 ms/token) compared to 30.3 ms/token for the LRU baseline.

Significance and Claims

The paper claims that per-sequence expert locality is more naturally handled as part of the routing procedure (specified, budgeted, and present during training) than as a behavior a model is asked to learn.

Practical Impact: The proposed method allows serving large MoE models with a fraction of the expert memory resident (e.g., 25%) while recovering nearly full-residence throughput, at the cost of a modest perplexity increase (e.g., +5.0%).
Scope and Limitations: The author is explicit that their quantitative results come from a synthetic, single-domain testbed (up to 32 experts). They do not claim the results generalize directly to production-scale models or multi-stream batched serving without further validation.
Future Work: The paper outlines a path for validation, including analyzing routing traces of existing open-weight MoEs and fine-tuning small open MoEs on real data to test the retrofit recipe and the "erosion" hypothesis.
Extrapolation: While the author provides arithmetic estimates suggesting that for a 2.4T parameter model, roster serving could reduce the footprint to a workstation scale (e.g., ~97 GB at $m=32$ ), they emphasize these are order-of-magnitude estimates based on unverified assumptions regarding real-world routing patterns and architecture specifics.

In summary, the paper proposes a shift from engineering systems to serve hostile routing demands to engineering the routing demand itself to be servable, demonstrating that a procedural "roster rule" combined with brief fine-tuning can achieve significant memory and throughput gains in controlled settings.