💬 NLP

The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism

This technical survey synthesizes the evolution of Mixture-of-Experts architectures in Large Language Models by organizing them into a dependency graph and analyzing them through four control planes (Expert Topology, Routing, Balance, and Expert Parallelism) to illustrate the field's shift from simply activating sparse parameters toward decoupling semantic routing, computational budgets, and physical execution.

Original authors: Jiguo Li

Published 2026-08-11

📖 10 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiguo Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

=== SUMMARY ===
Imagine the brain of a super-intelligent computer as a massive, bustling library. In the old days, to answer a single question, the computer would send a librarian to read every single book on the shelves, one by one, just to be safe. This was slow and wasted a lot of energy. Then, engineers invented a smarter way: a "Mixture of Experts." Instead of one librarian reading everything, they hired thousands of specialists. When a question about cooking comes in, only the "Chef" librarian opens their book; when a question about space comes in, only the "Astronomer" wakes up. This is the core idea of Mixture-of-Experts (MoE) in Large Language Models (LLMs): it lets AI models grow incredibly huge and smart without making every single calculation for every single word they process.

But here's the tricky part: if you have a million specialists, how do you decide who gets the work? If you send all the "Space" questions to the Astronomer, they get overwhelmed and slow everything down, while the "Chef" sits idle. This is the "routing" problem. It's like trying to manage a giant concert where you have to instantly decide which musician plays which note, ensuring no one is overwhelmed, no one is bored, and the music flows perfectly. If you mess up the routing, the whole system grinds to a halt.

This paper, written by Jiguo Li in August 2026, is a deep dive into how these "expert" systems have evolved over time. It argues that the history of MoE isn't just a list of newer, bigger models released year by year. Instead, it's a story of solving one specific traffic jam after another. The author suggests that the field has moved through eight major "milestones," shifting from simple statistical tricks to complex, dynamic systems where the computer can decide not just who helps, but how much help is needed, and even where that help physically lives in the computer's hardware. The paper finds that the future isn't just about adding more experts, but about untangling the logic of "what the AI thinks" from the physical reality of "how the computer runs," allowing for smarter, faster, and more efficient AI that doesn't get stuck in traffic.

The Story of the Eight Milestones

Think of the evolution of MoE like the history of a city's public transport system. It started with a single bus that stopped everywhere, then moved to a system where you could call a taxi, and eventually became a hyper-efficient, self-organizing network of drones and subways. The paper breaks this journey into eight distinct phases, or "milestones," where each step solved a major bottleneck but created a new challenge.

1. The Statistical Division of Labor (The Old Bus)
Back in the early days, the idea was simple: have a "gate" that decides which "expert" (a small part of the brain) should handle a task. But in the beginning, everyone still helped a little bit. It was like having a bus where every passenger had to stand up and wave to the driver, even if they weren't getting off. It was a statistical division of labor, but it didn't save any energy. The computer still did all the work; it just did it in a slightly smarter way.

2. The "Top-K" Switch (The Taxi Service)
Then came the big breakthrough: Sparse Conditional Computation. Imagine a switch that says, "Only the top 2 experts get to work; the other 98 can go home." This is the Top-K routing. Suddenly, the computer could have a massive library of knowledge (millions of parameters) but only use a tiny slice of it for each word. This was the "capacity lever": you could make the model smarter without making it slower. But, it introduced a new problem: what if the "Chef" gets 1,000 orders and the "Astronomer" gets none? The system started to get unbalanced.

3. The Cluster Scale-Up (The Subway Network)
As models grew to fit on thousands of computer chips (GPUs), the problem became moving the data. If the "Chef" lives on Chip A and the "Astronomer" on Chip B, how do you get the ingredients there? This era introduced Expert Parallelism and All-to-All communication. It's like building a massive subway network where tokens (the words) are the passengers, and they have to hop on trains to get to the right expert. The challenge shifted from "who does the math" to "how fast can we move the data without the train getting stuck."

4. The Open-Weight Era (The Public Transit App)
Then, models like Mixtral showed that this complex system could actually work for regular people, not just giant labs. They proved you could have a huge, coarse-grained MoE (big experts) that was open for everyone to use. But, the "experts" were still too big and clumsy. They tended to relearn the same basic things (like how to say "hello"), which was a waste of space.

5. Fine-Grained & Shared Experts (The Specialized Kiosks)
To fix the clumsiness, engineers started splitting the big experts into many tiny, fine-grained ones. It's like replacing one giant "General Store" with hundreds of tiny kiosks: one for "spicy food," one for "desserts," one for "vegan options." This allowed for much more precise routing. They also added Shared Experts—a permanent path for things everyone needs (like basic grammar)—so the routing system didn't have to waste time deciding on the basics. This made the system faster and smarter, but it created a new traffic jam: too many tiny kiosks meant too many tiny trips for the subway, slowing things down.

6. Ultra-Sparse Scaling (The Million-Expert Library)
Next, researchers asked: "What if we have way more experts, but even smaller ones?" Models like Kimi K2 and PEER started using hundreds of tiny experts (sometimes over a million!). The idea was to have a candidate pool so huge that the AI could find the perfect specialist for every single word. But, the paper notes a catch: if you have too many experts, the computer spends more time looking up who to call (retrieval) and moving data than actually doing the work. The "weight I/O" (loading the expert's brain) became the new bottleneck.

7. Dynamic Compute (The Flexible Budget)
Until now, every word got the same amount of help (e.g., "Top-2" experts). But some words are hard (like "quantum physics") and some are easy (like "the"). Dynamic Compute changes the rules: the AI can decide to use 1 expert for easy words and 4 for hard ones. Some models even use "zero-compute" slots, where the AI just says "I know this, no need to call anyone." This saves energy, but it creates a new risk: if a bunch of hard words show up at the same time, the system might get overwhelmed, causing delays (tail latency).

8. Decoupling Logic from Physics (The Teleportation Network)
The final frontier is Semantic Routing Decoupled from Physical Execution. Currently, if the AI decides to call the "Astronomer," the data must physically travel to the Astronomer's chip immediately. The new ideas (like ScMoE or Heterogeneous Experts) suggest we can separate the decision from the movement. Maybe the AI decides to use the Astronomer, but the data waits in a buffer, or the Astronomer is a different size depending on the task. It's like having a teleportation network where the destination is decided logically, but the physical transport is optimized separately to avoid traffic jams.

The Four Control Planes

The paper organizes all these changes into four "control planes," which are like the different layers of management in a giant company:

The Object Layer (Topology): This is the "Org Chart." It defines who the experts are, how big they are, and if they share a desk. It asks: "Do we have 8 big experts or 128 tiny ones?"
The Decision Layer (Routing): This is the "Dispatcher." For every single word, it decides: "Who do I call?" It uses a "Top-K" rule to pick the best specialists.
The Control Layer (Balance): This is the "HR Manager." It watches to make sure no expert is working 24/7 while others are sleeping. If the "Chef" is overloaded, it might gently nudge the system to send some orders to the "Baker" instead, using special math tricks (like Auxiliary-Loss-Free balancing) to do this without messing up the AI's learning.
The Execution Layer (Expert Parallelism): This is the "Logistics Team." It figures out how to actually move the data across the physical chips, handle the traffic, and make sure the math gets done fast.

What the Paper Rules Out and What It Suggests

The author is very clear about what doesn't work as a magic bullet. They argue against the idea that there is a single "perfect" structure for all MoE models. They suggest that simply adding more experts isn't the answer if the system can't move the data fast enough. They also point out that Soft MoE (where experts blend together continuously rather than picking one) hasn't been proven to work well on the massive, real-world systems we use today, even though it sounds nice in theory.

The paper suggests that the future lies in decoupling. We need to stop tying the "smart decision" (which expert to use) to the "physical reality" (which chip it lives on). They propose that the best systems will use dynamic budgets (changing how much work is done based on difficulty) and runtime placement (moving experts around while the computer is running to avoid traffic jams).

However, the paper is careful not to claim these are "solved" problems. For instance, while Heterogeneous Experts (experts of different sizes) sound great, the paper notes they are still in the "research frontier" stage and haven't been fully tested at the massive scale of trillion-parameter models. Similarly, Cross-layer sharing (where an expert from layer 1 helps layer 100) is a cool idea, but the author says we need more evidence to see if it actually works without causing confusion.

The Bottom Line

In simple terms, this paper tells us that the journey of making AI smarter isn't just about building a bigger brain. It's about building a better traffic system. We've moved from a single bus to a complex network of taxis, subways, and drones. The next step isn't just adding more vehicles; it's about making the traffic lights smarter, the roads more flexible, and the dispatchers able to handle rush hour without crashing. The author concludes that the "next generation" of AI won't be a single new model, but a combination of these smart routing tricks, dynamic budgets, and physical optimizations working together. It's a reminder that in the world of super-intelligent computers, sometimes the hardest part isn't the thinking—it's the getting there.

Technical Summary: The Evolution of Mixture-of-Experts Architectures in Large Language Models

Problem Definition and Analysis Scope

The paper addresses the challenge of understanding the architectural evolution of Mixture-of-Experts (MoE) in Large Language Models (LLMs). While existing surveys often organize MoE research by chronological model releases or full-stack taxonomies, this report argues that such approaches fail to explain why architectural changes occur. The core problem is identifying the specific bottlenecks that each architectural iteration removes and where the bottleneck subsequently migrates.

The scope is strictly limited to sparse MoE in decoder-only LLM pre-training, focusing on Feed-Forward Network (FFN) MoE. It explicitly excludes multimodal MoE, MoE-LoRA, external ensembles, and pure post-training fusion. The report posits that MoE is not a single-objective optimization but a simultaneous struggle to maximize total parameter capacity ( $P_{total}$ ) while controlling per-token active parameters ( $P_{active}$ ), maintaining semantic specialization, and ensuring system efficiency (avoiding stragglers and communication bottlenecks).

Methodology: A Four-Control-Plane Framework

Instead of a linear timeline, the author proposes a structural framework organized around four coupled control planes that define the internal operation of any MoE system:

Expert Topology (Object Layer): Defines the logical expert set ( $\mathcal{E}$ ), including granularity, sharing relations, and heterogeneity. This is a slow-changing variable determined during architecture design.
Routing (Decision Layer): Determines the subset of experts ( $\mathcal{K}_t$ ) selected for each token based on affinity scores. This operates at the token level and is a fast variable.
Balance (Control Layer): Aggregates load statistics ( $n_i$ ) over sequences or batches and applies feedback (via auxiliary losses, biases, or capacity constraints) to prevent collapse and ensure physical load distribution.
Expert Parallelism (Execution Layer): Maps logical expert selections to physical devices, handling dispatch, All-to-All communication, and kernel execution.

The author formalizes these planes as a closed-loop system where physical constraints (e.g., device topology) can retroactively constrain semantic routing, and runtime placement can alleviate load without changing the router's output.

Key Contributions: Eight Evolutionary Milestones

The paper identifies eight milestones not as successive generations, but as a dependency graph where nodes represent the migration of the dominant system bottleneck.

Mainline Developments (Nodes 1–6)

Node 1: Dense/Soft MoE (Statistical Division of Labor): Established the gating mechanism and weighted combination but coupled capacity with FLOPs (all experts compute).
Node 2: Sparse Conditional Computation (Top-k): Introduced by Shazeer et al., this decoupled total capacity from per-token FLOPs by activating only $k \ll N$ experts. This introduced the need for discrete routing, load balancing, and token dropping.
Node 3: Transformer MoE & Expert Parallelism: Systematized the execution chain (Router $\to$ Dispatch $\to$ All-to-All $\to$ Expert $\to$ Combine) and introduced stability mechanisms (e.g., z-loss, capacity factors) to enable training on thousands of devices.
Node 4: Open-Weight Coarse-Grained MoE: Demonstrated by Mixtral, this proved that standard coarse-grained MoE could support the full lifecycle of open-weight decoder models, shifting the bottleneck to knowledge redundancy and coarse composition.
Node 5: Shared + Fine-Grained Experts: Introduced by DeepSeekMoE/V2, this split large experts into smaller units to increase composition resolution and added "shared experts" to isolate common knowledge, reducing redundancy among routed experts.
Node 6: Ultra-Sparse Scaling: Explored by PEER and Kimi K2, this investigates whether increasing the total expert count ( $N$ ) further improves loss when active parameters are fixed, moving the bottleneck to weight I/O and retrieval overhead.

Orthogonal Branches (Nodes 7–8)

These nodes relax fundamental assumptions of the mainline rather than simply adding more experts.

Node 7: Dynamic Compute: Moves from fixed Top-k to token-dependent budgets (e.g., zero-compute experts, adaptive- $k$ ), decoupling semantic selection from computational budget.
Node 8: Semantic-Physical Decoupling: Decouples the semantic choice of an expert from its immediate physical execution path. This includes:
- ScMoE: Rearranging execution dependencies to overlap communication with dense paths.
- Heterogeneous Experts: Selecting experts of different sizes based on token difficulty.
- Cross-layer Sharing: Reusing experts across layers to reduce redundancy.
- Structured Communication: Reducing dynamic All-to-All traffic to deterministic patterns (e.g., Head Parallel).

Results and Experimental Design

The paper does not present new empirical training results but synthesizes findings from prior reports (DeepSeek-V3, Qwen3, Kimi K2, etc.) to validate the framework. It highlights that:

Balance Evolution: The field is moving from sequence-wise auxiliary losses to Auxiliary-Loss-Free (ALF) strategies (using non-gradient biases) and runtime placement to separate physical load balancing from gradient updates.
Communication Optimization: The focus has shifted from minimizing total All-to-All volume to minimizing exposed communication time via overlap (ScMoE, DualPipe) and topology-limited routing.
Expert Granularity: The trend is moving from isomorphic large experts to fine-grained, heterogeneous, and ultra-sparse configurations.

The author proposes a rigorous Experimental Design for Foundation-Model Pretraining to replace cross-model benchmark comparisons. They advocate for "equal-budget" ablation studies where training tokens, active parameters, and optimizer settings are fixed while varying:

Expert granularity and shared path presence.
Balance scope (sequence vs. global batch vs. ALF).
Execution topology (serial EP vs. ScMoE).
Dynamic compute mechanisms.

Significance and Claims

The paper claims its primary significance lies in shifting the perspective of MoE evolution from a "chronological list of model releases" to a structural analysis of bottleneck migration.

Complementary Views: The eight milestones describe historical bottleneck migration, while the four control planes provide a structural view of a single system. These are presented as complementary, not competing, taxonomies.
Decoupling: The main trend identified is the decoupling of semantic routing, computational budgets, and physical execution. Future architectures are not defined by a single "next-generation" structure but by the combination of freer routing, explicit budget control, topology awareness, and runtime placement.
Evaluation Shift: The paper argues that evaluating MoE must expand beyond validation loss to include joint analysis of specialization, active budget variance, exposed communication, and tail latency (P99).

The report concludes that there is no single optimal structure; rather, the "next generation" is a flexible combination of the identified architectural axes tailored to specific hardware constraints and training objectives.