🤖 machine learning

Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory

This paper introduces DODOCO to demonstrate that AlltoAll dispatch overhead in Mixture-of-Experts models is driven by intrinsic, architecture-specific routing imbalances rather than expert placement or mock benchmarks, revealing that model design (e.g., MLA/GDN vs. MHA/Mamba) dictates load distribution far more significantly than parallelism scale or synthetic data assumptions.

Original authors: Bole Ma, Jan Eitzinger, Harald Koestler, Gerhard Wellein

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bole Ma, Jan Eitzinger, Harald Koestler, Gerhard Wellein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-speed library where thousands of books (data) need to be sorted and sent to specific experts (specialized computers) who know how to read them. This is how modern AI models called "Mixture of Experts" (MoE) work.

The biggest bottleneck in this library isn't reading the books; it's the mailroom. Every time a book arrives, a router decides which expert gets it. If the router sends 90% of the books to Expert A and only 1% to Expert B, Expert A gets overwhelmed (a "straggler"), and the whole library has to wait for them to finish before moving on.

This paper, DODOCO, investigates three big myths the computer science community has been believing about how to fix this mailroom bottleneck.

The Three Myths Tested

Myth 1: "Just add more mailrooms, and the load will balance itself."

The Old Belief: If you have too many books for one expert, just hire more experts (scale up the system). The theory was that spreading the work across more people would naturally make the load even.
The Reality Check: The researchers tested this by adding more "mailrooms" (ranks) to their system. They found that adding more experts didn't fix the imbalance.
The Analogy: Imagine a popular coffee shop where everyone orders a "Latte." If you open 10 more registers, but everyone still orders a Latte, the barista making Lattes is still the bottleneck. The pattern of what people order (the model's decision) is the problem, not the number of registers. The imbalance is "baked into" the AI's brain, not the hardware.

Myth 2: "We can test our mailrooms using fake, random books."

The Old Belief: Because real books are hard to get, researchers use "mock tokens" (random numbers like "1, 5, 9, 2") to simulate traffic. They assumed these random numbers acted just like real language.
The Reality Check: This is a huge mistake. The researchers found that fake data makes the problem look much worse than it actually is.
The Analogy: It's like testing a traffic light system by throwing random rocks at the intersection. The rocks might cause a massive, chaotic pile-up (high imbalance). But when you switch to real cars driving on a road (real text), the traffic flows much more smoothly because cars follow patterns and rules. The fake data overestimated the traffic jam by up to 2.35 times. Even worse, the fake data suggested that bigger batches of traffic would cause more chaos, but real traffic stays steady regardless of batch size.

Myth 3: "All AI models are the same; we can treat them all the same way."

The Old Belief: Since all these models do similar things, we can design one mailroom system that works for everyone.
The Reality Check: The researchers found that the models split into two distinct teams with totally different behaviors.
- Team "Data-Resilient" (MHA, Mamba-2): These models are like disciplined students. When given real text, they distribute their work almost perfectly evenly. They are easy to manage.
- Team "Persistently Concentrated" (MLA, GDN): These models are like chaotic artists. No matter what text you give them (even real text), they keep dumping 80% of the work on a few specific experts. They are inherently unbalanced.
- The Middle Child (GQA): This one is somewhere in between.

The Big Takeaway

The paper argues that for years, engineers have been trying to fix a problem that doesn't exist (the "batch size scaling" myth) and using the wrong map (fake data) to design their systems.

Instead of trying to force every model to be balanced by adding more hardware, the authors suggest we should categorize models first:

If you have a Data-Resilient model, you can use standard, simple networking because the traffic is naturally balanced.
If you have a Persistently Concentrated model, you need special, complex networking designed specifically to handle the fact that one or two experts will always be swamped.

In short: You can't fix a bad routing decision just by adding more computers. And you can't design a traffic system based on a simulation of random rocks; you have to watch how real cars actually drive. The "shape" of the AI model itself is the most important factor in how much traffic it creates.

Technical Summary: Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory (DODOCO)

Problem Statement
AlltoAll dispatch is the dominant bottleneck in Mixture-of-Experts (MoE) expert parallelism (EP). The interconnect community has proposed four families of mitigations: predictive sample placement, adaptive expert relayout, hierarchical collectives, and EP-aware topology. These solutions rely on two unexamined assumptions:

Correctability (Assumption A): Routing imbalance is a workload property that a sufficiently clever system layer can reduce.
Benchmark Validity (Assumption B): Mock-token benchmarks (using uniform-random IDs) faithfully represent production routing behavior.

The paper argues that these assumptions have not been tested head-on, potentially leading to system designs optimized for phantom workloads.

Methodology
The authors introduce DODOCO, an empirical study designed to test these assumptions using a 5×6 factorial design across five distinct MoE architectures and six data conditions.

Architectures: The study covers five sequence-mixer designs spanning major model families:
- DeepSeek-V2-Lite: Multi-head Latent Attention (MLA).
- DeepSeek-MoE-16B: Standard Multi-Head Attention (MHA).
- Qwen3-30B-A3B: Grouped-Query Attention (GQA).
- Nemotron-30B-A3B: Mamba-2 (State Space Model).
- Qwen3.5-35B-A3B: Gated Delta Networks (GDN).
- Note: The MLA and MHA pair forms a controlled comparison, sharing expert counts, top-k, gating, and load-balancing strategies, differing only in the attention mechanism.
Data Conditions: Six conditions isolate independent properties of token streams:
1. Mock: Uniform-random token IDs (standard benchmark default).
2. Shuffled: Real wikitext tokens with global permutation (preserves frequency, destroys sequence).
3. Remapped: Real sequences with permuted vocabulary IDs (preserves sequence, destroys embeddings).
4. Romansh: Real text in a low-resource Romance language (unseen language structure).
5. Opus: Post-cutoff English reasoning traces (unseen content).
6. Wikitext: English Wikipedia (familiar structure and content).
Hardware & Scale: Experiments run on H100 GPUs (NHR@FAU Helma supercomputer) with NVLink intra-node and InfiniBand inter-node. The study includes a matched-window EP scan (4, 8, 16, 32 ranks) and a global batch size (GBS) sweep (64 to 2048).
Metrics: The study utilizes three scalar summaries of the send-counts matrix ( $S$ ):
- Per-expert max/mean ratio: To measure routing decision skew independent of expert-to-rank mapping.
- Gini Coefficient: To measure per-rank load imbalance.
- Fitted Symmetric Dirichlet $\alpha$ : To resolve differences in near-uniform regimes where Gini compresses values.

Key Findings

1. EP Scaling Does Not Reduce Stragglers (Q1)
Contrary to the assumption that increasing EP degrees smooths load distribution, the study finds that scaling EP does not reduce routing imbalance.

Within each architecture, the per-expert max/mean token ratio remains essentially flat (variations $\le$ 5%) across the EP range (4 to 32).
The straggler is intrinsic to the routing decision the model makes, not to how experts are mapped to ranks. Scaling EP changes which rank holds a specific expert, but not which experts receive tokens.
Therefore, the EP degree is not a lever for improving routing balance; the skew is bounded below by an EP-invariant concentration.

2. Mock Tokens Overestimate Imbalance and Fabricate Trends (Q2)
Mock-token benchmarks significantly misrepresent production routing.

Magnitude of Error: Mock tokens overestimate routing Gini by up to 2.35× compared to real text (Wikitext). On the Nemotron model, the gap in the Dirichlet $\alpha$ metric reaches 47×.
Batch Size Artifact: Mock data exhibits a false trend where imbalance increases with batch size (GBS). Real data shows essentially flat Gini across batch sizes. Practitioners relying on mock data may incorrectly conclude that larger batches worsen imbalance.
Mechanism: The router requires both sequential structure and learned embeddings. Shuffled real text (frequency preserved, sequence destroyed) or remapped text (sequence preserved, embeddings destroyed) performs as poorly as mock tokens.

3. Emergent Architecture Classes (Q3)
The five architectures do not behave interchangeably; they cleave into two stable bands based on how real data affects routing:

Data-Resilient (MHA, Mamba-2): These models achieve low Gini under real text (0.105 and 0.150, respectively) and show significant improvement from mock to real data. Their routing approaches uniformity.
Persistently Concentrated (MLA, GDN): These models remain highly concentrated regardless of data condition, staying above Gini 0.24 even on Wikitext and reaching 0.29–0.38 on mock data.
Intermediate (GQA): Falls between the two extremes, showing moderate improvement but remaining in the mid-band.
Depth Analysis: The "data-resilient" pair starts near-uniformly at the first MoE layer and concentrates with depth (U-shape), whereas the "persistently concentrated" pair is concentrated from the very first layer.

Significance and Claims
The paper claims that the workload model used to design and evaluate MoE-aware interconnects is at best a constant-factor approximation and at worst a phantom.

System Design Implications: The "two-class taxonomy" (data-resilient vs. persistently concentrated) is a more useful input for system design than EP degree or mock-data profiles.
- For data-resilient models, traffic is mostly bandwidth-bound; uniform-load models suffice.
- For persistently concentrated models, locality-aware dispatch and topology designs that exploit concentration on specific inter-node pairs are necessary.
Benchmarking: The authors recommend that MoE communication benchmarks report both mock and real-data conditions to expose the batch-size scaling artifact and architecture-class differentials.
Limitations: The study characterizes routing decisions and AlltoAll statistics but does not measure end-to-end training speedup from system interventions. It does not claim imbalance is unimportant, but rather that the assumptions about its correctability and measurement are flawed.

The paper concludes that routing imbalance is a model-and-data property that the system layer can localize and partially exploit, but cannot make uniform by EP scaling alone. The synthetic benchmarks currently in use miss this reality in two specific ways: overestimating imbalance and fabricating non-existent scaling trends.

The Three Myths Tested

The Big Takeaway

Technical Summary: Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory (DODOCO)

More like this