💻 computer science

Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock

This paper systematically diagnoses five distinct training failure modes of Token-Choice sparse Mixture-of-Experts in video Diffusion Transformers, proposes a "Functional Redundancy Hypothesis" to explain the observed selective deadlock, and offers a complete engineering solution alongside an evolutionary roadmap for scaling these architectures.

Original authors: Haiying Sha

Published 2026-05-20

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Haiying Sha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Trying to Upgrade a Video Generator

Imagine you have a very talented, single-person video editor (a "Dense Model") who can do everything: edit clips, add text, change colors, and follow instructions. They are great, but they are slow and expensive to run.

The researchers wanted to upgrade this editor into a Mixture-of-Experts (MoE) team. Think of this as hiring a team of specialists:

The Manager (Router): Decides which task goes to whom.
The Specialists (Routed Experts): Two clones of the original editor, ready to do specific jobs.
The Intern (Shared Expert): A new hire who learns general knowledge to help everyone.

The goal was to make the video generator smarter and faster by having the Manager send different parts of a video to different Specialists. However, the researchers discovered that this upgrade didn't work smoothly. Instead of a happy team, they found a system that often "froze" or collapsed.

The Three Rules of the Upgrade (The "Three Laws")

Before the experiments, the researchers realized they had to follow three strict rules to even start, or the whole thing would crash:

Don't Change the Uniforms (Structural Consistency): The Specialists must be built exactly like the original editor. If the original used a specific type of math (GELU), the Specialists must use the same. If you try to swap in a different style (like SwiGLU), the weights don't fit, and the model breaks immediately.
Don't Shrink the Signal (1:1 Cloning): When copying the original editor's brain to the Specialists, you must copy it exactly. Do not try to "scale down" the numbers to make them fit better. If you shrink the signal, the video output fades to black because the signal gets lost layer by layer.
The "Micro-Noise" Intern (Shared Expert Initialization): This is the trickiest part. The "Intern" (Shared Expert) starts with almost zero knowledge.
- The Trap: If you start the Intern with perfectly zero weights, the computer's math (specifically bfloat16 precision) is so rough that it rounds the tiny updates to zero. The Intern never wakes up.
- The Fix: You must give the Intern a tiny, almost invisible "spark" of noise (like a tiny static shock) to start. This is enough to wake them up without changing the video output, but it allows them to start learning.

What Went Wrong: The Diagnosis of Failure

The researchers ran the system for 5,000 steps and watched how the "Manager" (Router) assigned tasks. They found a hierarchy of failures:

1. The Linear Router: The "Flat Line" Problem

The Setup: They used a simple, straight-line Manager.
The Result: The Manager got confused. It couldn't tell the difference between tasks. It ended up sending everything to both Specialists equally, but in a way that made them identical.
The Analogy: Imagine a manager who can only draw a straight line on a map. If the terrain is complex (like a video with many tasks), a straight line can't separate the areas. The two Specialists became clones of each other (99% similar), and the system just added extra cost without adding any new skills.

2. The MLP Router: The "Selective Deadlock"

The Setup: They upgraded the Manager to be smarter (non-linear/MLP).
The Result: The global confusion stopped, but a new, sneaky problem appeared called Selective Deadlock.
The Phenomenon: About one-third of the video layers stopped using both Specialists. Instead, they picked one Specialist and ignored the other completely.
The Analogy: Imagine a team of two workers. The Manager realizes, "Hey, Worker A is doing 90% of the work, and Worker B isn't adding much." So, the Manager stops sending work to Worker B. Worker B sits idle. Even if you yell at the Manager (increase the penalty for not balancing), they won't change because the system has convinced itself that one worker is enough.
The Pattern: This didn't happen randomly. It happened in a U-shape:
- Top of the U (Early Layers): The "eyes" of the model (processing raw pixels) got stuck.
- Bottom of the U (Deep Layers): The "brain" of the model (processing complex meaning) got stuck.
- Middle: The middle layers worked fine.

3. The Cross-Attention Router: The "Self-Healing" Attempt

The Setup: They gave the Manager a superpower: the ability to read the text instructions while looking at the video (using Cross-Attention).
The Result: This was the best setup. Some layers that were "dead" actually woke up and started working again!
The Limit: Even with this superpower, about 9 layers remained stubbornly stuck. The Manager still couldn't figure out how to use both Specialists in those specific layers.

The "Functional Redundancy" Theory: Why Did It Happen?

The researchers proposed a theory to explain why the Specialists got stuck. They call it the Functional Redundancy Hypothesis.

The Metaphor: Imagine a "Two Masters + One Apprentice" team.
- The Masters (Routed Experts) are identical clones of the original expert.
- The Apprentice (Shared Expert) starts with almost no skills (micro-noise).
The Process:
1. Start: The Apprentice does nothing. The two Masters are identical. The Manager (Gate) sees no reason to use both Masters, so it picks one and ignores the other. The ignored Master becomes a "strategic reserve" (deadlocked).
2. Growth: The Apprentice slowly learns general skills.
3. The Awakening: Once the Apprentice is good enough to handle the boring, basic tasks, the Manager realizes: "I can give the basic stuff to the Apprentice!" This frees up the "dead" Master to learn something new and different.
The Conclusion: The "dead" layers aren't broken; they are waiting. They are waiting for the Apprentice (Shared Expert) to grow up enough to support them. Until the Apprentice is strong, the system stays in a "deadlock" to save energy.

The "Bfloat16" Trap

The paper also found a hidden technical trap. When training with a specific type of computer math (bfloat16), if a number is very small (like the tiny noise given to the Intern), the computer rounds the updates to zero. It's like trying to measure the growth of a seed with a ruler that only measures in meters. The seed grows, but the ruler says "0."

The Fix: Keep the "master copy" of the weights in high precision (float32) and only use the rougher math for the actual video generation steps.

The Roadmap: Where is This Going?

Based on these findings, the authors propose a three-step plan for the future:

Short Term: Fix the text generation. Currently, the model can't spell words well. They plan to add a specific "Text Expert" to the team that only handles letters and shapes.
Mid Term: Add Sound. They want to add an "Audio Expert" so the model can generate video and sound together, rather than making them separately.
Long Term: Build a "World Model." They want to add experts that understand physics (gravity, collisions) so the AI doesn't just make pretty pictures, but understands how the world actually works.

The Bottom Line

The paper concludes that under the current "Token-Choice" system (where tokens pick experts), deadlock is a structural problem, not a bug. You can't fix it just by tweaking numbers. To fully wake up all the experts, you either need to start with a smarter "Intern" (Shared Expert) or change the whole way the team is organized. This is one of the first detailed studies showing exactly why these video AI upgrades often fail to work as expected.

Technical Summary: Visual MoE Routing Diagnosis

1. Problem Statement

While Mixture-of-Experts (MoE) has successfully scaled language models, its application to video generation using Diffusion Transformers (DiTs) remains nascent and prone to failure. Existing attempts often yield limited benefits due to fundamental differences between visual and language tokens: visual tokens exhibit extreme spatial redundancy, distribution drift induced by diffusion timesteps, and conflicting routing signals in multi-task settings.

This paper addresses the lack of systematic diagnosis regarding the training dynamics of Token-Choice sparse MoE in video DiTs. Specifically, it investigates why routing mechanisms fail, leading to "deadlock" where specific layers or experts cease to function effectively, and seeks to understand the boundary conditions of the current Token-Choice paradigm.

2. Methodology

The research employs a rigorous, minimalist experimental setup designed to eliminate masking effects from pretrained capabilities:

Base Model: A pretrained dense video generation model (Kiwi-Edit, ~8B parameters, combining Wan2.2-TI2V-5B and Qwen2.5-VL-3B) is converted into an MoE architecture.
Conversion Strategy (The "Three Laws"):
1. Structural Consistency: Routed experts must match the original FFN structure exactly (e.g., GELU activation, bias configurations).
2. 1:1 Weight Cloning: Routed experts are initialized by exactly cloning the dense FFN weights without scaling (unlike standard random initialization strategies).
3. Shared Expert Initialization: Shared experts are initialized with all-zero weights for architectural verification (ensuring output equivalence to the dense model) and then re-initialized with extremely small non-zero noise (e.g., $\sigma=10^{-4}$ ) for training to break symmetry and avoid hardware truncation.
Router Architectures: Three router types are compared: Linear (Softmax), Non-linear MLP, and Cross-Attention (leveraging multimodal text instructions).
Training Regimen: A 9-in-1 multi-task video generation dataset (20k samples) is used for 5,000 steps on a single NVIDIA A100. Training uses bfloat16 mixed precision with trainable parameters (gates and shared experts) kept in float32 master copies.
Monitoring: Fine-grained logging of expert utilization (minority expert fraction) across 30 Transformer layers and over 65 million tokens.

3. Key Findings and Results

3.1 Hierarchy of Routing Failures

The study identifies a progression of failure modes:

Global Soft Saturation (Linear Routers): Linear routers suffer from global soft saturation where Softmax gradients vanish, leading to complete expert homogenization (cosine similarity >0.99). The MoE degenerates into a structurally uneven dense network with no capability gain.
Selective Deadlock (MLP Routers): Non-linear MLP routers resolve global saturation but introduce "selective deadlock." Approximately one-third of layers degenerate into a single-expert mode. Increasing the auxiliary loss (up to 0.2) fails to prevent this, indicating a structural system-level issue rather than a simple load-balancing problem.
Self-Recovery and Persistent Deadlock (Cross-Attention Routers): Cross-attention routers exhibit a unique "self-recovery" phenomenon where some deadlocked layers spontaneously rebound. However, roughly 9 layers remain stubbornly deadlocked.

3.2 The "U-Shaped" Deadlock Distribution

Deadlocked layers are not randomly distributed but follow a pronounced U-shaped structure:

Shallow Layers (1–10): Heavily concentrated due to the difficulty of distinguishing spatially redundant low-level visual features.
Deep Layers (18–25): Heavily concentrated due to compressed decision spaces in semantic integration layers.
Intermediate Layers: Maintain near-uniform utilization.

3.3 The bfloat16 Precision Trap

A critical numerical trap was identified: under bfloat16 mixed precision, tiny weight updates for shared experts (initialized near zero) are truncated to zero by hardware because the update magnitude falls below the Unit in the Last Place (ULP).

Solution: The paper documents the necessity of keeping trainable parameter master copies in float32 and initializing shared experts with micro-noise ( $10^{-4}$ ) rather than absolute zero to ensure gradient accumulation.

3.4 The Functional Redundancy Hypothesis

The paper proposes the Functional Redundancy Hypothesis to explain selective deadlock. It posits that deadlock is a rational, phased resource allocation strategy within a "gate, shared expert, routed experts" triad:

Initially, the shared expert (apprentice) has near-zero capability. The system relies on the two routed experts (masters), which are identical due to weight cloning.
The gate naturally selects one master to minimize redundancy, leaving the other as a "strategic reserve" (deadlocked).
As the shared expert gradually learns common knowledge, it creates new functional redundancy. This eventually allows the system to "release" the deadlocked routed expert for specialized division of labor.
The hypothesis suggests that eliminating deadlock requires the shared expert to reach a critical capability threshold before the system can reconfigure.

4. Engineering Contributions

Three Laws of Dense-to-MoE Conversion: A verified protocol for converting pretrained dense models to MoE without architectural mismatch or signal collapse.
Precision Trap Mitigation: A complete solution for the bfloat16 update truncation issue in MoE training.
In-Context Conditioning: Validation of sequence concatenation (vs. addition) for multimodal feature fusion to avoid feature contamination.
Text Rendering Solution: A proposed MoE-based architecture for text generation involving a dedicated "text expert" and a Glyph-ByT5 encoder to solve character-level collapse.

5. Significance and Claims

The paper claims to provide the first comprehensive empirical diagnosis of routing failure in real-world video generation MoE deployments. Its significance lies in:

Calibrating Boundaries: It precisely defines the capability boundary of the Token-Choice paradigm, concluding that under current single-GPU, frozen-expert conditions, complete elimination of dead experts is structurally impossible regardless of router complexity.
Theoretical Perspective: It offers a novel theoretical framework (Functional Redundancy) grounded in systems biology to interpret MoE training dynamics, moving beyond simple load-balancing metrics.
Evolutionary Roadmap: Based on these findings, the authors outline a path from "visual unification" to a "general intelligence engine," suggesting that future progress requires:
1. Improved shared expert initialization (pretrained knowledge).
2. Differentiated expert weight design.
3. Potential migration to Expert-Choice or dynamic routing paradigms.
4. Integration of physical priors and causal reasoning for world modeling.

The work emphasizes that while current Token-Choice MoE faces structural deadlock in visual domains, understanding these failure modes provides the necessary foundation for the next generation of multimodal generative models.