💻 computer science

SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs

The paper proposes SMoES, a novel routing strategy for Mixture-of-Experts Vision-Language Models that utilizes dynamic soft modality scores and mutual information regularization to align expert specialization with layer-dependent fusion patterns, thereby significantly improving both task performance and deployment efficiency.

Original authors: Zi-Hao Bo, Yaqian Li, Anzhou Hou, Rinyoichi Takezoe, Ertao Zhao, Tianxiang Pan, Jiale Yan, Mo Guang, Kaiwen Long

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zi-Hao Bo, Yaqian Li, Anzhou Hou, Rinyoichi Takezoe, Ertao Zhao, Tianxiang Pan, Jiale Yan, Mo Guang, Kaiwen Long

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, super-smart team of specialists working together to solve complex puzzles. Some team members are great at looking at pictures, others are wizards at reading text, and some are good at both. This team is called a Mixture-of-Experts (MoE) model, and it's the engine behind modern AI that can see and read simultaneously (Vision-Language Models).

The problem the paper tackles is like this: How do you tell the right specialist to work on the right part of the puzzle without causing chaos?

The Problem: The "Hard" vs. "Soft" Routing Dilemma

In the past, there were two main ways to manage this team:

The "Hard" Rule: You strictly assign specific people to pictures and others to text.
- The Flaw: It's too rigid. Sometimes a picture needs a word to make sense, or a word needs a picture. If you force a strict separation, the team misses out on these connections, and the AI gets confused.
The "Soft" Rule: Everyone can work on anything. The manager (the router) just guesses who is free.
- The Flaw: It's too messy. The team ends up with everyone doing everything, which wastes energy. Also, in real-world computing, if you send picture data to one computer and text data to another, they have to talk to each other constantly, slowing everything down.

The Solution: SMoES (The "Smart Team Manager")

The authors propose a new system called SMoES (Soft Modality-Guided Expert Specialization). Think of it as a dynamic, intelligent team manager that learns how to organize the specialists on the fly.

Here are the three main tricks SMoES uses:

1. The "Soft Score" (Not Just Black and White)

Instead of labeling a token (a piece of data) as strictly "Picture" or "Text," SMoES gives it a soft score.

The Analogy: Imagine a token is a person walking into a room. A "Hard" rule says, "You are 100% a Picture person, go to the Picture room."
SMoES says: "You look 70% like a Picture person and 30% like a Text person right now."
Why it matters: As the AI processes information through its layers (like reading a book page by page), the meaning changes. A picture token might start as just "image" but later become "image describing a story." SMoES tracks this changing identity dynamically, ensuring the right expert handles it at the right moment.

2. The "Expert Binning" (Grouping for Efficiency)

In big AI systems, the "experts" (the sub-networks) are often spread across different computers to handle the workload.

The Problem: If the manager sends random data to random computers, they have to shout back and forth constantly to share information. This "shouting" (communication) is slow and expensive.
The SMoES Fix: It groups experts into "bins" (teams) based on what they are good at.
- The Analogy: Imagine you have a warehouse with 100 workers. Instead of scattering them randomly, you put all the "Picture Experts" in Warehouse A and all the "Text Experts" in Warehouse B.
- Now, when a picture comes in, it stays in Warehouse A. The workers there do the work without needing to call Warehouse B. This drastically cuts down on the "shouting" (communication overhead) between computers.

3. The "Mutual Information" Coach (Teaching Specialization)

How does the manager know which expert belongs in which bin? It uses a mathematical "coach" called Mutual Information.

The Analogy: The coach watches the team and says, "Hey, if I see a picture, I want to know exactly which expert is handling it. If I see text, I want to know exactly which text expert is handling it."
The system rewards the team when they become very clear about who does what. It forces the "Picture Bin" to get really good at pictures and the "Text Bin" to get really good at text, without forcing them to be rigid.

The Results: Faster and Smarter

The paper tested this on four different AI models and 16 different tests (ranging from answering math questions to describing images).

Smarter: The AI got better at both looking at images and understanding text. It improved by about 0.9% on image tasks and 4.2% on text-only tasks compared to the old "soft" methods.
Faster: Because the "bins" kept similar data together, the computers didn't have to talk to each other as much. This cut communication costs by 56% and made the system 12% faster in real-world scenarios.

Summary

The paper argues that you don't need to force a strict "Picture vs. Text" rule, nor do you need to let everything mix chaotically. Instead, by using soft scores to track how data changes and grouping experts intelligently, you can build an AI that is both smarter at understanding the world and much faster at doing it. It's like turning a chaotic open-plan office into a well-organized factory where the right tools are always in the right hands.

1. Problem Statement

Mixture-of-Experts (MoE) architectures have become the standard backbone for Large Vision-Language Models (VLMs) to scale capacity efficiently. However, existing routing strategies in MoE-VLMs suffer from fundamental limitations regarding modality specialization:

Hard Routing: Pre-assigns experts to specific modalities (e.g., vision-only or text-only). While this creates specialization, it imposes rigid boundaries that fail to capture the natural, layer-dependent blending of modalities, leading to performance degradation in cross-modal tasks.
Soft Routing: Allows experts to process any token dynamically. However, without explicit guidance, this often results in "over-mixing" where experts fail to specialize, or relies on heuristic priors that ignore the evolving distribution of modalities across network layers.
Efficiency Bottleneck: In Expert-Parallel (EP) deployments, modality-agnostic routing scatters tokens across devices, causing high inter-device communication overhead. Vision tokens often dominate sequence length but have lower information density, while text tokens are dense but fewer. Standard load balancing often misallocates resources, hindering both performance and inference efficiency.

The core challenge is: How to guide expert routing using modality-specific signals to achieve dynamic, layer-adaptive specialization without sacrificing flexibility or efficiency?

2. Methodology: Soft Modality-Guided Expert Specialization (SMoES)

The authors propose SMoES, a framework that aligns expert routing with the dynamic fusion patterns of modalities through three key components:

A. Dynamic Soft Modality Scores

Instead of using static binary labels (hard routing), SMoES computes continuous soft modality scores ( $M \in [0, 1]$ ) for each token at every layer. These scores capture the evolving state of a token as it transitions from modality-specific to cross-modal representations. Two complementary estimators are used:

Attention-Accumulated Score: Propagates modality signals through the attention mechanism. It initializes with hard labels at the input layer and updates scores in subsequent layers by aggregating attended tokens' scores weighted by attention weights. This captures local, sequence-specific interactions.
Gaussian-Statistics Score: A global estimator that maintains online Gaussian distributions (mean and variance) for text and vision tokens via Exponential Moving Average (EMA). It computes the log-likelihood of a token under each distribution to infer its modality affiliation, capturing global statistical regularities independent of layer initialization.

B. Expert Binning

To address deployment efficiency, SMoES partitions the pool of experts into bins (groups).

Adaptive Binning: Experts are dynamically sorted and grouped based on their historical load from specific modalities (tracked via EMA). Experts with similar modality preferences are placed in the same bin.
Deployment Alignment: These bins serve as the unit for device placement in EP deployments. By co-locating bins with similar modality affinities on the same device, the method minimizes cross-device token transfer.

C. Inter-Bin Mutual Information (MI) Regularization

To drive effective specialization, SMoES introduces a loss function that maximizes the Mutual Information between the soft modality scores ( $M$ ) and the expert bins ( $B$ ).

Objective: $\mathcal{L}_{MI} = -\sum I(M; B)$ .
Mechanism: This encourages different bins to specialize in distinct modality patterns. If a bin is selected, it should provide high information about whether the token is visual or textual. This is optimized at the bin level to align with device granularity, avoiding the instability of optimizing at the individual expert level.

Training Objective: The total loss combines the task loss (language modeling), a bin-level load-balancing loss, and the inter-bin MI regularization.

3. Key Contributions

Dynamic Soft Modality Scores: Introduced attention-accumulated and Gaussian-statistics estimators that capture layer-dependent fusion patterns, moving beyond rigid hard routing or unguided soft routing.
Expert Binning & MI Regularization: Proposed a structural mechanism (binning) coupled with an MI-driven objective that promotes coherent modality grouping, enhancing both model effectiveness and deployment efficiency.
Comprehensive Validation: Demonstrated state-of-the-art performance across four diverse MoE backbones (DeepSeekMoE, OLMoE, Moonlight-MoE, Qwen3-MoE) and 16 benchmarks, proving that data-driven specialization unlocks capacity without sacrificing load balance.

4. Experimental Results

The authors evaluated SMoES on 16 benchmarks (10 multimodal, 6 language-only) across four VLM architectures.

Effectiveness (Accuracy):
- Multimodal Tasks: Achieved an average gain of 0.9% over soft routing baselines.
- Language-Only Tasks: Achieved a significant average gain of 4.2%, demonstrating that SMoES preserves (and enhances) language capabilities, unlike hard routing which often degrades them.
- Comparison: Outperformed Hard Routing, Hybrid Routing (MoIIE), and KL-divergence based methods (SMAR). Notably, while SMAR improved specialization, it often hurt language performance due to load-balancing conflicts; SMoES resolved this.
Efficiency (Deployment):
- Communication Overhead: In a realistic 2-GPU edge deployment (NVIDIA Orin), SMoES reduced EP communication overhead by 56.1% for vision tokens during the prefill stage.
- Throughput: Achieved a 12.3% improvement in throughput (tokens per second) under realistic deployment conditions due to reduced communication and better parallelism.
- Latency: Reduced Time to First Token (TTFT) by up to 22% and Time Per Output Token (TPOT) by 10-11% depending on batch size.
Ablation Studies:
- Confirmed that soft scores (attention/Gaussian) significantly outperform hard binary scores.
- Showed that MI regularization is superior to KL-divergence for guiding specialization without collapsing expert capacity.
- Validated that adaptive binning (based on runtime statistics) is superior to fixed binning strategies.

5. Significance

This work addresses a critical gap in the scaling of MoE-VLMs. It demonstrates that modality specialization should not be a static, hand-crafted constraint but a dynamic, learned property that evolves with network depth.

Theoretical Insight: It reveals that modality fusion is heterogeneous and layer-dependent, requiring soft, data-driven routing rather than rigid separation.
Practical Impact: By aligning expert specialization with deployment constraints (EP), SMoES offers a pathway to deploy large-scale multimodal models on resource-constrained edge devices (e.g., autonomous vehicles) with significantly lower communication costs and higher throughput, while maintaining high accuracy on both vision and language tasks.