💬 NLP

Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

This paper introduces the concept of Language Routing Isolation in multilingual Mixture-of-Experts models and proposes RISE, a framework that leverages these isolation patterns to selectively adapt language-specific expert subnetworks, thereby significantly improving low-resource language performance while preserving cross-lingual capabilities.

Original authors: Kening Zheng, Wei-Chieh Huang, Jiahao Huo, Zhonghao Li, Henry Peng Zou, Yibo Yan, Xin Zou, Jungang Li, Junzhuo Li, Hanrong Zhang, Xuming Hu, Philip S. Yu

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kening Zheng, Wei-Chieh Huang, Jiahao Huo, Zhonghao Li, Henry Peng Zou, Yibo Yan, Xin Zou, Jungang Li, Junzhuo Li, Hanrong Zhang, Xuming Hu, Philip S. Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Super-Factory" Problem

Imagine a massive, futuristic factory called a Mixture-of-Experts (MoE) Model. This factory is designed to solve problems in dozens of different languages (like English, Bengali, Swahili, and Chinese).

Inside this factory, there are thousands of specialized workers (called Experts). When a question comes in, a smart manager (the Router) decides which specific workers should handle it.

If the question is in English, the manager might send it to a team of experts who are great at English logic.
If the question is in Bengali, the manager sends it to a different team.

The Problem:
The factory works amazingly well for popular languages (like English) but struggles terribly with less common languages (like Bengali or Swahili). The answers are often wrong, incomplete, or nonsensical.

Why? The researchers asked: "Are the workers for Bengali actually different, or is the manager just sending Bengali questions to the wrong people?"

The Discovery: "Language Routing Isolation"

The researchers discovered a fascinating phenomenon they call Language Routing Isolation.

The Analogy: The Two Separate Cafeterias
Imagine the factory has a giant cafeteria with 1,000 food stations.

High-Resource Languages (English, Chinese): These languages have their own private, exclusive cafeteria. They have access to a huge variety of stations, and they rotate through them constantly. They are like VIPs with a full menu.
Low-Resource Languages (Bengali, Swahili): These languages are stuck in a tiny, cramped corner of the cafeteria. They are forced to use the same 3 or 4 stations over and over again, no matter what they are trying to order.

The Finding:
The VIPs and the corner-dwellers almost never use the same food stations. Their paths are completely separate (isolated).

The VIPs are well-trained and have many options.
The corner-dwellers are stuck with a limited, repetitive set of workers who haven't been trained enough to handle complex tasks.

The researchers also found that this separation changes depending on where in the factory you are:

The Entrance (Shallow Layers): Languages start with very different workers.
The Middle Hall (Middle Layers): Everyone mixes together! The workers here are "universal" and handle general concepts that apply to all languages.
The Exit (Deep Layers): The languages split up again, going to their specific, specialized workers to finish the job.

The Solution: RISE (Routing Isolation-guided Subnetwork Enhancement)

The researchers didn't just want to point out the problem; they wanted to fix it without rebuilding the whole factory (which would be too expensive). They created a method called RISE.

The Analogy: The "Targeted Renovation"
Instead of firing everyone and hiring new staff, or trying to teach every single worker in the factory everything, RISE acts like a smart renovation crew.

Map the Territory: First, they study the factory to see exactly which workers are currently handling Bengali questions.
Pick the Right Team:
- At the Entrance and Exit: They pick the specific workers who are only used for Bengali (the "Bengali Specialists").
- In the Middle Hall: They pick the "Universal Workers" who are good at everything and help everyone.
The Renovation: They lock the doors to the rest of the factory (freezing the other parameters) and only train the selected team on Bengali data.

Why this is brilliant:

No Collateral Damage: Because they didn't touch the workers who handle English, the factory's English performance doesn't get worse.
Efficiency: They only trained a tiny fraction of the workers (about 2-3%), saving massive amounts of time and energy.
Result: The Bengali team suddenly became experts, giving much better answers, while the English team kept doing their job perfectly.

The Results: What Happened?

The researchers tested this on 10 different languages.

Before RISE: The factory was great at English but terrible at Bengali (sometimes getting the math wrong or stopping mid-sentence).
After RISE: The Bengali performance jumped significantly (up to 10.85% improvement in accuracy).
The Best Part: The factory didn't forget how to speak English, French, or Japanese. The "Universal Workers" in the middle kept everything running smoothly.

Summary in One Sentence

The paper discovered that different languages in AI models use completely separate sets of "workers," and by identifying and specifically training just the right few workers for a struggling language, we can fix that language's performance without breaking the rest of the model.

Think of it like this: If your car's engine is making a weird noise only when you drive in the rain, you don't replace the whole car. You just find the specific part that gets wet and fix that. RISE does exactly that for AI languages.

1. Problem Statement

Mixture-of-Experts (MoE) models have become a dominant architecture for scaling Large Language Models (LLMs) due to their ability to handle massive parameter counts while maintaining inference efficiency. However, these models exhibit significant performance disparities across languages. High-resource languages (e.g., English, Chinese) often achieve high accuracy, while low-resource languages (e.g., Bengali, Swahili) suffer from substantial degradation.

Existing research has analyzed expert activation frequencies or layer-wise distributions in isolation but has failed to explain the internal mechanisms driving these gaps. Specifically, it remains unclear how MoE models route different languages through their expert networks and whether distinct subnetworks exist for specific languages that could be targeted for adaptation without catastrophic forgetting of other capabilities.

2. Key Empirical Findings

The authors conducted a systematic analysis of routing patterns in multilingual MoE models (specifically Qwen3-30B-A3B and Phi-3.5-MoE-Instruct) and uncovered two critical phenomena:

Language Routing Isolation: High-resource and low-resource languages tend to activate largely disjoint sets of experts.
- High-resource languages (e.g., English) utilize a flexible, diverse set of experts with little overlap in their top-30 choices.
- Low-resource languages repeatedly trigger a small, fixed subset of experts, likely due to limited training data.
- The intersection of expert sets between high- and low-resource groups is negligible (near-orthogonal), suggesting that low-resource languages rely on a specific, isolated subnetwork.
Layer-wise Convergence–Divergence Pattern: Routing behavior changes systematically with model depth:
- Shallow Layers: High divergence; experts are language-specific (encoding).
- Middle Layers: High convergence; experts are shared across languages (language-agnostic semantic processing).
- Deep Layers: High divergence; experts are language-specific again (generation).

3. Methodology: RISE (Routing Isolation-guided Subnetwork Enhancement)

Based on these findings, the authors propose RISE, a framework designed to adapt low-resource languages by identifying and training specific expert subnetworks while freezing the rest of the model.

Core Mechanism:
RISE employs a tripartite selection strategy based on layer-aware analysis:

Shallow & Deep Layers (Language-Specific):
- Uses a Specificity Score to identify experts predominantly activated by the target low-resource language.
- The score compares the activation probability of an expert on the target language against its average activation across all languages.
Middle Layers (Language-Agnostic):
- Uses an Overlap Score (based on the coefficient of variation) to select experts that are shared across languages.
- This preserves the model's general semantic understanding and cross-lingual capabilities.
Composite Scoring:
- To ensure selected experts are not just specific but also highly active, the authors combine specificity/overlap scores with absolute activation magnitudes.
- A total budget of $K$ experts is allocated across the three layer groups (Shallow, Middle, Deep) using predefined ratios.

Training Protocol:

Only the selected expert subnetwork parameters are updated via standard causal language modeling loss on the target language data.
All other parameters (including non-selected experts and shared layers) remain frozen.
Theoretical Guarantee: The paper provides a mathematical proof (Gradient Isolation) showing that if the routing supports of the target and non-target languages are disjoint, training the target subnetwork induces zero gradient updates for the non-target languages, theoretically preventing catastrophic forgetting.

4. Experimental Results

The authors evaluated RISE on 10 languages using two benchmarks: TyDiQA-GoldP (Question Answering) and MGSM (Math Reasoning).

Target Language Performance: RISE achieved significant F1 score improvements for low-resource languages.
- On Phi-3.5-MoE-Instruct, Bengali F1 improved by 10.85% over the vanilla baseline.
- On Qwen3-30B-A3B, RISE achieved the best performance across all competing methods (Random selection, Top-K, ESFT, LoRA).
Cross-Lingual & Cross-Task Preservation:
- Unlike LoRA, which caused severe interference (e.g., Thai accuracy dropped by 16.5% on Phi-3), RISE maintained performance on non-target languages.
- General capabilities (TriviaQA, MMLU, HellaSwag, ARC) remained stable, with changes of less than 1%.
- Models trained on TyDiQA (QA) did not degrade performance on MGSM (Math), confirming the subnetworks are language-specific rather than task-general.
Ablation Studies:
- Removing shallow or deep layer experts caused the largest performance drops, confirming their critical role in language-specific encoding/generation.
- The method is robust to hyperparameter choices (activation scale factor $\alpha$ , budget allocation ratios).

5. Key Contributions

Empirical Discovery: First to systematically identify and characterize Language Routing Isolation and the Layer-wise Convergence–Divergence pattern in multilingual MoE models.
Methodological Innovation: Proposed RISE, an interpretable, layer-stratified expert selection method that adapts low-resource languages by training only the relevant subnetwork.
Theoretical Insight: Provided a formal proof of Gradient Isolation, demonstrating that routing isolation leads to functional decoupling, allowing for targeted adaptation without catastrophic forgetting.
Efficiency: Demonstrated that training a tiny fraction of parameters (e.g., 2.1% of experts in Qwen3) yields massive gains for low-resource languages while preserving global capabilities.

6. Significance

This work shifts the paradigm for adapting multilingual MoE models from "fine-tuning the whole model" or "generic parameter-efficient fine-tuning (PEFT)" to mechanistic, routing-aware subnetwork adaptation.

Equity: It offers a practical solution to the "low-resource language gap" in MoE models, enabling equitable performance across languages without retraining the entire model.
Interpretability: It validates that MoE models possess an interpretable internal topology where languages are processed by distinct pathways, which can be exploited for efficient adaptation.
Scalability: As MoE models scale to trillions of parameters, RISE provides a blueprint for efficiently updating specific language capabilities without incurring the computational cost of full fine-tuning or the interference risks of standard PEFT methods like LoRA.