💬 NLP

Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models

This paper proposes RA-MoE, a three-stage fine-tuning framework that leverages the language-universal alignment zone in middle layers of Mixture-of-Experts models to align target-language routing patterns with English task-expert activations, thereby significantly improving multilingual downstream performance.

Original authors: Guanzhi Deng, Kuan Wu, Haibo Wang, Shing Yin Wong, Sichun Luo, Linqi Song

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guanzhi Deng, Kuan Wu, Haibo Wang, Shing Yin Wong, Sichun Luo, Linqi Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, super-smart library of knowledge (a Large Language Model) that speaks perfect English. Inside this library, there isn't just one giant brain doing all the work. Instead, it's organized like a Mixture-of-Experts (MoE) system: think of it as a huge team of specialized consultants. When you ask a question, a "router" (like a receptionist) decides which specific consultants to call into the room to help answer.

The problem is that while this library is a genius at English, it often stumbles when asked to do the same tasks in other languages (like Spanish, Chinese, or Arabic).

The Discovery: The "Middle" is the Magic Zone

The researchers noticed something weird about how this library works. They found that the "receptionist" behaves differently depending on which layer of the building you are on:

The Lobby (Early Layers): The receptionist is very picky about language here. If you speak English, they call English consultants. If you speak Spanish, they call Spanish consultants.
The Penthouse (Late Layers): Same thing. The language barrier is strong.
The Middle Floors: Here, the receptionist stops caring about the language. Whether you speak English or Spanish, they call the exact same group of experts to solve a math problem or follow instructions.

The researchers realized that for many tasks, the library already knows how to solve the problem in the target language (because the middle experts are the same), but the "receptionist" is failing to send the request to the right people. It's like having a brilliant chef in the kitchen who can cook a perfect dish in any language, but the waiter keeps sending the wrong order to the wrong station.

The Solution: RA-MoE (The "Guide the Receptionist" Method)

The paper proposes a new training method called RA-MoE (Routing-Aligned Mixture-of-Experts). Instead of just teaching the whole library new words, they teach the "receptionist" to behave better.

Here is how it works in three simple steps:

1. The "Double-Check" Audit (Stage 1)
They take a list of questions and ask the library the same question in English and in the target language (e.g., Spanish). They categorize the results into four groups:

CC: Got it right in both.
II: Got it wrong in both (the library just doesn't know the answer).
IC: Got it wrong in English but right in Spanish (rare).
CI (The Gold Mine): Got it right in English but wrong in Spanish.

The researchers focus only on the CI group. These are the cases where the library has the knowledge (it solved it in English) but failed in Spanish. This proves the problem isn't a lack of knowledge; it's a routing error.

2. Mapping the Experts (Stage 2)
They look at the "middle floors" of the library where the language doesn't matter. They identify exactly which specific consultants (experts) the English version called to solve the problem. They say, "Okay, for this math problem, the English version called Experts #4, #7, and #12. Those are the 'Task Experts'."

3. The Alignment Lesson (Stage 3)
Now, they fine-tune the library using the Spanish questions. But they add a special rule:

When the library sees a CI example (where it failed in Spanish but would have succeeded in English), they tell the receptionist: "Hey, look at how the English version handled this. It called Experts #4, #7, and #12. You need to call those same experts for the Spanish version, too."

They don't just teach the model to speak Spanish; they teach the model to route the Spanish question to the same experts that solved the English version.

Why It Works

Think of it like a school. If a student knows how to solve a math problem in English but fails in French, you don't need to teach them math from scratch. You just need to teach them that the same math teacher should be helping them, regardless of the language they are using to ask the question.

The paper found that:

This method works better than standard training (which just tries to memorize answers in the new language).
It works better than other methods that try to "steer" the model without actually changing its internal wiring.
The more "CI" examples a task has (meaning the model knows the answer in English but fails in the target language), the more this method helps.
The "Task Experts" they found in the middle layers are universal. Once they figure out which experts handle a math problem for English, those same experts work for Spanish, French, and Japanese without needing to be re-identified.

The Bottom Line

RA-MoE is a smart way to fix multilingual AI. Instead of trying to build a new brain for every language, it fixes the "receptionist" inside the existing brain so that it sends questions to the right experts, no matter what language the question is asked in. It turns a language barrier into a simple routing error that can be easily corrected.

Technical Summary: Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models

Problem Statement

Mixture-of-Experts (MoE) models have become the dominant paradigm for scaling Large Language Models (LLMs) efficiently. However, adapting these pretrained MoE models to non-English downstream tasks remains challenging. Existing fine-tuning approaches typically treat MoE models as monolithic learners, applying standard Supervised Fine-Tuning (SFT) on target-language data. This approach ignores the heterogeneous routing structure that develops during pretraining.

Recent studies indicate a "U-shaped" pattern in cross-lingual routing divergence: early and late layers are highly language-specific, while middle layers exhibit strong cross-lingual alignment. Despite this alignment, non-English performance often degrades because the model fails to engage the specific task-relevant experts for non-English inputs. The core problem is how to leverage this cross-lingual routing structure to improve fine-tuning for non-English tasks without retraining the entire model or ignoring the sparse routing dynamics.

Methodology: RA-MoE

The authors propose RA-MoE (Routing-Aligned MoE Fine-Tuning), a three-stage framework designed to align target-language routing patterns with English task-expert activation patterns in the middle layers of the model.

Stage 1: Parallel Data Construction and Categorization

The method begins by constructing a parallel dataset where English task examples $(x_{en}, y_{en})$ are translated into a target language $(x_{tgt}, y_{tgt})$ . The MoE model performs inference on both versions, and the examples are categorized into a four-way taxonomy based on correctness:

cc: Correct in both English and target language.
ci: Correct in English, incorrect in target language.
ic: Incorrect in English, correct in target language.
ii: Incorrect in both.

The authors hypothesize that the ci category represents a "language-driven" performance gap where the model possesses the requisite task knowledge (demonstrated in English) but fails to apply it in the target language due to routing misalignment.

Stage 2: Middle Layer and Task Expert Identification

This stage analyzes the model's routing behavior to identify the optimal targets for alignment:

Middle Layer Identification: The authors compute the Jensen-Shannon (JS) divergence between English and target-language routing distributions across layers. They confirm the U-shaped pattern, identifying a contiguous range of middle layers ( $L_{mid}$ ) where divergence is minimal. These layers serve as the "language-universal alignment zone."
Task Expert Identification: Within these middle layers, the authors identify "task experts" by contrasting routing weights on English task data (restricted to correctly answered examples) against a general English corpus. Experts that are preferentially activated for the task (high specificity score) are selected as the target set ( $E_{task}$ ).

Stage 3: Routing-Aligned SFT

The final stage involves fine-tuning the model on the target-language data with a modified objective function:
$\mathcal{L} = \mathcal{L}_{CE} + \lambda \cdot \mathcal{L}_{align}$

$\mathcal{L}_{CE}$ : Standard cross-entropy loss applied to all examples.
$\mathcal{L}_{align}$ : A routing alignment loss applied selectively to ci-type examples. For these examples, the model is encouraged to route target-language inputs through the same task experts in the middle layers as the English reference. The loss is computed as the KL divergence between the renormalized routing distribution of the target language (restricted to $E_{task}$ ) and the English reference distribution.

This selective application ensures the alignment signal is grounded in cases where the English routing pattern is a valid, reliable reference for the task.

Key Contributions

Validation of U-Shaped Divergence in Task Settings: The paper empirically validates that the U-shaped cross-lingual routing divergence pattern holds in task-specific settings, demonstrating that middle-layer routing divergence reliably predicts per-language task performance gaps.
Four-Category Taxonomy and Predictor: The authors introduce the cc/ci/ic/ii taxonomy for diagnosing cross-lingual gaps. They provide empirical evidence that the proportion of ci examples in a task-language pair is a strong predictor (correlation $r=0.70$ ) of the potential benefit from routing alignment.
Task Expert Identification and Alignment Objective: A procedure is proposed to localize task-relevant experts in middle layers using English data, coupled with a fine-tuning objective that closes the cross-lingual expert activation gap without requiring language-specific expert re-identification.
Cross-Lingual Transferability: The study demonstrates that task experts identified in one language (e.g., Chinese) can be transferred to fine-tune the model for linguistically distant languages (e.g., Arabic, Bengali) without re-running the identification stage, suggesting middle-layer task expertise is largely language-agnostic.

Experimental Results

The authors evaluated RA-MoE across three MoE models (OLMoE-1B-7B, Qwen1.5-MoE, DeepSeek-V2-Lite), three downstream tasks (GSM8K, IFEval, MMLU), and six target languages (Arabic, Bengali, Chinese, French, Japanese, Spanish).

Performance Gains: RA-MoE consistently outperformed standard SFT and strong baselines, including Routing Steering (inference-time intervention) and RISE (selective fine-tuning of language-specific subnetworks).
Correlation with CI Proportion: The performance gains were most pronounced on task-language pairs with a higher proportion of ci examples, confirming the hypothesis that the method targets language-driven gaps rather than knowledge gaps.
Mechanistic Changes: Analysis showed that RA-MoE substantially reduced middle-layer routing divergence compared to standard SFT, bringing target-language routing closer to the English reference. Furthermore, the task expert selection rate for target languages increased more steeply and converged to a higher plateau under RA-MoE.
Efficiency: The method incurs negligible computational overhead. The alignment loss adds fewer than $10^{-3}\%$ to the total FLOPs, and the preparatory stages (data annotation and routing profiling) add at most 15% to the total wall-clock time, reducible to under 10% if task experts are shared across languages.

Significance and Limitations

The paper claims that RA-MoE bridges the gap between the theoretical cross-lingual alignment of MoE middle layers and practical downstream task performance. By explicitly aligning routing patterns rather than just output probabilities, the method leverages the sparse structure of MoEs to improve multilingual capabilities.

The authors note specific limitations:

Dependence on English Performance: The method presupposes the base model has strong English task performance; if the English routing pattern lacks task-relevant information, the alignment signal is unreliable.
Pretraining Coverage: For languages severely under-represented in the pretraining corpus, RA-MoE offers limited gains because the bottleneck is a lack of foundational language capacity, not routing misalignment.
Modality: Experiments are limited to text-only models; generalization to multimodal LLMs remains an open question.

The work suggests that for MoE models, the middle layers serve as a stable, language-universal zone where task expertise can be effectively transferred across languages through routing alignment.