Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation

Technical Summary: Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation

Problem Statement
Mixture-of-Experts (MoE) models are the standard for scaling Large Language Models (LLMs) due to their ability to maintain massive parameter counts while keeping inference costs manageable via sparse activation. However, the internal routing behavior of MoE models in multilingual settings remains underexplored. While dense multilingual models are known to rely on English as an internal "pivot" in middle layers with language-specific neurons in the first and last layers, it is unclear how MoEs develop specialized routing strategies for different languages. Furthermore, adapting large MoEs to new, low-resource languages is computationally expensive. Although their modular structure theoretically allows for parameter-efficient adaptation by updating only a subset of experts, it is currently unknown which experts to update without a prior understanding of multilingual routing dynamics.

Methodology
The authors conduct a controlled study using the OLMoE-Base model (a decoder-only transformer with 64 experts per layer, selecting top-8) to investigate routing dynamics during continual pre-training.

1. Continual Pre-training Setup: The English-centric OLMoE-Base model is continually pre-trained on a balanced 35-billion-token corpus spanning seven high-resource languages (English, Arabic, Czech, Spanish, Finnish, Hindi, Russian) sampled from the CulturaX dataset. The resulting model is referred to as OLMoE-M7.
2. Routing Analysis Metrics: To characterize how routing mass is distributed, the authors analyze:
   • Router Entropy: Measures how concentrated expert activation is for a given language. Lower entropy implies tokens are routed to a small subset of experts.
   • Cross-lingual Routing Divergence (JSD): Jensen–Shannon Divergence between language-level expert usage distributions to quantify how distinct routing patterns are between language pairs.
   • Vocabulary Overlap: Token-level vocabulary overlap between language pairs is analyzed to see if it drives routing similarity more than typological or family-based similarity.
3. Adaptation Strategy (SSFT): Motivated by the routing analysis, the authors propose Selective and Shared Expert Finetuning (SSFT) for adapting to low-resource languages.
   • Expert Selection: They define an "activation gap" (the difference in activation frequency between an expert's top two languages) to identify language-specific experts.
   • Hybrid Update: SSFT updates only the final two layers of the model. It selects a small set of language-specific experts (those with a high activation gap for the target language) and augments them with a small set of "shared" experts (highly active across all languages).
   • Baselines: The method is compared against Selective Expert Finetuning (SEFT, language-specific only), Random Expert Finetuning, All-Experts Finetuning (AEFT), and Full-Model Finetuning (Full-FT).

Key Findings on Routing Dynamics

• Layer-wise Specialization: Contrary to the hypothesis that balanced multilingual training induces language separation throughout the model, the study finds that routing remains diffuse and language-agnostic in early and middle layers. Distinct language specialization emerges primarily in the final layers.
• Vocabulary Overlap vs. Language Family: In the final layers, routing similarity correlates more strongly with token-level vocabulary overlap than with language family, syntactic, or geographic similarity. For example, Hindi and Marathi (high vocabulary overlap) show similar routing patterns, while English and Dutch (shared family but low overlap) show divergent patterns.
• English Dominance: The base model exhibits English-dominated routing patterns. Continual pre-training reduces this dominance but does not eliminate it; English tokens still maintain distinct routing patterns compared to non-English languages in the final layers.

Results on Low-Resource Adaptation
The authors evaluate SSFT on six low-resource languages (Catalan, Estonian, Marathi, Slovak, Ukrainian, Urdu) paired with high-resource anchors.

• Performance: SSFT achieves a strong performance-efficiency trade-off. On the MultiBLiMP benchmark, SSFT achieves an average accuracy of 83.6%, outperforming SEFT (78.7%) and Random-SEFT (73.8%). It approaches the performance of Full-FT (90.5%) while updating significantly fewer parameters.
• Efficiency: SSFT updates less than 2% of the model parameters (approx. 75–138M parameters). In terms of computational cost, SSFT requires approximately 10x fewer GPU-hours and 100x fewer FLOPs compared to Full-FT.
• Shared Experts: The inclusion of shared experts in SSFT is critical; updating only language-specific experts (SEFT) yields lower performance, and simply increasing the number of language-specific experts (SEFT-Top20) does not match the performance of the hybrid SSFT approach.
• Robustness: The authors note that while Full-FT leads to catastrophic forgetting of previously learned languages when trained on larger datasets, SSFT demonstrates less catastrophic forgetting.

Significance and Claims
The paper claims to provide practical insights for low-resource multilingual adaptation in MoE models. By demonstrating that language specialization in MoEs is concentrated in the final layers and driven by vocabulary overlap, the authors show that efficient adaptation does not require full model fine-tuning. The proposed SSFT strategy leverages these dynamics to achieve competitive performance with minimal parameter updates. The work suggests that effective adaptation relies on a hybrid approach: targeting specialized experts for the new language while preserving shared experts to maintain cross-lingual stability. The authors conclude that these findings offer a pathway to scalable, efficient language adaptation without the computational burden of full fine-tuning.

Limitations
The authors acknowledge that their experiments are limited to the OLMoE-Base architecture (1B active / 7B total parameters) due to computational constraints. They also note that their strategy relies on the existence of a high-resource "anchor" language with significant vocabulary overlap, which may limit applicability to language isolates or low-resource languages without close relatives in the pre-training data. Finally, the findings are specific to continual pre-training on balanced corpora and may differ under other training regimens.