Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

Technical Summary: Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

Problem Statement
While Large Language Models (LLMs) demonstrate strong capabilities in multilingual machine translation (MT), fine-tuning them on parallel corpora introduces a significant challenge: parameter interference. When an LLM is fine-tuned for a downstream task like translation using new data, its parameters shift to optimize for the new task, often causing the degradation of its original, strong monolingual language understanding and generation capabilities. Existing approaches, such as generic Mixture-of-Experts (MoE) architectures or parameter isolation methods, often lack task-specific designs that explicitly leverage the pre-trained monolingual knowledge to aid translation, or they rely on routing mechanisms that consider only semantic content, potentially overlooking underlying structural or rhythmic patterns in text.

Methodology
The authors propose Mix-MoE, a mixed Mixture-of-Experts framework designed to train LLMs for multilingual MT while mitigating parameter interference. The framework transforms a dense LLM into a sparse MoE model with two distinct groups of experts and employs a two-stage training strategy:

1. Specialized Expert Groups:

   • Language Model (LM) Experts: Designed to capture and retain the monolingual knowledge learned during pre-training. These are initialized by "upcycling" the weights of the original Feed-Forward Network (FFN) layers of the LLM.
   • Machine Translation (MT) Experts: Specialized to acquire and store bilingual translation knowledge. These are initialized from the pre-trained LM Experts.
   • The model targets the FFN layers for MoE transformation, as these layers function as key-value memories for linguistic knowledge, while keeping Attention layers dense to maintain context awareness.

2. FFT-Enhanced Routing Mechanism:

   • To facilitate effective interaction between experts and capture diverse linguistic cues, the authors introduce a routing mechanism enhanced by Fast Fourier Transform (FFT) features.
   • Spectral features are extracted from the model's hidden states via FFT and concatenated with the original semantic representations.
   • Separate routing networks (one for LM experts, one for MT experts) use these augmented representations to select experts. This allows the router to consider not just semantic content but also frequency-domain patterns indicative of text structure (e.g., sentence rhythm, morphological complexity).

3. Two-Stage Post-Pretraining Strategy:

   • Stage 1 (LM Expert Training): The model is trained on monolingual corpora. Only the LM Experts and their corresponding routing network are updated. The objective is to refine the representation of individual language structures.
   • Stage 2 (MT Expert Training): The model is trained on parallel bilingual corpora. The LM Experts and their router parameters are frozen to preserve the acquired monolingual knowledge. Only the MT Experts and their routing network are trained to specialize in the translation task.

Key Contributions

1. Mix-MoE Architecture: A novel MoE architecture featuring specialized LM and MT experts, coupled with an FFT-enhanced routing mechanism that leverages both semantic and potential structural cues from text.
2. Two-Stage Training Strategy: A tailored training pipeline that allows MT Experts to benefit from pre-trained knowledge while specializing in translation, effectively separating monolingual and bilingual learning objectives to mitigate parameter interference.
3. Empirical Validation: Extensive experiments demonstrating that the proposed method achieves state-of-the-art results across multiple translation tasks and language pairs.

Experimental Results
The authors evaluated Mix-MoE on 14 language directions using the WMT dataset (including Finnish, Czech, German, Russian, Turkish, Estonian, and Chinese to/from English) and the FLORES-200 benchmark.

• Performance: Mix-MoE consistently outperformed baselines, including Full Fine-tuning, LoRA, LLaMA-Pro, and generic MoE variants (e.g., MoE-LPR, LLaMA-MoE v2). For instance, in the DE-EN task, Mix-MoE achieved a BLEU score of 31.8, significantly surpassing the Two-Stage Full Fine-tuning baseline (20.8).
• Parameter Interference Mitigation: Ablation studies confirmed that the separation of experts is crucial. A standard two-stage MoE training (updating all experts in Stage 2) performed worse than Mix-MoE, indicating that the specific freezing of LM Experts prevents catastrophic forgetting.
• FFT Routing: Comparisons with Random Projection and Discrete Cosine Transform (DCT) showed that FFT-derived features significantly improved routing effectiveness, confirming that frequency-domain patterns provide valuable structural cues for expert selection.
• Scalability: The method demonstrated robustness and scalability when extended to larger models (Llama3.1-8B and Qwen2.5-14B), consistently outperforming full fine-tuning approaches.

Significance and Claims
The paper claims that Mix-MoE addresses the critical challenge of parameter interference in fine-tuning LLMs for multilingual translation. By explicitly separating the learning of monolingual language modeling and bilingual translation into distinct expert groups, and by utilizing a routing mechanism sensitive to structural text patterns, the framework successfully preserves the original language capabilities of the LLM while enhancing translation performance. The authors position this work as a step forward in efficient MT training, offering a practical solution for deploying multilingual translation systems that maintain high-quality language understanding without the redundancy of massive parameter updates.