Technical Summary: DeltaMoE – A Data-Efficient Path to Multilingual LLMs

1. Problem Statement

Expanding Large Language Models (LLMs) to new languages is currently a resource-intensive process. The standard pipeline involves Continued Pre-Training (CPT) on massive multilingual corpora followed by a complex post-training alignment phase (e.g., SFT and RL). This approach demands extensive data replay to prevent catastrophic forgetting of original capabilities and requires vast amounts of high-quality instruction data for alignment.

Recent "data-free" merging techniques attempt to bypass the alignment phase by fusing a CPT-enhanced model with an existing instruct model. However, these methods face a critical trade-off:

• Linear Averaging (Model Soups): Dilutes the newly acquired language knowledge, limiting gains in expanded languages.
• Delta Merging: Directly applying parameter changes often causes catastrophic forgetting of the model's original language capabilities.

The core challenge is to acquire new language and alignment capabilities efficiently without sacrificing the model's original performance.

2. Methodology: DeltaMoE

The authors propose DeltaMoE, a two-stage framework that upcycles a dense model into a Mixture-of-Experts (MoE) architecture to resolve the aforementioned trade-off.

Stage 1: Continued Pre-Training via Sparse Upcycling

The dense base model is transformed into an MoE architecture by creating deep copies of the original Feed-Forward Networks (FFNs) to serve as "experts."

• Architecture: The model consists of $N$ experts and a router. The original parameters (specifically the 0-th expert) are frozen to act as a dedicated repository for original language knowledge.
• Training: Only the newly added expansion experts and the router are updated during CPT on the new language data.
• Objective: The model minimizes Next Token Prediction (NTP) loss while using a load-balancing loss (LLB) to ensure efficient token distribution among experts. This ensures the original knowledge remains intact while new languages are learned by the active experts.

Stage 2: MoE Model Merging (Delta Integration)

Once the MoE base model is trained on new languages, it lacks alignment capabilities. Instead of costly post-training, the authors graft a parameter delta ($\Delta_{post}$) from a well-aligned, dense instruct model.

• Delta Calculation: $\Delta_{post} = \theta_{post-trained} - \theta_{base}$, representing the alignment knowledge gained during the instruct model's training.
• Component-wise Grafting:
  • Shared Parameters: For layers common to both dense and MoE architectures (e.g., Attention, Embeddings), the delta is applied directly.
  • Expert Parameters: For the FFN layers, a single FFN-specific delta is computed from the dense model's FFN weights. This delta is then uniformly applied to all experts in the MoE architecture.
• Rationale: This strategy treats all experts as inheriting the same foundational structure, allowing the alignment knowledge to be broadcast across all experts without diluting the specific language knowledge stored in the frozen or trained experts.

3. Key Contributions

1. Novel Framework: Introduction of DeltaMoE, which integrates MoE upcycling with a refined delta merging strategy to expand languages and acquire alignment abilities without heavy post-training procedures.
2. Resolution of Trade-offs: The method effectively decouples knowledge acquisition (via new experts) from knowledge retention (via frozen experts) and alignment transfer (via delta grafting), resolving the conflict between expanding capabilities and preserving original ones.
3. Generalizability: The approach is validated across different base models (Qwen2.5-7B, LLaMA-3.1-8B) and different sources of alignment deltas (standard instruct, Tulu3 SFT), demonstrating robustness.

4. Experimental Results

Experiments were conducted on expanding Qwen2.5-7B and LLaMA-3.1-8B to Hungarian, Serbian, and Bengali, evaluated against baselines with matched data, FLOPs, and parameter counts.

• Performance on Expanded Languages: DeltaMoE achieved State-of-the-Art (SOTA) performance, improving average scores on expanded languages by 1.7 points over other delta-based methods with comparable parameters.
• Preservation of Original Capabilities: Unlike dense delta merging which caused significant forgetting, DeltaMoE preserved original language performance, outperforming dense delta baselines by 1.5 points in retention metrics.
• Efficiency: Even when dense baselines were trained on twice the amount of CPT data (to match the FLOPs of the MoE training), DeltaMoE maintained a significant lead (approx. 1.94 points) in expanded language performance.
• Ablation Studies:
  • Merging Strategy: Linear averaging within an MoE architecture failed to resolve the trade-off, confirming the necessity of the delta merging approach.
  • Router Tuning: A minimal data replay strategy (0.15B tokens) combined with router tuning allowed DeltaMoE to surpass dense averaging baselines in knowledge retention.
  • Scalability: Increasing the number of trainable experts beyond three yielded negligible gains, suggesting 3 trainable experts are sufficient for multiple expanded languages.

5. Significance and Claims

The paper claims that DeltaMoE offers a practical and scalable pathway for extending the multilingual alignment of existing LLMs. By upcycling dense models into MoE architectures and grafting alignment deltas, the method:

• Eliminates the need for massive instruction datasets and complex RL alignment phases for new languages.
• Solves the "forgetting vs. learning" dilemma inherent in current merging techniques.
• Provides a data-efficient alternative that achieves superior performance with significantly less computational overhead compared to scaling up dense training data.

The authors acknowledge limitations, noting that the evaluated languages do not fully represent global linguistic diversity and that the MoE architecture introduces non-trivial computational overhead compared to dense models during inference. However, the framework is presented as a robust solution for efficient multilingual expansion.