Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights

1. Problem Statement

Large Language Models (LLMs) exhibit significant performance disparities between high-resource languages (e.g., English, Spanish) and low-resource languages (e.g., Basque, Galician, Catalan). While existing adaptation methods like continual pre-training and instruction tuning can improve performance, they face critical barriers for low-resource communities:

• Computational Cost: Training or fine-tuning large models requires massive GPU resources.
• Data Scarcity: High-quality instruction-response pairs in low-resource languages are often non-existent or difficult to curate.
• Repetitive Effort: Every time a new, more capable "instructed" base model is released (e.g., Llama 3.1 $\to$ Llama 3.2), low-resource communities must repeat the expensive adaptation process from scratch.

The paper posits that model merging offers a lightweight, compute-efficient alternative to transfer language proficiency from a language-specific base model to a general instructed model without requiring new instruction data or full retraining.

2. Methodology

The authors propose a two-stage pipeline to create language-adapted instructed LLMs:

A. Continued Pre-training (Language Adaptation)

First, they create language-specific base models by continuing pre-training on massive corpora for four Iberian languages: Basque (eu), Galician (gl), Catalan (ca), and Spanish (es).

• Corpus: They utilized the Latxa corpus (Basque), CorpusNÓS (Galician), and filtered subsets of CulturaX (Spanish/Catalan).
• Constraint: To ensure fair comparison, they limited the token count for all languages to be roughly equivalent (approx. 3.5B–3.8B tokens).
• English Injection: A small English corpus was included to prevent catastrophic forgetting of general capabilities.

B. Model Merging

The core innovation involves merging the newly trained language-specific base model with an existing general instructed model (e.g., Llama 3.1 Instruct or Qwen 3 Instruct). The goal is to inject language proficiency while retaining instruction-following capabilities.
The study evaluates four merging strategies:

1. Linear Merging: Simple weighted interpolation of parameters ($\theta_{merge} = w_{base}\theta_{base} + w_{instruct}\theta_{instruct}$).
2. Task Arithmetic: Merging based on task vectors (differences between the fine-tuned model and the base).
3. DARE (Drop And REscale): Sparsifies task vectors by randomly dropping parameters and rescaling the rest to reduce interference.
4. Breadcrumbs: Filters out negligible and extreme parameter updates, retaining only mid-range informative updates.

The authors tested these methods in two settings:

• Monolingual Merging: Adapting a model to a single target language.
• Multilingual Merging: Combining multiple language-specific base models into a single multilingual instructed model.

3. Key Contributions

1. Feasibility of Merging for Low-Resource Languages: Demonstrated that merging can successfully transfer language proficiency to instructed models, eliminating the need for language-specific instruction datasets.
2. Comprehensive Evaluation: Conducted extensive experiments across four languages and two model families (Llama 3.1 and Qwen 3, ranging from 8B to 14B parameters).
3. Benchmark Expansion: Extended the IFEval (Instruction Following Evaluation) benchmark to Basque and Galician, creating new evaluation datasets for these languages.
4. Resource Release: Publicly released the continued pre-trained base models, the merged instructed models, and the new IFEval variants for Basque and Galician.

4. Experimental Results

Performance on Benchmarks and Translation

• Language Proficiency: Merged models significantly outperformed the original non-adapted instructed baselines in their target languages. For low-resource languages like Basque, the improvements were substantial, often matching or approaching the performance of models trained via costly joint pre-training and instruction tuning.
• Machine Translation: Merged models showed marked improvements in BLEU scores for translation tasks, particularly in low-resource directions.
• Multilingual Capability: A single model merged with multiple language-specific experts successfully acquired competence in all target languages, though with slightly lower peak performance per language compared to monolingual merges.

Instruction Following (IFEval)

• Mixed Results: While merging successfully transferred language knowledge, instruction-following capabilities were more sensitive to the merging method.
• Method Sensitivity:
  • Linear Merging generally provided the most stable balance between language proficiency and instruction following.
  • Task Arithmetic showed high variance; it sometimes achieved excellent instruction following but could degrade language performance if the weighting ($w_i$) was not optimal.
  • Breadcrumbs and DARE often led to performance collapse in machine translation tasks for certain languages (e.g., Galician/Catalan).
• Comparison to SOTA: While merged models did not always surpass the performance of models trained via joint pre-training + instruction tuning (the current SOTA for low-resource adaptation), they achieved competitive performance with a fraction of the computational cost.

Hyperparameter Analysis

• The weighting parameter ($w_i$) is critical. Increasing the weight of the language-specific base model improves language proficiency but degrades instruction following.
• The optimal range for $w_i$ was found to be between 0.5 and 1.0 for Linear merging, providing a robust trade-off.

5. Significance and Conclusion

This paper establishes model merging as a viable, efficient strategy for democratizing LLM access for low-resource languages.

• Efficiency: It drastically reduces the computational barrier. Instead of retraining an instructed model from scratch for every new language, researchers can simply merge a pre-trained language base with a new state-of-the-art instructed model.
• Scalability: The approach allows smaller research groups to adapt frontier models to their local languages without access to massive GPU clusters.
• Future Direction: The authors note that while merging works well for language transfer, preserving alignment (instruction following) remains a challenge. Future work should focus on "merge alignment" techniques to further stabilize instruction-following capabilities during the transfer process.

In summary, "Merge and Conquer" proves that we can effectively "teach" new languages to instruction-tuned LLMs by simply adding their weights, offering a sustainable path forward for multilingual AI development.