💬 NLP

One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging

This paper investigates why standard weight-space model merging fails in multilingual machine translation, revealing that fine-tuning redistributes language-specific neurons to increase representational divergence in higher layers, thereby undermining the geometric compatibility required for successful merging.

Original authors: Baban Gain, Asif Ekbal, Trilok Nath Singh

Published 2026-04-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Baban Gain, Asif Ekbal, Trilok Nath Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Trying to Mix Specialized Chefs

Imagine you have four brilliant chefs.

Chef A is a master of Indian cuisine (Hindi).
Chef B is a master of Bengali cuisine.
Chef C is a master of Tamil cuisine.
Chef D is a master of Telugu cuisine.

Each chef has spent years in their own kitchen, learning the exact spices, techniques, and flavors needed to make perfect dishes for their specific region. They are all experts, but they have very different "muscle memory" for cooking.

The Goal: You want to combine all four chefs into one single super-chef who can cook any of these dishes perfectly, without hiring four separate people.

The Method (Model Merging): Instead of training a new chef from scratch (which is expensive and slow), you try to simply "mix" the brains of the four chefs together. You take their mental notes (weights) and average them out, hoping the result is a chef who knows everything.

The Result: The paper finds that this doesn't work well. The resulting "super-chef" is confused. They can't cook the dishes as well as the individual experts, and sometimes they can't cook them at all.

The Journey to "Mount Doom"

The title refers to The Lord of the Rings. The authors tried to merge these models to create a perfect translator, but they found that the process is like a journey to Mount Doom: it's a dangerous path where the "ring" (the perfect merged model) is destroyed by the journey itself.

Here is what they discovered along the way:

1. The "One-to-Many" Problem (The Confused Translator)

The researchers tested two main scenarios:

Scenario A (Many-to-One): Taking four different languages (Hindi, Bengali, etc.) and trying to translate them all into English.
- Result: This worked okay. The merged model became a decent "generalist" translator. It wasn't as perfect as the experts, but it was stable.
Scenario B (One-to-Many): Taking English and trying to translate it into four different languages (Hindi, Bengali, etc.).
- Result: Total Disaster. The merged model collapsed. It retained almost none of the ability to speak the specific target languages. It was like a chef who knows how to eat food but forgot how to cook it.

Why? The paper suggests that when you teach a model to speak a specific language, it reorganizes its brain in a very specific way. When you try to mix two different "target languages" (like Hindi and Tamil), their brain structures clash violently.

2. The Anatomy of the Failure: Where the Magic Happens

To understand why the merge failed, the authors looked inside the model's "brain" (its neural network), layer by layer. They used a metaphor of a factory assembly line:

The Bottom Layers (The Warehouse): These layers handle the basic vocabulary (the raw ingredients).
- Finding: These layers are very similar across all languages. Whether you are cooking Hindi or Tamil, the "flour and sugar" (basic words) are stored in the same place. Merging these is easy.
The Middle Layers (The Prep Station): These layers start organizing the ingredients.
- Finding: Still mostly shared. The chefs agree on how to chop vegetables.
The Top Layers (The Plating & Serving): These layers decide the final flavor and presentation.
- Finding: This is where the disaster happens. When a model is fine-tuned for Hindi, the "plating" section of its brain changes completely to suit Hindi. When tuned for Tamil, it changes again.
- The Conflict: When you merge the models, you are trying to average the "plating instructions." The result is a confused instruction that satisfies neither language. The specific "flavor" of Hindi gets diluted by the "flavor" of Tamil, and both get ruined.

3. The "Ghost in the Machine" (Neuron Behavior)

The authors found something surprising about how the neurons (the tiny decision-makers in the AI) behave:

They don't split up: You might think that for Hindi, the model uses "Neuron Group A," and for Tamil, it uses "Neuron Group B."
Reality: They use the same neurons for everything.
The Twist: However, the way those neurons fire changes drastically.
- Imagine a group of musicians playing the same instruments. For a Hindi song, they play a specific rhythm. For a Tamil song, they play a different rhythm.
- When you merge the models, you try to average the sheet music. The result is a chaotic rhythm that sounds like noise. The neurons are there, but they are playing out of sync.

The Takeaway: Why "One Size Fits All" Fails Here

The paper concludes that Weight-Space Merging (the technique of just averaging model weights) works great for tasks like "writing a poem" or "solving math" because those tasks are universal.

But for Machine Translation, especially when the target languages are different, the geometry of the model's brain is too specific.

Fine-tuning (training on one language) doesn't just add a new skill; it reshapes the entire landscape of the model's brain to fit that specific language.
When you try to merge two reshaped landscapes, they don't fit together. They are like trying to glue a square peg into a round hole.

Summary Analogy

Imagine you have four different maps of the same city, but each map is drawn by a different cartographer who has optimized the map for a specific type of traveler (a cyclist, a hiker, a driver, and a swimmer).

If you try to average these four maps to make one "Super Map," you won't get a better map. You'll get a blurry mess where the roads for cyclists overlap with the swimming lanes, and the hiking trails lead into the ocean.
The paper shows that for translation, the "roads" (the internal logic of the model) are so specialized for the destination language that you cannot simply average them together. You need a different approach than just "mixing the weights."

1. Problem Statement

While Large Language Models (LLMs) have advanced significantly, training a single multilingual model that performs well across diverse language pairs is computationally expensive and often leads to poor performance on low-resource languages. A common alternative is to independently fine-tune smaller monolingual or task-specific models. However, deploying many specialized models is costly. Model merging offers a solution by combining weights of independently fine-tuned models without accessing original training data.

Despite success in multitask settings (e.g., vision or English-centric tasks), the behavior of weight-space merging in multilingual machine translation (MT) remains poorly understood. Specifically, it is unclear whether merging fully fine-tuned bilingual generative models (trained on million-scale corpora) can successfully consolidate into a single multilingual system, or if the distinct linguistic geometries cause catastrophic failure.

2. Methodology

Experimental Setup

Base Model: Qwen-2.5-3B-Instruct.
Task: Machine Translation on Indic-English pairs (Hindi, Bengali, Tamil, Telugu $\leftrightarrow$ English).
Data: Large-scale bilingual corpora from the Samanantar dataset (ranging from ~5M to 10M sentence pairs per direction).
Fine-tuning: Full-parameter fine-tuning using LLaMAFactory to create strong, task-specialized upper-bound models.
Merging Strategies Evaluated:
- Task Arithmetic: Linear combination of task vectors.
- TIES: Pruning low-magnitude updates and resolving sign conflicts.
- DARE: Randomly dropping and rescaling delta parameters to induce sparsity.
- SCE-Merging: Selecting significant weight changes based on update strength.
Configurations:
1. Many $\to$ One (Shared Target): Merging models translating from different Indic languages to English.
2. One $\to$ Many (Shared Source): Merging models translating from English to different Indic languages.
3. Bidirectional: Merging $L \to E$ and $E \to L$ models to create bidirectional systems.

Analysis Techniques

To explain performance degradation, the authors employed three internal representation analyses:

Span-Conditioned Neuron Selectivity: Analyzing MLP gate activations to identify neurons specific to source (encoding) vs. target (generation) spans and languages.
Neuron Usage Alignment (NUA): Measuring cosine similarity of activation rates across models to determine if they use the same computational units.
Layer-wise Centered Kernel Alignment (CKA): Quantifying the geometric similarity of hidden representations between models at different layers.
Principal Angle Analysis: Measuring the geometric divergence of representation subspaces.

3. Key Results

Performance Degradation

Many $\to$ One (Indic $\to$ En): Merging improves multilingual coverage and stability compared to the base model but fails to preserve peak accuracy. The merged model achieves an average BLEU/CHRF significantly lower than the individual fine-tuned upper bounds.
One $\to$ Many (En $\to$ Indic): Merging causes severe degradation, often performing worse than the pre-trained base model. For example, in English $\to$ Hindi, the merged model retained only ~22% of the fine-tuned BLEU score.
Bidirectional Merging: Exhibits the most extreme directional asymmetry.
- $L \to E$ (Indic to English) is partially preserved.
- $E \to L$ (English to Indic) collapses almost entirely (BLEU retention < 5% for Tamil/Telugu).
Conclusion: Merging distinct target-language generation spaces is significantly more destructive than aggregating diverse source encoders.

Internal Representation Findings

Neuron Localization: Language-specific neurons are not uniformly distributed.
- Embedding Layers: Show increased specialization for Indic inputs.
- Upper Transformer Blocks: Show the most significant reorganization for generation (target spans).
Redistribution vs. Sharpening: Fine-tuning does not simply sharpen language selectivity; it redistributes it. Neurons for supervised languages become less exclusive, while those for unsupervised languages become more isolated.
Geometric Misalignment:
- NUA: Independently fine-tuned models activate largely the same neurons (high overlap), suggesting the failure is not due to disjoint subnetworks.
- CKA & Principal Angles: Despite using the same neurons, the geometric orientation of their activations diverges sharply in the final decoding layers (layers 28–36).
- Asymmetry: Source-side representations remain relatively aligned across models, but target-side generation subspaces become highly language-specific and geometrically incompatible.

4. Key Contributions

Systematic Study: One of the first comprehensive evaluations of weight-space merging for full-parameter, large-scale multilingual MT, with released checkpoints for benchmarking.
Discovery of Directional Asymmetry: Demonstrated that merging fails significantly more when target languages differ (One $\to$ Many) compared to when source languages differ (Many $\to$ One), and that bidirectional merging is particularly fragile.
Mechanistic Explanation: Provided a structural explanation for merging failure:
- Multilingual fine-tuning reshapes the geometry of upper transformer layers (generation blocks) in a target-specific manner.
- Standard merging strategies (linear interpolation) assume geometric compatibility, but these models occupy misaligned generative subspaces in the final layers.
- Merging combines these misaligned subspaces, causing a suppression of language-specific generative features.

5. Significance

This work challenges the assumption that weight-space merging is a universal solution for model consolidation. It reveals that multilingual specialization creates deep, structural geometric shifts in the generative layers of LLMs that standard merging techniques cannot resolve.

The findings suggest that:

Simply averaging weights is insufficient for consolidating specialized generative models.
Future merging strategies must account for layer-wise geometric alignment, particularly in the final decoding blocks where language-specific generation occurs.
The "separation of concerns" in LLMs (semantic vs. language-specific circuits) implies that while semantic understanding may merge well, the mechanisms for controlling output language are highly sensitive to the specific fine-tuning trajectory, leading to the observed "Mount Doom" (catastrophic failure) in multilingual merging scenarios.