💬 NLP

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

This paper demonstrates that improved cross-lingual alignment does not guarantee better downstream transfer performance because alignment and task objectives are often orthogonal and embedding similarity alone is an unreliable predictor of task gains, leading the authors to propose practical guidelines for integrating alignment with task-specific fine-tuning.

Original authors: Yana Veitsman, Yihong Liu, Hinrich Schütze

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yana Veitsman, Yihong Liu, Hinrich Schütze

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a master translator, a brilliant AI named XLM-R, who speaks 100 languages. You want to teach it a new skill, like identifying the "part of speech" for every word (is this a noun? a verb?) or deciding if a sentence is happy or sad.

The common belief in the AI world has been: "If we make the AI's internal dictionary more aligned across languages, it will get better at everything."

Think of "alignment" like organizing a library. If the English book on "Apples" is sitting right next to the Spanish book on "Manzanas," the theory goes that the AI will easily transfer its knowledge from one to the other. The authors of this paper decided to test this theory by literally forcing the AI to move its "Apples" and "Manzanas" closer together in its digital brain.

The Big Surprise:
They found that making the library more organized didn't actually make the AI a better librarian. In fact, sometimes it made the AI worse at its specific jobs.

Here is the breakdown of why this happened, using some everyday analogies:

1. The "Map vs. Compass" Problem

The researchers found that alignment (making the library organized) and task performance (actually finding the book) are two different goals.

The Analogy: Imagine you are training a hiker.
- Alignment is like drawing a perfect, beautiful map where all the mountains are labeled correctly in every language.
- The Task is like teaching the hiker to climb a specific mountain.
- The Problem: Just because the map is perfect doesn't mean the hiker knows how to climb. In fact, if you spend all your time redrawing the map (alignment), you might forget to teach the hiker how to use their boots (the specific task). The paper shows that the "gradients" (the instructions the AI follows to learn) for drawing the map and climbing the mountain are often pointing in opposite directions. It's like trying to walk North while your compass tells you to walk South.

2. The "One-Size-Fits-All" Trap

The researchers tested different ways to organize the library (different "loss functions").

The Analogy: Imagine trying to organize a library by only looking at the color of the book covers.
- If you only care about making "Red" books (English) and "Red" books (Spanish) sit next to each other, you might accidentally shove a cookbook next to a physics textbook just because they both have red covers.
- The paper found that simply making words "look" similar (closer in the digital space) doesn't mean they are useful for the specific job. Sometimes, pushing words closer together actually pushes the AI away from the correct answer for a specific task.

3. The "Fine-Tuning" Dilemma

The paper looked at two ways to teach the AI the new skill:

Linear-Only Fine-Tuning: You keep the AI's brain frozen and just change the very last layer (the "output" layer).
- Analogy: You give the hiker a new pair of boots but don't touch their muscle memory. This worked well. The AI kept its organized library but learned the new skill without messing up the map.
Full Fine-Tuning: You let the AI rewire its entire brain to learn the new skill.
- Analogy: You tell the hiker to forget everything they know about maps and just run up the mountain. This often failed. The AI got so busy trying to learn the new skill that it scrambled the library, moving the "Apples" and "Manzanas" apart again.

4. The "Language Family" Factor

The results also depended on which languages you were teaching.

The Analogy: If you are teaching a French speaker to read Spanish, it's easy because the languages are cousins. But if you are teaching a Japanese speaker to read Kazakh, it's much harder.
The paper found that if you align languages that are very different (like Japanese and Kazakh), the AI sometimes gets confused. The "alignment" helped the retrieval of sentences (finding the right book) but hurt the token-level tasks (reading the words on the page).

The Bottom Line

The paper concludes that just because two things look similar in an AI's brain, doesn't mean the AI will be good at using that similarity.

Don't trust the "Distance" metric: Just because the AI says two words are "close" in its brain, it doesn't mean the AI will perform better on a test.
Be careful with the training: If you want an AI to be good at a specific task, you shouldn't just force it to align languages. You have to be very careful about how you teach it. Sometimes, it's better to leave the "library" alone and just give the AI a better pair of boots (fine-tuning only the last layer) rather than trying to rebuild the whole library.

In short: Better alignment sounds like a good idea, but in the real world of AI, it's like rearranging the furniture in a house. It might look nicer, but it doesn't necessarily help you cook dinner faster. Sometimes, it just gets in the way.

1. Problem Statement

In multilingual Natural Language Processing (NLP), cross-lingual alignment is a widely adopted strategy to improve cross-lingual transfer. The prevailing assumption is that explicitly aligning the embedding spaces of different languages (making semantically equivalent inputs closer) will automatically lead to better performance on downstream tasks in low-resource languages.

However, empirical evidence suggests a persistent mismatch: while explicit alignment techniques often successfully increase embedding similarity (e.g., lower cosine distances), they frequently fail to improve, and sometimes even degrade, token-level downstream performance (such as POS Tagging or Named Entity Recognition). The paper seeks to answer: Why does "better" alignment not reliably translate into "better" transfer?

2. Methodology

The authors conducted a systematic analysis using XLM-R (a 270M parameter encoder-only Transformer) to investigate the interaction between alignment objectives and downstream task objectives.

Experimental Setup

The study followed a three-phase pipeline:

Phase I (Alignment Tuning): The base XLM-R model was fine-tuned using contrastive learning (specifically a modified HiCTL objective) on four language pairs: German-English, English-Kazakh, Japanese-Kazakh, and Korean-Kazakh.
- Loss Configurations: The authors tested 7 distinct combinations of loss coefficients ( $\lambda$ ) for Sentence-level ( $S$ ), Word-level ( $W$ ), and Masked Language Modeling ( $M$ ) objectives.
Phase II (Downstream Fine-tuning): The aligned checkpoints were fine-tuned on English data for two tasks:
- Token-level: POS Tagging (Universal Dependency treebanks).
- Sentence-level: Sentence Classification (SIB-200).
- Fine-tuning Modes: Both Full Model Fine-tuning (updating all parameters) and Linear Layer Only (freezing the backbone, updating only the classifier) were tested.
Phase III (Zero-Shot Evaluation): Models were evaluated on a diverse set of 9 languages (including those present and absent in the alignment phase) to measure transfer performance.

Analysis Metrics

Beyond standard accuracy, the authors employed representational and optimization analyses:

Embedding Distances: Measuring cosine distances between token and sentence embeddings across different layers (4th, 8th, 12th).
Gradient Similarity: Calculating the cosine similarity between gradients of the alignment objective and the downstream task objective to detect conflicts.
Sentence Retrieval: Evaluating cross-lingual retrieval performance on FLORES-200.

3. Key Contributions

Decoupling Alignment from Transfer: The paper demonstrates that embedding distance reduction is not a reliable predictor of downstream task performance. Models can show significantly improved alignment (lower distances) while suffering performance degradation on specific tasks.
Gradient Orthogonality: The authors provide evidence that alignment objectives and downstream task objectives are largely orthogonal (or even conflicting). Optimizing for alignment often pushes model parameters in directions that do not help, and sometimes hurt, the optimization of the specific downstream task.
Task-Level Sensitivity: The study reveals that the negative impact of alignment is highly dependent on the task type. Token-level tasks (POS Tagging) are more susceptible to performance degradation from alignment than sentence-level tasks (Classification).
Practical Guidelines: The authors propose that Linear Layer Only fine-tuning is more stable and beneficial after alignment than Full Model fine-tuning, as the latter amplifies gradient interference.

4. Key Results

Performance vs. Alignment

Misleading Metrics: While alignment tuning successfully reduced cosine distances (improved alignment) in the 8th layer (the language-neutral layer), this did not correlate with accuracy gains. In many cases, particularly for POS Tagging, performance dropped significantly despite "better" alignment.
Loss Configuration Impact:
- Word-level loss only ( $S_0W_1M_0$ ) was consistently detrimental.
- Including Sentence-level loss ( $S$ ) generally benefited sentence classification but had mixed results for POS tagging.
- The MLM objective ( $M$ ) was crucial for maintaining performance, especially when combined with task-appropriate alignment losses.

Gradient Analysis (The Core Finding)

Orthogonality: Table 3 shows that gradients for POS Tagging are frequently negative or near-zero when compared to alignment gradients, indicating that the two objectives pull the model in opposing directions.
Task Difference: Sentence Classification gradients showed positive alignment with alignment objectives, explaining why sentence-level tasks often tolerate or benefit from alignment, whereas token-level tasks suffer.
Language Factors: Alignments involving English (e.g., de-en, en-kk) showed stronger task separation (conflict) compared to non-English pairs (e.g., ja-kk), suggesting English-centric alignment might introduce specific interference.

Fine-tuning Strategy

Linear vs. Full: Linear-only fine-tuning yielded more consistent improvements across tasks. Full model fine-tuning often reversed gains because updating the entire encoder amplified the conflict between the alignment and task gradients.

5. Significance and Implications

Rethinking Alignment Evaluation: The community should stop relying solely on embedding similarity metrics (like cosine distance or recall) to judge the success of cross-lingual alignment. These metrics fail to capture the optimization dynamics relevant to specific downstream tasks.
Objective Design: Simply adding a contrastive alignment loss is insufficient. The design of the contrastive objective must be carefully matched to the representational level of the downstream task (token vs. sentence).
Training Strategy: For practitioners, the paper suggests that if alignment tuning is performed, freezing the backbone and only fine-tuning the classification head is a safer strategy to avoid catastrophic forgetting or gradient conflict.
Theoretical Insight: The findings highlight that cross-lingual transfer is not a monolithic process; it is governed by a complex interplay of linguistic relatedness, loss design, and the specific optimization landscape of the target task.

In conclusion, the paper argues that "better" alignment is not a silver bullet. Without careful consideration of gradient compatibility and task-specific requirements, aggressive alignment can be counterproductive for encoder-based models.