💬 NLP

Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfer

This paper demonstrates that cross-lingual transfer in large language models is driven primarily by task-format alignment rather than linguistic relatedness, as models show uniform improvements across language families regardless of their baseline performance or the use of chain-of-thought reasoning.

Original authors: Ahmed Haj Ahmed, Ruochen Zhang, Alvin Grissom II

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmed Haj Ahmed, Ruochen Zhang, Alvin Grissom II

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of seven different students (the AI models), ranging from a bright but inexperienced high schooler to a seasoned professor. You want to see if teaching them a specific subject in Arabic helps them suddenly become experts at answering questions in Hebrew, Amharic, and Maltese (languages related to Arabic), or if it just helps them answer questions in Japanese, Korean, and French (unrelated languages) equally well.

The researchers' big discovery is this: Teaching them Arabic didn't magically transfer "Arabic knowledge" to the related languages. Instead, it just taught them how to take the test.

Here is the breakdown using simple analogies:

1. The Setup: The "Language Family" Test

The researchers picked the Semitic language family (Arabic, Hebrew, Amharic, Maltese) because they are like cousins. They share deep family roots, even though they look different on paper (Arabic and Hebrew use a script like a skeleton of letters, Amharic uses a different script, and Maltese uses the standard Latin alphabet we use in English).

The idea was: If we teach a model Arabic, will it naturally get better at Hebrew or Amharic because they are "family"?

2. The Experiment: The "Arabic Crash Course"

They took seven large AI models and gave them a "crash course" (fine-tuning) using only Arabic dialects. They didn't teach them anything about the other languages. Then, they tested the models on reading comprehension questions in all the other languages without any further training (zero-shot).

3. The Results: It Was All About "Test-Taking Skills"

The results were surprising.

The "Weak" Models (The Struggling Students): The models that started out doing poorly (like the GPT-OSS models) got a massive boost after the Arabic training. They jumped from getting 45% of answers right to nearly 80%.
- The Twist: They improved just as much on Japanese and French as they did on Hebrew and Amharic.
- The Analogy: Imagine a student who doesn't know how to fill out a bubble sheet correctly. You teach them how to fill out the sheet using an Arabic practice test. Suddenly, they get better at every test, even the ones in Japanese, just because they finally learned how to mark the bubbles correctly. They didn't learn Japanese; they learned the format.
The "Strong" Models (The Top Students): The models that were already doing very well (like the 671-billion-parameter DeepSeek) barely improved at all.
- The Analogy: If a student is already an A+ student who knows exactly how to fill out the bubble sheet, giving them more practice on the sheet doesn't help them much. They were already calibrated.

4. The "Chain-of-Thought" Proof

To prove this wasn't about learning new language facts, the researchers tried a different trick: Chain-of-Thought (CoT). Instead of retraining the models, they just asked them to "think out loud" (write a reasoning step) before answering.

The Result: The exact same models that got better from the Arabic training also got better from "thinking out loud."
The Conclusion: If the Arabic training had transferred "linguistic knowledge" (like grammar rules), thinking out loud shouldn't have helped as much. But since both methods helped the same models by the same amount, it proves the problem wasn't a lack of knowledge. The problem was that the models didn't know how to align their answers with the test format.

5. The Script Myth

The researchers also checked if the writing system mattered. Since Arabic and Hebrew both use a script that looks like a skeleton of letters (abjad), they thought maybe Hebrew would get a special boost.

The Reality: Hebrew did not get a special boost. The improvement was the same whether the language used the same script as Arabic or a completely different one. The "family tree" connection didn't matter.

The Bottom Line

The paper concludes that when we see an AI get better at a language after training on a related one, we shouldn't assume it's because the AI "learned the language family."

Instead, the AI was likely just learning the rules of the game. It was learning how to look at a question, process the options, and pick the right answer number (1, 2, 3, or 4). Once it learned that "game format" using Arabic, it could apply that same skill to any language, whether it was a cousin (Hebrew) or a stranger (Japanese).

In short: The models didn't learn to speak Hebrew; they learned how to take the test.

Technical Summary: Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfer

Problem Statement

While Large Language Models (LLMs) have advanced multilingual capabilities, the mechanisms driving cross-lingual transfer remain unclear. It is often assumed that fine-tuning a model on a high-resource language (e.g., Arabic) yields performance gains in linguistically related low-resource languages (e.g., Hebrew, Amharic) due to shared morphological or syntactic structures. However, prior work suggests that factors like model scale and instruction tuning alone are insufficient to explain transfer behavior, and the "curse of multilinguality" may dilute performance in fixed-capacity models.

This paper investigates whether observed gains in zero-shot reading comprehension after fine-tuning on Arabic are driven by linguistic relatedness (transfer of Semitic-specific knowledge) or task-format alignment (learning how to structure outputs for the evaluation task). The authors posit that if transfer is linguistic, gains should be concentrated in Semitic languages; if it is alignment-based, gains should be uniform across all languages, regardless of family or script.

Methodology

Experimental Design

The study employs a controlled cross-lingual transfer setup using the Belebele benchmark, a massively parallel multiple-choice reading comprehension dataset spanning 120+ languages.

Source Language: Fine-tuning is performed exclusively on Arabic dialects (Mesopotamian, Tunisian, South Levantine, North Levantine, Najdi, and Moroccan), totaling 4,800 examples.
Target Languages: Zero-shot evaluation is conducted on:
- Semitic Targets: Hebrew, Amharic, Maltese.
- Non-Semitic Controls: Japanese, Korean, French.
Strict Control: The dataset uses identical question identifiers across languages, ensuring no train-test overlap in content, only in format.

Models

Seven multilingual LLMs were evaluated, varying in scale (4B to 671B parameters) and architecture:

Dense Architectures: Qwen3-4B-Instruct, Qwen3-8B, Qwen3-8B-Base, Qwen3-32B.
Mixture-of-Experts (MoE): GPT-OSS-20B, GPT-OSS-120B, DeepSeek-V3.1 (671B).

Training and Evaluation Protocol

Fine-tuning: Supervised fine-tuning (SFT) using LoRA (Rank 32) on Arabic dialect data. The training format mimics the evaluation prompt exactly (Arabic instruction, target language content, single-digit answer). No Chain-of-Thought (CoT) annotations were provided during training.
Evaluation: Zero-shot accuracy measured via logprob scoring. The model is prompted with the passage and question, followed by "ANSWER: ", and the log-probability of tokens '1' through '4' is computed. The digit with the highest log-probability is selected.
Ablation Study: A Chain-of-Thought (CoT) reasoning ablation was conducted on baseline (non-fine-tuned) models. Models were asked to generate a reasoning trace before scoring the answer, allowing the authors to compare the effects of parameter updates (fine-tuning) against inference-time reasoning.

Key Contributions

Controlled Cross-Lingual Study: The authors present a rigorous study across the Semitic family that jointly varies linguistic relatedness, script (Abjad, Abugida, Latin, CJK), model scale, and architecture (Dense vs. MoE) on parallel data.
Task-Format Alignment vs. Linguistic Transfer: The paper demonstrates that observed performance gains are attributable to task-format alignment rather than language-family transfer. Regression analysis controlling for baseline accuracy shows that Semitic languages do not gain more than non-Semitic controls; in fact, they gain slightly less once baseline performance is accounted for.
CoT Reasoning Ablation: The authors introduce a CoT ablation showing that models benefiting most from fine-tuning are the same ones that benefit most from inference-time reasoning. This suggests both mechanisms address the same underlying deficit: task-format calibration.

Results

1. Uniform Gains Across Language Families

Weak Baseline Models (MoE): GPT-OSS-120B and GPT-OSS-20B showed dramatic improvements after Arabic fine-tuning (average gains of +33.5 pp and +22.0 pp, respectively). Crucially, these gains were uniform across all target languages. For GPT-OSS-120B, gains on non-Semitic languages (French +33.0 pp, Korean +32.0 pp) were comparable to Semitic languages (Hebrew +44.0 pp, Maltese +41.0 pp).
Strong Baseline Models: DeepSeek-V3.1 (671B) and Qwen3-32B, which started with high baseline accuracy, showed only marginal improvements (+2.3 pp and +1.0 pp, respectively) regardless of language family.
Small Dense Models: Models like Qwen3-8B showed near-zero transfer, with some languages experiencing slight degradation, suggesting LoRA fine-tuning on Arabic data provided insufficient signal for these scales.

2. Regression Analysis

A regression analysis of fine-tuning gains against baseline accuracy and a binary "Semitic" indicator revealed:

Baseline accuracy alone explained ~48% of the variance in gains ( $R^2 = 0.479$ ).
Adding the Semitic indicator yielded a negative coefficient ( $\beta_{Semitic} = -8.2$ pp), indicating that after controlling for baseline accuracy, Semitic languages actually gained less than non-Semitic controls. This contradicts the hypothesis of linguistic transfer.
Script similarity (Abjad vs. others) was also found to be non-significant in predicting transfer.

3. Reasoning Ablation

The CoT ablation confirmed that the models improving most with fine-tuning (GPT-OSS models) also improved most with CoT reasoning.

GPT-OSS-120B gained +33.5 pp from fine-tuning and +41.0 pp from CoT.
GPT-OSS-20B gained +22.0 pp from fine-tuning and +22.8 pp from CoT.
Strong models (DeepSeek-V3.1) saw negligible or negative gains from CoT.
Interpretation: Both fine-tuning and CoT compensate for a lack of task-format calibration. Weak models possess the knowledge but fail to express it via direct logprob scoring; both mechanisms help them surface this knowledge.

4. Distributional Calibration

Analysis of answer-choice logprob distributions showed that:

Weak models exhibited high entropy (uncertainty) at baseline. Both fine-tuning and CoT reduced entropy and increased the probability mass on the correct answer ( $\Delta P^* > 0$ ).
Strong models already had peaked distributions; fine-tuning sometimes increased entropy (diffusing the distribution) without improving accuracy, indicating a "catastrophic forgetting" or interference effect in well-calibrated small models.

Significance and Claims

The paper argues that Arabic fine-tuning does not induce Semitic-specific cross-lingual transfer in the context of multiple-choice reading comprehension. Instead, the primary mechanism is task-format alignment.

Methodological Caution: The authors claim that gains observed on multilingual benchmarks should not be uncritically attributed to cross-lingual knowledge transfer without controlling for task-format alignment. What appears to be "transfer" may simply be the model learning the specific output format of the evaluation task.
Implications for Low-Resource Strategies: Practitioners seeking to improve performance on low-resource Semitic languages via Arabic fine-tuning should not expect a special advantage from language-family relatedness. The more reliable lever is improving general task-format alignment.
Architecture vs. Language: The study suggests that architectural paradigm (MoE vs. Dense) and baseline capability are stronger predictors of fine-tuning benefits than the linguistic properties of the target language.

The authors conclude that for the specific paradigm of logprob-scored multiple-choice reading comprehension, the bottleneck for weaker models is not a lack of linguistic knowledge, but a lack of alignment with the evaluation format. This finding inverts the "alignment tax" narrative, suggesting that insufficient format alignment—not insufficient knowledge—is the primary barrier for weaker models.