💻 computer science

Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer

This paper introduces the Hardness Adjusted Transfer (HAT) Score to isolate true cross-lingual transfer from general source-language improvements, revealing that while small models transfer effectively and progress has been made over time, scaling models yields slower-than-expected gains in transfer strength.

Original authors: Prasoon Bajpai, Eleftheria Briakou, Colin Cherry, Preethi Jyothi, Vihari Piratla

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Prasoon Bajpai, Eleftheria Briakou, Colin Cherry, Preethi Jyothi, Vihari Piratla

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Faking It" Trap

Imagine you are a teacher grading students from two different countries: Country A (where the teacher speaks the language fluently) and Country B (where the teacher is still learning).

For a long time, researchers measured how well a computer model learned a new language by simply looking at its test scores in that new language. They thought: "If the score goes up, the model is getting better at transferring its knowledge!"

The paper argues this is a trick.

The authors say that often, the model isn't actually getting better at translating its knowledge; it's just getting smarter overall.

The Analogy: Imagine a student who is terrible at math but gets a tutor. The tutor helps them understand the concepts better. Now, when the student takes a test in their native language, they get a 90%. When they take the same test in a foreign language, they get a 60%.
Later, the student gets a better tutor. They understand the concepts even deeper. Now they get a 98% in their native language and a 65% in the foreign language.
The Mistake: If you just look at the foreign language score (60% → 65%), you might think the student improved their language skills. But really, they just got better at the math. The "gap" between their native and foreign scores actually got worse (from 30 points down to 33 points).

The paper says current methods are like that: they confuse "getting smarter generally" with "getting better at cross-language transfer."

The Solution: The "HAT Score" (Hardness Adjusted Transfer)

To fix this, the authors invented a new way to measure progress called the HAT Score.

The Analogy: Imagine a "Perfect Transfer" line. This is a magic line where if a student gets 80% in their native language, they should get 80% in the foreign language if they have truly mastered the transfer.

Old Method: Just looked at the final number (e.g., "65%").
HAT Score: Looks at the relationship. It asks: "Given that the model got X% in the source language, how much higher (or lower) did it perform in the target language compared to what we expected?"

If a model performs above the expected line, it's a true transfer win. If it just follows the line because it got smarter generally, the HAT score stays the same. It filters out the "noise" of general improvement to find the "signal" of true language learning.

What They Found (The Results)

The researchers tested 20 different AI models (from companies like Google, OpenAI, and Anthropic) on three different types of tasks. Here is what their "HAT Score" revealed:

1. Small Models Aren't Broken

The Myth: People thought small AI models were terrible at learning new languages.
The Reality: When you use the HAT Score, small models actually do a decent job! They aren't "broken"; they just have lower overall scores. The old metrics made them look worse than they really were.

2. Progress is Slower Than We Thought

The Myth: As AI models get bigger (more parameters), they magically become perfect at cross-language transfer.
The Reality: Using the HAT Score, the authors found that making models bigger helps, but the improvement is much slower than we hoped. We aren't seeing the massive leaps in language transfer that the raw scores suggested.

3. Time is Helping (But Only on Some Tasks)

The Reality: Newer models (released in 2025/2026) are genuinely better than older ones at reasoning tasks (like math and logic puzzles). On these tests, the HAT Score shows they are almost "solved"—meaning they transfer knowledge almost perfectly.
The Catch: This progress hasn't happened for fact-recall tasks (like answering trivia questions). On those, progress has stalled.

4. The "Script" Barrier

The Reality: Does it matter if the languages use different alphabets? (e.g., English vs. Arabic).
The Finding: It depends on the task.
- For reasoning (math/logic), the alphabet doesn't matter much. The model figures it out.
- For facts (trivia), switching alphabets creates a huge barrier. The model struggles much more when the script changes.

5. "Thinking" Helps

The Reality: Models that are allowed to "think" (generate a chain of reasoning before answering) perform better at cross-language transfer.
The Observation: The models seem to use this extra thinking time to bridge the gap between languages, effectively translating the problem in their "mind" before solving it.

The Conclusion

The paper concludes that while AI is getting better, we have been celebrating "general smarts" as if they were "language skills."

The Good News: We are making real progress on reasoning tasks, and small models are surprisingly capable.
The Bad News: We are making slower progress on fact-based tasks, and the gap between languages is still stubborn, especially when different alphabets are involved.
The Call to Action: The current tests (benchmarks) are too easy for the newest models. We need harder, more complex tests that force models to struggle with multiple steps, where the "translation" part of the problem is the real challenge.

In short: Don't just look at the final score; look at how the model got there. The HAT Score is the new ruler for measuring true language learning.

Technical Summary: "Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer"

Problem Statement

Cross-lingual transfer (XLT) is defined as a model's ability to generalize capabilities from well-represented source languages (typically English) to under-represented target languages. While multilingual benchmarks are commonly used to track progress in this area, the authors argue that existing metrics conflate genuine transfer improvements with general improvements in source language accuracy.

Current standard measures include:

Absolute Target Accuracy: Measuring performance on the target language alone.
XLT Gap: The absolute difference between source and target accuracy.
Transfer Score: The proportion of target questions answered correctly given a correct source prediction.

The paper identifies a critical flaw in these metrics: source and target accuracies are strongly linearly coupled (with $R^2 > 0.94$ ). As general model capabilities (e.g., reasoning, factuality, or scale) improve, both source and target accuracies rise predictably. Consequently, an increase in target accuracy may simply reflect better source performance rather than improved transfer mechanisms. Similarly, the XLT gap can misleadingly suggest worsening transfer as models scale, even when the underlying transfer profile improves.

Methodology

The Hardness Adjusted Transfer (HAT) Score

To address the confounding effect of general source improvements, the authors propose the Hardness Adjusted Transfer (HAT) Score. This metric is derived from first principles to isolate the fraction of performance that translates from source to target, independent of the source's absolute difficulty or the model's general proficiency.

Theoretical Definition:
Let $S$ and $T_\ell$ denote the pass-rates for the source and target language $\ell$ , respectively. The ideal transfer profile is defined by the conditional expectation $E[T_\ell | S=s, M]$ . If a model has perfect transfer, this profile aligns with the line $y=x$ .

The HAT Score is defined as the normalized expected target accuracy under a uniform distribution of source accuracies, effectively removing the bias introduced by a model's specific source accuracy distribution:
$\text{HAT Score} = 2 \int_{0}^{1} E[T_\ell | S=s, M] \, ds$
The multiplier 2 ensures the score is bounded within $[0, 1]$ .

Empirical Estimation:
Since $E[T_\ell | S, M]$ is not directly observable, the authors estimate it using a linear regression approach:

For each dataset instance, the model is prompted $N=10$ times in both source and target languages to estimate pass-rates ( $s^{(i)}$ and $t^{(i)}$ ).
A linear regressor is fitted to the pairs $(s^{(i)}, t^{(i)})$ to approximate the conditional expectation, constrained such that $T=0$ when $S=0$ .
The slope of this fitted line ( $\hat{m}$ ) serves as the HAT Score.

Experimental Setup

The study evaluates 20 diverse Large Language Models (LLMs) across five families (Gemini, Gemma, Qwen, GPT-OSS, Claude), spanning sizes from 1B to 120B parameters and varying "thinking" budgets (reasoning depths).

Evaluation is conducted on three multilingual benchmarks:

ECLeKTic: Cross-lingual factoid QA.
MGSMv2: Multilingual math reasoning (a corrected version of MGSM).
MMLU-ProX-Lite: Multilingual multiple-choice reasoning and conceptual knowledge.

For MGSMv2 and MMLU-ProX-Lite, English is the source language. For ECLeKTic, the source language varies per question based on the original Wikipedia article.

Key Results

1. Validation of HAT vs. Traditional Metrics

The authors demonstrate that traditional metrics often misrepresent progress:

Target Accuracy: Can falsely indicate progress when the transfer profile remains unchanged, simply because the source accuracy distribution has shifted to higher values (e.g., larger models).
XLT Gap: Can falsely indicate regression (worsening transfer) as models scale, even when the transfer profile (slope) improves.
HAT Score: Remains robust to shifts in source accuracy distribution, providing a reliable indicator of true transfer strength.

2. Progress with Model Size

Small Models: Transfer is not "broken" in small models (1B–4B). HAT scores for these models range from 50% to 80%, indicating that target accuracy is a significant fraction of source accuracy. Traditional metrics, however, provide overly pessimistic estimates for small models.
Scaling Trends: While larger models show improved transfer, the progress is slower than expected relative to increases in model size. The HAT score reveals that gains in XLT are more gradual than the raw target accuracy suggests.

3. Progress Over Time

Reasoning Benchmarks: For MGSMv2 and MMLU-ProX-Lite, newer models (post-2025) show a significant shift toward the "perfect transfer" line. The median HAT scores improved from 0.75 to 0.93 (MMLU-ProX-Lite) and 0.90 to 0.97 (MGSMv2), suggesting these benchmarks are nearly solved in terms of XLT.
Factoid Benchmarks: No such trend is observed for ECLeKTic, where progress appears stalled.

4. Factors Influencing Transfer

Model Architecture & Pretraining: Differences in HAT scores across model families of comparable size (e.g., GPT-OSS vs. Gemma vs. Qwen) suggest that pretraining recipes are more influential on transfer than model size alone.
Thinking/Reasoning Budget: Enabling "thinking" (reasoning depth) improves XLT across all benchmarks. The authors observe that models allocate additional tokens for non-English prompts (5–20% longer), suggesting a mechanism for language bridging.
Script Similarity: Transfer is significantly weaker between languages with different scripts only on ECLeKTic (factoid QA). For reasoning tasks (MGSMv2, MMLU-ProX-Lite), script differences do not significantly degrade HAT scores.

Significance and Claims

The paper claims that the field has been misinterpreting progress in cross-lingual transfer due to the coupling of source and target accuracies. By introducing the HAT Score, the authors provide a metric that disentangles general capability improvements from true transfer gains.

The authors conclude that:

Transfer in small models is functional, contrary to pessimistic views derived from raw target accuracy.
Progress in XLT is slower than anticipated with respect to model scaling, challenging the assumption that simply increasing parameters automatically solves multilingual gaps.
Current benchmarks are nearing saturation for reasoning tasks (MGSMv2, MMLU-ProX-Lite), necessitating the development of more challenging, practically motivated benchmarks (e.g., multi-step tasks) to continue measuring meaningful progress.

The work serves as a call to action for the community to adopt more principled metrics like HAT to accurately evaluate and drive research in cross-lingual generalization.