💬 NLP

Rethinking the Multilingual Reasoning Gap with Layer Swap

This paper demonstrates that the performance gap between native-language and English-pivoted reasoning in multilingual LLMs is significantly smaller than previously thought when using comparable supervision, and proposes a "Layer Swap" technique that transfers English reasoning capabilities from middle layers to native specialists, effectively closing the gap while preserving the target language for chain-of-thought generation.

Original authors: Maxence Lasbordes, Amélie Chatelain, Djamé Seddah

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maxence Lasbordes, Amélie Chatelain, Djamé Seddah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "English Middleman"

Imagine you ask a brilliant, multilingual student (an AI) a complex math problem in French. Even though you asked in French, the student instinctively starts thinking, "Hmm, let me solve this..." in their head using English. They do all the hard reasoning in English, and only switch back to French for the final answer.

This is called English-pivoted reasoning. It works well, but it has downsides:

Lost in Translation: If the student makes a small translation error while switching back and forth, the final answer might be wrong.
Hard to Understand: If you don't speak English, you can't see how they solved the problem because their "thought process" (Chain-of-Thought) is in a language you don't know.

Researchers tried to force the student to think entirely in French (Native Reasoning). However, previous studies showed that when they did this, the student got significantly worse at solving problems. It was like forcing a master chef to cook a French dish using only French recipes, but they kept messing up the ingredients because they were used to English measurements.

The New Experiment: A Fairer Test

The authors of this paper wanted to see if the "Native Reasoning" gap was real or just because the students weren't trained enough. They decided to run a massive, fair experiment:

The Students: They took a smart base AI (Qwen3-8B) and trained it to be a specialist in six languages: English, French, German, Spanish, Chinese, and Swahili.
The Training: They gave each specialist a huge amount of practice data (about 10 billion words worth) so everyone got a fair shot.
The Comparison: They compared two groups:
1. The English Thinkers: These students solved problems in English but answered in the local language.
2. The Native Thinkers: These students solved and thought entirely in the local language.

The Surprise Result:
When everyone got the same high-quality training, the gap between the two groups shrank dramatically. The "Native Thinkers" were almost as good as the "English Thinkers" (only 2–3% worse on average). This suggests that with enough practice, AI can think in its native language without losing its smarts.

The Secret Sauce: The "Layer Swap"

Even though the gap was small, the "English Thinkers" were still slightly better at the hardest math and science puzzles. The researchers wanted to know: Why?

They looked inside the AI's "brain" (its neural network layers) and found a fascinating structure:

The Outer Layers (Top and Bottom): These are like the ears and mouth. They handle the specific language (hearing French, speaking French). These parts are very different between languages.
The Middle Layers: These are like the logic center. The researchers found that when the AI learns to solve math or science problems, the "logic center" looks almost identical whether the AI is speaking French, Chinese, or English. It's a universal reasoning engine.

The Solution: The "Layer Swap"
Since the middle "logic" part of the English specialist was slightly sharper, the researchers performed a "surgery." They took the middle layers from the English specialist and swapped them into the Native specialists.

The Analogy: Imagine you have a French chef who is great at cooking but slightly slow at chopping vegetables. You take the "chopping hands" (the middle layers) from a world-class English chef and attach them to the French chef.
The Result: The French chef can now chop vegetables just as fast as the English chef, but they still speak French and cook French dishes. The AI kept thinking in French (preserving the "thought process" in the local language) but gained the reasoning power of the English model.

The Final Outcome

By doing this "Layer Swap," the researchers closed 80–90% of the remaining performance gap for languages like French and German. For Spanish, they completely closed the gap. The AI could now:

Think in the user's language (making it easier to understand).
Solve hard problems just as well as if it were thinking in English.

Summary

The paper shows that the idea that "AI must think in English to be smart" is mostly a training issue, not a fundamental limit. By training AI properly and then swapping in the "smartest middle brain" from an English model, we can create AI that reasons perfectly in any language, keeping the thinking process transparent and accessible to everyone.

Technical Summary: Rethinking the Multilingual Reasoning Gap with Layer Swap

Problem Statement

Recent Large Language Models (LLMs) with reasoning capabilities predominantly generate Chain-of-Thought (CoT) traces in English, even when prompted in non-English languages. This "English-pivoted reasoning" regime (where the model reasons in English and answers in the input language) is the default for open reasoning models. While constraining the CoT to the input language ("native reasoning") is desirable for interpretability and cultural nuance, prior studies suggest it substantially degrades performance.

However, previous measurements of this "native reasoning gap" often rely on inference-time interventions or fine-tuning on limited native-language datasets, failing to match the scale of English-pivoted post-training. It remains unclear whether the gap persists when native reasoning is trained under strictly comparable, large-scale supervision.

Methodology

Data Construction

The authors constructed a large-scale multilingual reasoning dataset spanning six languages: English, French, German, Spanish, Chinese, and Swahili.

Source: The dataset was derived from allenai/DolciThink-SFT-32B, a decontaminated English post-training dataset covering mathematics, code, science, and general knowledge.
Scale: Approximately 500,000 samples per language, with context lengths up to 32,768 tokens.
Translation Pipeline: To create native corpora, the English source was translated into the five target languages using google/gemma-3-27b-it. To mitigate long-context degradation and reasoning-trace loss, the authors translated samples component-wise (question, reasoning trace, answer) in ~2k-token chunks, followed by rigorous filtering for empty outputs, compression anomalies, and length deviations.

Experimental Setup

Base Model: All models were fine-tuned from Qwen/Qwen3-8B-Base.
Training Regimes: For each non-English language, two specialists were trained on the same question-answer pairs with matched token budgets (~10–11 billion tokens per language):
1. Native Specialist: CoT generated in the target language.
2. English-Pivoted Specialist: Questions and answers in the target language, but CoT generated in English.
Infrastructure: Training utilized distributed H100 nodes with DeepSpeed ZeRO-3, Ulysses sequence parallelism, and FlashAttention-3 to handle 32k sequence lengths.
Benchmarks: Evaluation covered four domains: Mathematics (MGSM-Rev2, AIME 24/25), Science (GPQA-Diamond), General Knowledge (Global-MMLU-Lite), and Code (HumanEvalPlus).

Layer Swap Technique

The authors introduced a training-free method called Layer Swap, inspired by prior work on model composition.

Weight-Space Analysis: The authors analyzed the fine-tuning updates ( $\Delta\theta$ ) across layers. They found that updates in the middle layers (specifically layers 13–22) were highly aligned across all six languages (high cosine similarity and low SVD variance), suggesting a "language-agnostic reasoning core." Conversely, early and late layers showed language-specific divergence.
Mechanism: The method transfers the middle layers (the "reasoning core") from the English specialist into the native specialist, while retaining the native specialist's early and late layers (which handle language-specific input/output).
Ablation: A layer-range ablation on the French specialist determined that swapping layers 13–22 maximized reasoning performance while maintaining ~100% language fidelity (i.e., the CoT remained in French). For Chinese, the optimal window was slightly earlier (13–20) to prevent reversion to English.

Key Results

The Native Reasoning Gap Under Matched Supervision

Contrary to prior reports suggesting large gaps, the authors found that under strictly matched supervision (same data volume, same base model), the native reasoning gap is significantly smaller:

Average Gap: The performance gap between native and English-pivoted specialists shrinks to 1.9%–3.5% across the five non-English languages.
Distribution: The gap is concentrated in complex reasoning benchmarks (AIME 24/25), where native specialists trail by 5–12% depending on the language. On other benchmarks (e.g., HumanEvalPlus), the gap is negligible.
Resource Tier: High-resource languages (French, German, Spanish, Chinese) show gaps of 1.9–3.1%, while the low-resource Swahili shows a slightly larger gap of 3.5%.

Effectiveness of Layer Swap

Transferring the English specialist's middle layers into native specialists yielded significant improvements while preserving the target-language CoT:

Gap Closure: The Layer Swap closed 83–89% of the native reasoning gap for French and German, 60% for Swahili, and 27% for Chinese. For Spanish, the swapped model matched the English-pivoted ceiling.
Language Fidelity: The swapped models maintained ~100% fidelity, generating CoT in the target language without reverting to English.
Benchmark Performance: The method led on HumanEvalPlus across all languages and MGSM-Rev2 in most. The remaining performance deficit was primarily on reasoning-intensive tasks (AIME, GPQA).

Input Understanding Ablation

The authors isolated the "understanding gap" by evaluating native specialists on English inputs versus native inputs.

Finding: Native specialists scored higher on English inputs despite never seeing English during SFT.
Cause: This is attributed to pretraining data imbalances and the fact that intermediate representations in English-centric base models are more aligned with English spaces for high-resource Latin-script languages than for typologically distant or low-resource languages (like Swahili).

Significance and Contributions

The paper claims four primary contributions:

Dataset Release: A publicly available long-CoT reasoning corpus across six languages (including Swahili and Chinese) with 32k context and target-language CoTs.
Controlled Measurement: A large-scale, strictly controlled measurement of the native-vs-English-pivoted gap, demonstrating that the gap is much smaller (1.9–3.5%) than previously reported when supervision scales are matched.
Layer Swap: A training-free technique that exploits the discovery of a language-agnostic reasoning core in the middle layers. This method closes the majority of the remaining gap while preserving native CoT generation.
Input Understanding Analysis: An ablation showing that input understanding remains a primary bottleneck for non-English reasoning, distinct from the reasoning process itself.

Limitations

The authors note several limitations:

The study is limited to a single model family (Qwen) and scale (8B).
The corpus relies on machine translation, which may introduce artifacts.
Tokenization efficiency varied across languages, meaning the "matched token budget" did not strictly equate to matched example counts for all languages.
Hyperparameter sweeps for the Layer Swap window were not exhaustive.
The models were trained solely with Supervised Fine-Tuning (SFT) and did not incorporate Reinforcement Learning (RL) or preference tuning.

The authors conclude that the "recipe" for Layer Swap is directly applicable to other model families, as the required English specialist is often available off-the-shelf, offering a practical path to high-performance native reasoning without retraining the entire model.