💬 NLP

ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection

The paper introduces ExpLang, a novel post-training pipeline that enhances Large Reasoning Models by enabling on-policy thinking language selection during reinforcement learning, thereby improving both exploration and exploitation through multilingual advantages while outperforming English-only training.

Original authors: Changjiang Gao, Zixian Huang, Kaichen Yang, Jiajun Chen, Jixing Li, Shujian Huang

Published 2026-02-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Changjiang Gao, Zixian Huang, Kaichen Yang, Jiajun Chen, Jixing Li, Shujian Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Giving AI a "Multilingual Brain"

Imagine you are trying to solve a really hard math puzzle. You have a brilliant assistant (the AI), but they only speak English. They are smart, but sometimes, when they get stuck, they just keep spinning their wheels in English, over and over again.

ExpLang is a new training method that teaches this assistant to switch languages when they need to think. It turns out that for some problems, thinking in Spanish, Chinese, or Vietnamese might actually help the AI find the solution faster or more creatively than sticking to English.

The paper argues that by letting the AI choose which language to think in, we can make it smarter, faster, and more compliant with users who speak different languages.

The Problem: The "English-Only" Trap

Currently, most advanced AI models are trained mostly on English data. They are like a chef who only knows how to cook with one specific set of spices.

The Issue: If you ask them to solve a problem, they default to English. If English isn't the best "tool" for that specific puzzle, they might struggle.
The Old Way: Researchers tried to force the AI to speak other languages by adding a "prefix" (like telling it "Now speak French"). But this was clunky and unstable, like putting a fake mustache on a dog; it looks like a mustache, but the dog doesn't actually be a mustache-wearer.

The Solution: ExpLang (Explore & Exploit)

The authors created a training pipeline called ExpLang. Think of it as a two-phase boot camp for the AI's brain.

Phase 1: The "Exploration" Stage (The Traveler)

The Analogy: Imagine the AI is a traveler in a new city. In the beginning, they don't know which street leads to the treasure. So, they wander around randomly, trying different neighborhoods (languages).
What happens: The AI is rewarded for trying different languages. If it usually thinks in English, the system says, "Hey, try thinking in Vietnamese today!"
The Goal: To cast a wide net. Maybe the solution to a geometry problem is clearer in Japanese logic, or a logic puzzle clicks better in German. By forcing the AI to try many languages, it discovers new ways to solve problems that it would have missed if it stayed in English.

Phase 2: The "Exploitation" Stage (The Pro)

The Analogy: After the traveler has explored the city, they realize that "Neighborhood A" (English) and "Neighborhood B" (Chinese) are the only ones with the treasure. Now, they stop wandering and focus entirely on the best routes.
What happens: The AI looks back at all the solutions it found. It realizes, "Wow, when I thought in English, I got the answer right 90% of the time. When I thought in French, I got it right 60%."
The Goal: The AI learns to pick the best language for the specific problem. It doesn't stop speaking other languages, but it becomes a master at knowing when to use which one.

Why This is a Game-Changer

1. It's "On-Policy" (The AI Chooses)
In older methods, humans forced the AI to speak a language. In ExpLang, the AI chooses the language itself as part of its decision-making process.

Analogy: It's the difference between a teacher forcing a student to write an essay in French vs. a student realizing, "I understand this concept better in French, so I'll write it in French." The latter leads to better learning.

2. It's "Orthogonal" (Plug-and-Play)
The paper mentions that this method works with almost any existing AI training algorithm.

Analogy: ExpLang isn't a new car engine; it's a new GPS system you can install in any car. You don't need to rebuild the whole car to get the benefit of better navigation.

3. The "English-Multilingual-English" Journey
The paper found a fascinating pattern in how the AI's brain changed:

Start: It only speaks English.
Middle: It goes crazy, trying 12 different languages (Exploration).
End: It settles back down, but now it's smarter. It mostly speaks English again, but it has "absorbed" the logic and efficiency it learned from the other languages.

Result: The final model is better at English reasoning than a model that never tried other languages, because it learned from the diversity.

The Results: What Did They Find?

Smarter AI: The ExpLang models solved hard math problems better than models trained only in English, even with the same amount of computing power.
Better Compliance: If you ask the AI to "Think in Vietnamese," it actually does it (99% of the time), whereas other models often ignore you and speak English anyway.
Generalization: Even if you ask the AI to think in a language it wasn't explicitly trained on (like a rare language), it still does a great job. It learned the skill of switching languages, not just the specific words.

Summary

ExpLang is like teaching a genius student to be a polyglot. Instead of forcing them to stick to one language, you let them experiment with many. By the end of the training, they have a "super-brain" that knows the best way to think for any given problem, making them faster, more accurate, and much more helpful to people all over the world.

1. Problem Statement

Current Large Reasoning Models (LRMs) have demonstrated exceptional performance on challenging tasks (e.g., math reasoning) after Reinforcement Learning with Verifiable Rewards (RLVR) post-training. However, existing research and training pipelines predominantly focus on English-only reasoning. This limitation stems from:

Data Bias: Training data is English-dominant, leading models to perform best in English.
The "Curse" View: Multilingual reasoning is often viewed as a hindrance to performance or merely a user-service feature rather than a capability enhancer.
Missed Potential: Preliminary evidence suggests that multilingual Chain-of-Thought (CoT) can offer a larger exploration space and higher expected returns, but current models lack the ability to self-select the optimal thinking language during the reasoning process.
User Needs: Global users require native thinking traces for better readability and trust, which English-only models fail to provide consistently.

2. Methodology: ExpLang

The authors propose ExpLang, a novel post-training pipeline that enables on-policy thinking language selection. Instead of forcing a specific language or using static prefixes, the model learns to choose its thinking language as an action within the RL framework. The pipeline consists of three key building blocks:

A. High-Quality Multilingual Data Synthesis

Teacher-Student Distillation: A larger variant of the base model (e.g., Qwen3-32B) generates multilingual reasoning traces for an English math dataset, guided by 12 non-English prefixes (6 European, 6 Asian).
Filtering: Generated traces are filtered using LangDetect for language compliance and HuggingFace Math-Verify for answer correctness.
Tagging: Each sample is tagged with a specific format <lang_select>[lang]</lang_select> before the thinking trace to teach the model explicit language control.

B. Cold-Start via LoRA SFT

Goal: Align the base model (e.g., Qwen3-4B) with the language selection tags without causing catastrophic forgetting of its original reasoning capabilities.
Technique: Supervised Fine-Tuning (SFT) is performed using LoRA (Low-Rank Adaptation) rather than full fine-tuning. This reduces parameter changes, preserving the model's mathematical reasoning while learning to adhere to language selection tags.

C. RLVR with Two-Stage On-Policy Selection

The core innovation is a two-stage RL training process where language selection is an action optimized by the reward function:
$r = \lambda_f r_f + \lambda_c r_c + \lambda_d r_d + \lambda_p r_p + \lambda_v r_v$

Exploration Stage (First 25% of steps):
- Objective: Maximize diversity and extend the action space.
- Reward: A Language Diversity Reward ( $r_d$ ) is applied to encourage the model to select under-sampled languages.
- Mechanism: $r_d$ is higher for languages with fewer responses in the current batch, forcing the model to try different languages.
- KL Loss: Disabled to prevent premature convergence and encourage output diversity.
Exploitation Stage (Remaining 75% of steps):
- Objective: Leverage the best-performing languages found during exploration.
- Reward: A Language Pass@k Reward ( $r_p$ ) is introduced. For correct answers, all other responses in the batch sharing the same thinking language are rewarded. This encourages the model to converge on the most efficient language(s) for a given problem.
- KL Loss: Re-enabled to stabilize the policy and slow down language concentration.

3. Key Contributions

On-Policy Language Selection: Unlike previous "prefix-hacking" or forced-selection methods, ExpLang allows the model to dynamically choose the thinking language as part of its reasoning policy.
Two-Stage RL Strategy: The separation into an exploration phase (diversity-driven) and an exploitation phase (efficiency-driven) effectively balances the trade-off between discovering new reasoning paths and refining the best ones.
Orthogonality: The method is compatible with most existing RL algorithms (e.g., GRPO, DAPO, GSPO) as it primarily modifies the reward function and action space.
First Unified Pipeline: This is the first work to simultaneously achieve high reasoning performance, strict language compliance for seen languages, and generalization to unseen languages within a single training pipeline.

4. Experimental Results

The method was evaluated on the Qwen3-4B model using the OpenR1-Math-220k dataset and tested on MATH-500, AIME 2025, and Olym-MATH.

Performance vs. English-Only: ExpLang consistently outperforms English-only baselines (both Naive GRPO and Controlled SFT+GRPO) with the same training budget.
- MATH-500: Achieved 91.5% accuracy (vs. 91.4% for Naive) and 96.8% Pass@k.
- AIME 2025: Achieved 31.9% accuracy (vs. 30.8% for Naive) and 53.3% Pass@k.
Language Compliance:
- Seen Languages: Achieved near-perfect compliance (~99.9%) when forced to use specific non-English languages, significantly outperforming baselines which often reverted to English.
- Unseen Languages: Demonstrated strong generalization to low-resource languages (Indonesian, Hebrew, Romanian, Swahili) without specific training data for them, maintaining high compliance and accuracy.
Efficiency: The model reduced the number of thinking tokens required to solve problems while maintaining or improving accuracy.
Behavioral Analysis:
- "English-Multilingual-English" Pattern: The model initially shifted to multilingual thinking during the exploration stage (depressing English preference) and then converged back to English dominance during the exploitation stage, suggesting English was the most efficient vehicle for the learned reasoning capabilities.
- Non-English Advantage: Analysis showed that for a subset of problems, non-English thinking yielded higher accuracy than English. ExpLang successfully leveraged these "winning" non-English traces to improve the overall English reasoning capability (cross-lingual transfer).

5. Significance and Conclusion

ExpLang fundamentally shifts the paradigm of multilingual LLM reasoning from a "service feature" to a performance-enhancing mechanism.

Enhanced Exploration: By allowing the model to think in multiple languages, it accesses a richer semantic space, leading to better problem-solving strategies.
Global Accessibility: It enables models to provide reasoning traces in the user's native language without sacrificing the high performance typically associated with English-only training.
Future Impact: The approach is scalable and compatible with advanced RL algorithms, offering a new perspective on how multilingualism can be leveraged to build more robust, efficient, and globally accessible Large Reasoning Models.