Technical Summary: LANG – Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance

1. Problem Statement

While Reinforcement Learning with Verifiable Rewards (RLVR) has successfully enhanced multi-step reasoning in Large Language Models (LLMs) for English, these benefits have not fully translated to multilingual contexts. Current models face a fundamental trade-off:

• Language Consistency vs. Reasoning Quality: Explicitly constraining models to reason in the input language often leads to significant performance degradation in non-English settings, as the model's strong English reasoning capabilities do not reliably transfer.
• Language Drift: When language constraints are relaxed to preserve reasoning quality, models frequently drift toward English, reducing readability and interpretability for native users.

Existing approaches, such as prompt-based constraints or adding language-consistency rewards to RL, struggle with reward sparsity. In multilingual scenarios, especially for low-resource languages, trajectories that satisfy strict language and correctness constraints are extremely scarce, hindering effective exploration during training.

2. Methodology: LANG

The authors propose LANG, a language-adaptive hint-guided RL framework designed to bootstrap exploration in non-English reasoning tasks while preventing dependency on external aids. The framework consists of two core mechanisms:

A. Scheduled Multilingual Hint Decay

LANG utilizes language-conditioned hints (partial reasoning traces from a teacher model) to guide the student model during early training. To address the issue of "hint dependence" (where models fail to reason autonomously once hints are removed), LANG employs a cosine annealing decay schedule.

• Mechanism: At training step $t$, a prefix of the hint $h$ is injected into the prompt. The ratio of the hint injected, $p_t^l$, follows a cosine schedule that starts at 1.0 (full guidance) and smoothly decays to 0.0.
• Goal: This transitions the model from hint-conditioned rollouts to autonomous multilingual reasoning, narrowing the training-inference discrepancy.

B. Language-Adaptive Switch

Recognizing that learning difficulties vary across languages, LANG avoids a single global switch point for removing hints. Instead, it categorizes languages into resource groups (High, Medium, Low) and employs a dynamic switching strategy.

• Mechanism: The system monitors an effective-update rate ($\bar{u}_R(t)$), defined as the fraction of instances in a language group's batch that yield at least one trajectory with a positive advantage.
• Switching Criterion: When $\bar{u}_R(t)$ exceeds a threshold $\tau$, the hint injection for that specific language group is turned off.
• Benefit: Low-resource languages retain hints longer to alleviate reward sparsity, while high-resource languages switch to autonomous reasoning earlier, optimizing the learning pace for each language's specific difficulty.

C. Policy Optimization

The framework optimizes the policy using GRPO (Group Relative Policy Optimization). The reward function is a conjunction of three criteria:

1. Language Consistency ($R_{lc}$): The reasoning trace and final answer must match the input language.
2. Format ($R_{format}$): Correct use of thinking tags and boxed answers.
3. Accuracy ($R_{acc}$): The final answer must match the ground truth.

3. Key Contributions

• Novel Framework: Introduction of LANG, which leverages language-conditioned hints to bootstrap exploration in non-English reasoning, addressing the reward sparsity issue inherent in multilingual RLVR.
• Training-Inference Alignment: The development of a progressive hint-decay schedule and a language-adaptive switch that effectively mitigates the training-inference discrepancy, ensuring models can reason autonomously at test time without relying on hints.
• Empirical Validation: Extensive experiments demonstrating that LANG substantially enhances multilingual reasoning performance without compromising language consistency, outperforming both prompting-based and training-based baselines.
• Generalization: Evidence that the method generalizes beyond mathematical reasoning to other multilingual tasks (e.g., MMLU-ProX, XWinograd) and maintains language consistency across different model layers.

4. Experimental Results

The authors evaluated LANG on two challenging benchmarks: MMATH (10 languages) and PolyMath (18 languages), using Qwen2.5 (3B, 7B, 32B) and Llama3.1-8B models.

• Performance Gains: LANG significantly outperformed the strongest baseline (LC-GRPO). On the MMATH benchmark, it improved the Language Consistency & Accuracy (LC&Acc) metric by 24.1% on average across models. On PolyMath, it achieved an 18.7% improvement.
• Low-Resource Languages: The gains were particularly pronounced for low-resource languages. For instance, with Qwen2.5-7B, LANG improved LC&Acc by 39.0% on Thai (MMATH) and 24.6% on Vietnamese (PolyMath) compared to mGRPO.
• Out-of-Distribution Generalization: LANG demonstrated strong generalization to out-of-domain languages and different model scales, consistently improving LC&Acc over baselines.
• Layer-wise Analysis: Analysis of intermediate hidden states revealed that LANG maintains high language consistency throughout the model's layers, preventing the "cross-layer language switching" phenomenon where models reason in English and translate only at the output.

5. Significance and Claims

The paper claims that LANG provides a robust solution to the trade-off between reasoning accuracy and language consistency in multilingual LLMs. By using hints as temporary scaffolding that is progressively removed based on language-specific learning progress, the framework enables models to internalize multilingual reasoning patterns.

The authors emphasize that their approach is teacher-agnostic (performing well with hints from different teacher models) and translation-robust (gains remain stable regardless of the translation model used for data construction). They conclude that LANG effectively bridges the gap between English-centric reasoning capabilities and the need for native-language interpretability, fostering more consistent language alignment across model layers without sacrificing performance.

The work acknowledges limitations, noting that it currently relies on a single teacher model (DeepSeek-R1) for hint generation and that performance trade-offs across languages may stem from pre-training data imbalances, suggesting future work in data-centric strategies.