Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models

1. Problem Statement

The paper investigates a critical failure mode in Reinforcement Learning with Verifiable Reward (RLVR) applied to multilingual Large Language Models (LLMs). While RLVR (specifically using Group-Relative Policy Optimization, or GRPO) has successfully elicited strong reasoning capabilities in English, the authors observe a phenomenon they term Cross-lingual Collapse.

• The Phenomenon: When a multilingual model is trained via RLVR on non-English tasks (e.g., math problems in Ukrainian or Korean) requiring long Chain-of-Thought (CoT) reasoning, the model's internal reasoning trace systematically drifts away from the target language and reverts to its pre-training dominant language (typically English).
• The Trade-off: This language switch is not a bug but a feature of the optimization process: the model achieves higher task accuracy by reasoning in English, even when prompted in another language. This creates a persistent trade-off between performance (accuracy) and fidelity (linguistic consistency).
• The Gap: Existing research focuses on pre-training and instruction tuning for multilingualism, but the specific interaction between long-CoT reasoning and RLVR in non-English contexts remains under-explored.

2. Methodology

The authors conducted a systematic empirical study to quantify, analyze, and mitigate this collapse.

• Models: They tested a range of backbones, including:
  • English-dominant: OLMo-2-1B (trained primarily on English).
  • Multilingual: Llama-3.2-3B and Qwen-2.5-1.5B.
• Datasets:
  • Primary: GSM8K (grade-school math) translated into five target languages: Chinese (ZH), Korean (KO), Ukrainian (UK), Thai (TH), and Japanese (JA).
  • Secondary: SimpleRL-Zoo (harder math problems) and MATH500 for evaluation.
  • General Knowledge: Global MMLU-Lite (Korean) to test domain generalization.
• Training Algorithm: Group-Relative Policy Optimization (GRPO), a standard RLVR algorithm that optimizes policy based on relative rewards within a group of sampled outputs.
• Evaluation Metrics:
  • Task Accuracy: Standard math problem solving accuracy.
  • Target Word Ratio (Target WR): The percentage of tokens in the CoT belonging to the target language.
  • English Word Ratio (EN WR): The percentage of English tokens in the CoT.
  • Cross-lingual Consistency: Measuring if a model solves a problem correctly in both English and the target language.

3. Key Findings & Results

A. The Existence of Cross-lingual Collapse

• Systematic Drift: As training progresses and accuracy rises, the Target WR drops precipitously, while EN WR rises.
• Resource Sensitivity: The severity of the collapse correlates with the pre-training resource level of the target language:
  • High-resource (Chinese, Japanese): Moderate drift; models largely maintain fidelity.
  • Mid-resource (Korean): Moderate drift.
  • Low-resource (Ukrainian, Thai): Catastrophic collapse. For example, on Ukrainian GSM8K, the OLMo-2 model's Target WR dropped to 0.3% while accuracy surged, meaning the model reasoned almost entirely in English despite Ukrainian prompts.
• Mechanism: The model discovers that English reasoning paths yield higher rewards (more correct answers) during the exploration phase of GRPO. The advantage-weighted credit assignment reinforces these English trajectories, creating a self-reinforcing loop.

B. Triggers and Amplifiers

• Task Difficulty: Increasing task difficulty (e.g., adding SimpleRL-Zoo problems) accelerates the collapse. A model stable on standard GSM8K may collapse into English when faced with harder problems.
• Decoding Entropy: High-entropy decoding (high temperature, high top-p) during rollout generation allows the model to explore English paths more freely, triggering the collapse.
• Backbone Bias: English-centric backbones (like OLMo) collapse faster and more severely than multilingual backbones (like Llama/Qwen), though even the latter are not immune.

C. Domain Generality

The collapse is not limited to mathematics. Experiments on Global MMLU-Lite (general knowledge) showed the same pattern: models fine-tuned with harder curricula achieved higher accuracy but suffered severe language drift.

D. Mitigation Strategies and the Persistent Trade-off

The authors tested three interventions, all of which revealed a performance-fidelity trade-off:

1. Language-Consistency Reward: Adding an auxiliary reward for target-language tokens prevents collapse (maintaining high Target WR) but results in a significant drop in accuracy (e.g., ~7–9 percentage points).
2. Decoding-Time Controls: Lowering temperature or top-p restricts exploration, preventing the model from finding English shortcuts. This preserves language but also reduces accuracy.
3. Multilingual Mixing: Training on a mix of languages (e.g., Ukrainian + Korean + Chinese + English) partially restores fidelity but still incurs an accuracy penalty compared to the "collapsed" English-only reasoning mode.

4. Significance and Contributions

• Formalization of a New Failure Mode: The paper defines and quantifies "Cross-lingual Collapse," providing a clear metric (Target WR vs. Accuracy) to track it.
• Mechanistic Insight: It reveals that RLVR optimization in English-centric models inherently favors English reasoning paths as "shortcuts" to high rewards, undermining the goal of multilingual reasoning.
• The Inherent Trade-off: The most critical finding is that there is currently no "free lunch." Strategies that force the model to reason in the target language (fidelity) consistently degrade performance (accuracy). This suggests that the "best" reasoning path for these models is currently English, and forcing them to stay in lower-resource languages limits their reasoning depth.
• Implications for RLVR: The findings challenge the assumption that RLVR automatically improves multilingual reasoning. Instead, without specific constraints, it may degrade multilingual capabilities by reinforcing English priors.
• Future Directions: The paper calls for:
  • Redesigning exploration mechanisms to constrain surface form (language) while keeping semantic search broad.
  • Developing constrained or multi-objective RL that explicitly balances accuracy and language fidelity (Pareto frontier optimization).
  • Re-evaluating the necessity of "interpretable" CoT in non-English settings, suggesting latent reasoning with target-language summaries might be a viable compromise.

Conclusion

The paper concludes that Cross-lingual Collapse is a natural consequence of English-dominant pre-training combined with reward-driven optimization. While RLVR boosts reasoning capabilities, it does so by exploiting the model's strongest linguistic priors (English), often at the expense of the target language. Securing linguistic diversity in long-CoT settings requires not just better training data, but fundamental changes to how RLVR objectives are designed to prevent the model from "cheating" by switching languages.