Cross-lingual Self-Consistency for Multilingual Reasoning with Language Models

Technical Summary: Cross-lingual Self-Consistency for Multilingual Reasoning with Language Models

Problem Statement

Despite the expanding multilingual coverage of Large Language Models (LLMs), their advanced reasoning capabilities remain heavily concentrated in high-resource languages, particularly English and Chinese. While models can perform basic tasks across many languages, complex reasoning tasks (e.g., mathematical word problems) often fail when executed in the native language of the query.

Existing solutions attempt to bridge this gap by:

1. Reasoning in English: Translating the source query to English, reasoning in English, and translating the answer back.
2. Supervised Fine-Tuning (SFT): Training on parallel data (translated questions and reasoning traces).
3. Logit Manipulation: Forcing the model's output distribution to mimic English reasoning patterns.

These approaches suffer from significant limitations: they rely on scarce, high-quality parallel reasoning data, depend on the quality of machine translation (which introduces noise), and often fail to generalize to unseen languages or domains. Furthermore, strategies that force reasoning in English do not necessarily improve the model's intrinsic ability to reason in the source language.

Methodology

The authors propose an unsupervised Reinforcement Learning (RL) framework that enhances multilingual reasoning by enforcing cross-lingual self-consistency. The core principle is that a model should produce the same final answer for equivalent problems posed in different languages, even without access to ground-truth labels.

Key Components:

1. Unsupervised Signal: The method requires only monolingual questions (specifically English in the experiments) for training. It does not require gold answers or pre-existing parallel reasoning traces.
2. Self-Translation: To generate parallel inputs, the model itself translates the English training questions into target languages. While noisy, this approach is scalable and avoids injecting external knowledge from third-party translation engines.
3. Reward Function (RL):
   • The framework extends monolingual self-consistency (which minimizes the entropy of the answer distribution) to a multilingual setting.
   • Instead of minimizing entropy within a single language, the method minimizes the cross-entropy between the answer distribution of a specific language ($\hat{p}_\pi$) and a target distribution ($\hat{q}_\pi$) derived from other languages.
   • Target Distribution Strategies: The authors explore three ways to define $\hat{q}_\pi$:
     • Source Language: Equivalent to standard monolingual self-consistency.
     • Reference Language (English): Aligns all languages to the English distribution (acting as cross-lingual distillation).
     • Combined: Averages the answer distributions across all languages. This variant is used as the primary method, directly optimizing for consistency across the entire set of languages.
4. Training Algorithm: The authors utilize GRPO (Group Relative Policy Optimization) with the VeRL backend. The reward is computed based on the consistency of the final answers across the sampled trajectories in different languages.

Key Contributions

• Novel Framework: The first unsupervised RL approach to improve multilingual reasoning by explicitly enforcing cross-lingual self-consistency without requiring parallel data or ground-truth labels.
• Scalability: The method is practical for scenarios where multilingual reasoning data is unavailable, leveraging the model's own translation capabilities to create training signals.
• Generalization: The approach demonstrates strong generalization to unseen languages and out-of-distribution (OOD) domains, addressing the scarcity of high-quality multilingual reasoning datasets.

Experimental Results

The method was evaluated on the MGSM benchmark (10 languages) and three OOD benchmarks (mGPQA, MMATH, PolyMath) using Qwen2.5 models (1.5B, 7B, 14B) and Llama3 (8B).

• Performance Gains:
  • On MGSM, the method achieved average accuracy improvements of 7.8% (1.5B), 15.1% (7B), and 21.7% (14B) over the original models.
  • Gains were observed across nearly all languages, with some low-resource languages (e.g., Russian, Telugu) seeing relative improvements exceeding 100%.
  • The method outperformed prior baselines, including test-time scaling (S1) and DPO-based methods (LIDR), particularly in terms of token efficiency. While S1 achieved competitive results, it required approximately 3x more tokens for longer reasoning traces.
• Generalization:
  • Unseen Languages: When trained on a subset of languages, the model showed an 18.2% mean improvement on unseen languages within MGSM.
  • OOD Benchmarks: The method yielded average gains of 4.7% on mGPQA and 6.2% on MMATH, outperforming LIDR, which caused performance degradation on OOD tasks.
• Consistency: Analysis confirmed that the method significantly improved both monolingual and cross-lingual self-consistency. Models trained with this method reached near-peak performance with fewer samples (Maj@8) compared to baselines, indicating efficient optimization of inference compute.
• Resource Tiers: The method significantly reduced the performance gap between English and other languages, particularly in high- and medium-resource tiers, though low-resource languages still lagged due to the underlying model's proficiency limits.

Significance and Claims

The paper claims that consistency-based methods offer a viable path to improving the multilingual capabilities of LLMs without the prohibitive cost of supervised data annotation.

• Equity and Robustness: By narrowing the performance gap between high-resource and low-resource languages, the method advances the equity and robustness of LLMs across different linguistic contexts.
• Intrinsic Improvement: Unlike "translate-to-English" strategies that rely on English as a proxy, this method improves the model's intrinsic ability to reason in the source language.
• Practicality: The reliance on self-translation and monolingual data makes the approach highly scalable and applicable to domains where multilingual reasoning data is non-existent.

The authors conclude that while the method relies on current model scales (sub-20B) and faces limitations regarding the availability of low-resource benchmarks, it paves the way for enhancing multilingual reasoning performance at scale without extra data annotation. Future work is suggested to explore non-verifiable tasks and larger model architectures.