Technical Summary: Improving Cross-Lingual Factual Recall via Consistency-Driven Reinforcement Learning

Problem Statement

Large Language Models (LLMs) trained predominantly on English data encode vast amounts of world knowledge but frequently fail to access this knowledge reliably when queried in non-English languages. This phenomenon, termed cross-lingual factual inconsistency, is not primarily caused by a deficit in knowledge storage. Instead, mechanistic studies suggest the failure occurs during the "language transition" phase: the model may retrieve the correct concept in intermediate layers (often in an English-centric or language-agnostic latent space) but fails to map this concept reliably into the correct target-language tokens in the final layers.

While Continual Pretraining (CPT) on parallel corpora has been shown to improve translation fluency, it often fails to substantially boost performance on demanding tasks like factual recall. This suggests that while parallel data aligns internal representations, it does not sufficiently teach the model to access that aligned knowledge through non-English interfaces.

Methodology

1. The POLYFACT Dataset

To address the lack of large-scale, fully parallel multilingual factual data, the authors introduce POLYFACT.

• Construction: Derived from Wikidata truthy triples, the dataset contains 100,000 facts across 12 typologically diverse languages (including high-resource languages like English, Chinese, and Spanish, and low-resource languages like Swahili and Bengali).
• Format: It is a multiple-choice question (MCQ) dataset where each fact is fully parallel across all 12 languages. For each fact, three distractors are sampled from co-occurring objects of the same property to ensure plausibility.
• Quality: The dataset underwent rigorous quality control using a web-grounded LLM judge (GPT-4o) and manual review, achieving 91% agreement between the LLM and human annotators.

2. Training Approaches

The study compares three post-training strategies on two 7B-scale models (OLMo-2-1124-7B and Qwen-2.5-7B):

1. Continual Pretraining (CPT): Light pretraining on a balanced parallel corpus (TED2025) to align representations before fine-tuning.
2. Supervised Fine-Tuning (SFT): Training on POLYFACT using a joint objective that combines standard classification loss with a consistency term. The consistency term minimizes the KL divergence between the model's output distributions across the 12 parallel languages, encouraging the model to produce consistent answers regardless of the input language.
3. Consistency-Driven Reinforcement Learning (GRPO): A Group Relative Policy Optimization approach where the reward function explicitly incentivizes cross-lingual consistency.
   • Reward Mechanism: For a group of 8 rollouts (each containing 12 parallel generations), the reward is calculated based on correctness (+1), hallucination penalties (-0.5), and a specific bonus (+1) if all 12 languages in the group are answered correctly. This forces the model to learn a shared retrieval pathway rather than language-specific memorization.

3. Mechanistic Analysis

The authors employ two interpretability tools to analyze how these methods reshape the model's internal architecture:

• LAHIS: Analyzes attention heads to determine language-specific processing contributions.
• LAPE: Identifies language-specific neurons in MLP layers by measuring activation frequency and entropy.

Key Results

Performance Gains

• GRPO vs. SFT: GRPO consistently outperforms SFT across all evaluation benchmarks (POLYFACT, KLAR, and Global-MMLU).
  • On POLYFACT, GRPO achieves the highest accuracy in both high- and low-resource languages.
  • On KLAR (a free-form generation benchmark), GRPO improves performance across almost all languages, whereas SFT often regresses. This indicates that SFT primarily learns to match answer candidates (pattern matching) rather than improving genuine cross-lingual retrieval.
  • On Global-MMLU (requiring multi-step reasoning), GRPO models generalize better than SFT or CPT-only models.
• Role of CPT: Light CPT on parallel data provided limited additional gains when combined with post-training. In some cases, CPT followed by post-training resulted in lower performance than post-training from the base model alone, suggesting that alignment alone does not guarantee better knowledge access.

Cross-Lingual Transfer

GRPO demonstrates superior generalization to held-out languages (languages not seen during post-training, such as Greek, Ukrainian, and Hebrew).

• Gains are most pronounced for languages sharing scripts or subwords with training languages (e.g., Latin-script languages or those sharing Perso-Arabic subwords).
• Performance drops are observed in "script-isolated" languages (e.g., Greek, Korean), indicating that while GRPO improves shared representations, it is still constrained by tokenization boundaries.

Mechanistic Insights

• Language Routing: GRPO fundamentally restructures multilingual computation. Unlike the base model, which makes language-routing decisions immediately (often in the first layer), GRPO delays linguistic specialization to later layers. This creates a larger, shared language-agnostic intermediate space.
• Neuron Specialization: GRPO increases the proportion of English-specific neurons (by ~38%) while reducing specialization in other languages (e.g., Bengali and Swahili). This suggests the RL objective "squeezes" the model's behavioral space, encouraging reliance on the most stable representational backbone (English) as a pivot for other languages.
• Head Sharing: While SFT increases pairwise head overlap significantly (especially among Indo-European languages), GRPO achieves a more moderate increase in overlap but drives a deeper reorganization of processing, shifting it away from early routing layers.

Significance and Claims

The paper posits that cross-lingual factual inconsistency is less a problem of missing knowledge and more a problem of unreliable access to existing latent knowledge.

• Primary Contribution: The authors demonstrate that consistency-driven reinforcement learning (GRPO) is a more effective mechanism than Supervised Fine-Tuning (SFT) or light Continual Pretraining for improving cross-lingual factual recall. GRPO succeeds by promoting shared internal representations and reorganizing multilingual routing to ensure that factual concepts are accessed consistently across languages.
• Dataset Release: The release of POLYFACT provides a critical resource for studying and benchmarking cross-lingual factual consistency, addressing the previous lack of large-scale, fully parallel multilingual QA data.
• Modest Scope: The authors acknowledge that their method improves factual access but does not necessarily solve broader multilingual reasoning challenges (as seen in Global-MMLU results). Furthermore, the approach is currently limited to 7B-scale models and may not generalize to significantly different architectures or larger models without further investigation. The paper also notes that biases in the underlying Wikidata source (e.g., English-centric proper nouns) can inadvertently reinforce certain failure modes, such as answering proper-noun questions in English rather than the target language.