Technical Summary: Source-Grounded Semantic Reinforcement Learning for Low-Resource Target-Language Generation

1. Problem Statement

Large Language Models (LLMs) exhibit significant performance disparities across languages. While high-resource languages benefit from abundant pretraining and instruction data, low-resource languages suffer from unstable generation, hallucinations, and poor factual grounding. Standard approaches to extend LLMs to new languages—such as continual pretraining followed by Supervised Fine-Tuning (SFT)—rely heavily on target-language text or task-specific supervision, which is often unavailable for genuinely low-resource languages.

In cross-lingual generation, the bottleneck is exacerbated by the scarcity of high-quality parallel data. While high-resource source-language monolingual data (e.g., Chinese or English news) is abundant, standard SFT cannot utilize it directly because it requires a target-language reference for every training instance. The central research question is: Can abundant source-language monolingual data be converted into useful supervision for low-resource target-language generation without requiring target-language references during training?

2. Methodology: SG-SRL Framework

The authors propose Source-Grounded Semantic Reinforcement Learning (SG-SRL), a "train–reinforce–recover" framework designed to leverage source-language monolingual data for cross-lingual semantic alignment.

2.1 Data Sources

The framework utilizes two distinct data sources:

• $D_{para}$: A small parallel corpus (source-target pairs) used for target-form initialization and recovery.
• $D_{src}$: A large source-language monolingual corpus used for semantic mid-training, containing no target-language references.

2.2 Three-Stage Procedure

1. Target-Form Initialization: The base model is fine-tuned on the small parallel corpus ($D_{para}$) for a few epochs. This establishes a "cold-start" generator capable of basic target-language fluency, task format, and style, though with limited semantic coverage.
2. Source-Grounded Semantic RL: The initialized model undergoes Reinforcement Learning (RL) on the large source-language corpus ($D_{src}$).
   • Reward Mechanism: A cross-lingual semantic reward is constructed using a multilingual reranker. The source input is treated as a query, and the generated target-language output is treated as a candidate. The reranker scores the semantic relevance between the two.
   • Reward Hacking & Safeguards: Direct optimization of this relevance reward leads to "verbosity-based reward hacking," where the model generates excessively long outputs to maximize concept coverage, degrading fluency and format. To mitigate this, the authors employ three safeguards:
     • Language Gate: A hard filter ensuring the output is predominantly in the target language.
     • Length Penalty: A batch-relative penalty that activates when outputs exceed 2.5x the median batch length.
     • Repetition Penalty: A penalty for repeated 4-grams.
   • Optimization: The policy is updated using GRPO (Group Relative Policy Optimization), treating the RL stage as "semantic mid-training" rather than final optimization.
3. Target-Form Recovery: The intermediate RL checkpoint, which possesses strong semantic grounding but degraded surface form, is fine-tuned again on the small parallel corpus ($D_{para}$). This stage restores fluency, conciseness, and task format while preserving the semantic gains acquired during RL.

2.3 Reward Instantiation

• LLM-based Reranker: In the primary experiments (Chinese-to-Thai), a generative yes/no judge (reranker) provides a scalar reward based on the probability of an affirmative answer to a relevance query.
• Encoder-based Embedding: In the Tibetan experiments, where a strong reranker is unavailable, an encoder-based embedding scorer (trained on Tibetan-Chinese pairs) provides a similarity score as the reward.

3. Key Contributions

• Framework Proposal: SG-SRL is introduced as a novel framework that converts source-language monolingual data into reference-free semantic supervision, enabling RL in low-resource settings.
• Semantic-Form Trade-off Identification: The authors identify that direct optimization of relevance-oriented rewards induces verbosity-based reward hacking. They demonstrate that while the intermediate RL checkpoint suffers from poor fluency and length control, it successfully learns useful cross-lingual semantic alignment, justifying the "recover" stage.
• Empirical Validation:
  • Chinese-to-Thai: Using SmolLM3-3B, SG-SRL significantly outperforms cold-start SFT in semantic adequacy and factual faithfulness.
  • Generalization: The learned semantic alignment transfers to long-form translation tasks, not just the specific title-generation format used in training.
  • Low-Resource Adaptation: In a Tibetan setting lacking a strong reranker, the framework successfully adapts using an encoder-based embedding reward, proving its applicability to realistic low-resource scenarios.

4. Experimental Results

• Main Task (Chinese-to-Thai Title Generation):
  • On a 5k development set, SG-SRL ranked first in 995 examples and second in 2,602 examples against a gold reference and a cold-start SFT baseline.
  • In pairwise comparisons, SG-SRL won against cold-start SFT in 72.3% of cases.
  • The improvement was most pronounced in semantic dimensions (adequacy, factual faithfulness) rather than surface-level overlap.
• Semantic vs. Form Analysis:
  • The intermediate RL checkpoint (GATE+BATCHLEN) achieved the highest semantic alignment scores (Entity, Event, Factual) but produced outputs with an average length of 531.7 characters, compared to 162.1 for gold references.
  • Removing safeguards (GATE-ONLY) led to severe verbosity (866.6 chars).
  • Aggressive length control (REFLEN) reduced length but significantly degraded semantic scores, confirming the trade-off.
• Transfer Learning: SG-SRL showed superior performance in long-form Chinese-to-Thai translation, maintaining higher entity/event alignment and factual consistency compared to baselines.
• Tibetan Generalization: Using an embedding-based reward, SG-SRL achieved comparable BLEU scores whether the semantic anchor was the source (Chinese) or target (Tibetan) text, demonstrating the framework's flexibility when strong rerankers are unavailable.

5. Significance and Claims

The paper claims that SG-SRL establishes a robust pipeline for aligning weak target languages using high-resource source anchors. Its primary significance lies in:

1. Decoupling Semantic and Form Learning: By separating source-grounded semantic learning from target-form regularization, the framework effectively neutralizes the verbosity-based reward hacking inherent in relevance optimization.
2. Reference-Free RL: It demonstrates that high-resource monolingual data can serve as a scalable supervision signal for low-resource generation without requiring parallel data for the RL phase.
3. Practical Applicability: The framework is adaptable to scenarios where strong multilingual rerankers do not exist, utilizing encoder-based embedding models as a practical substitute for the reward signal.

The authors conclude that this approach paves the way for more equitable multilingual model expansion, though they acknowledge that the framework currently relies on the availability of some form of cross-lingual relevance scorer (either a reranker or a domain-specific embedding model), which remains a constraint for the most resource-constrained languages.