💬 NLP

Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

This paper proposes a reinforcement learning approach using Group Relative Policy Optimization (GRPO) with semantic rewards to expand large language models into low-resource languages, effectively mitigating the "alignment tax" and catastrophic forgetting typically caused by supervised fine-tuning while preserving general capabilities and improving semantic quality.

Original authors: Zeli Su, Ziyin Zhang, Zhou Liu, Xuexian Song, Zhankai Xu, Longfei Zheng, Xiaolu Zhang, Rong Fu, Guixian Xu, Wentao Zhang

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zeli Su, Ziyin Zhang, Zhou Liu, Xuexian Song, Zhankai Xu, Longfei Zheng, Xiaolu Zhang, Rong Fu, Guixian Xu, Wentao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Alignment Tax"

Imagine you have a brilliant, world-traveling chef (a Large Language Model) who can cook amazing dishes in English, Spanish, and French. But they've never cooked in Tibetan.

To teach them Tibetan, the old method (called Supervised Fine-Tuning or SFT) is like forcing the chef to stand in a kitchen and copy a single recipe book word-for-word. The chef memorizes the exact spelling of every ingredient and the precise order of every step.

The Catch: Because the Tibetan recipe book is small and the chef is so focused on copying it perfectly, they start to forget how to cook their famous French dishes. This is what the paper calls the "Alignment Tax." You gain a new skill (Tibetan), but you lose your old skills (English/French) because the training was too rigid.

The New Solution: "Semantic Space" Alignment

The authors propose a different way to teach the chef. Instead of forcing them to copy the exact words on the page, they teach them to understand the meaning behind the dish.

They use a new method called Reinforcement Learning with Semantic Rewards. Here is how it works:

The Goal: The goal isn't to match the recipe book word-for-word. The goal is to make a dish that tastes the same as the original, even if the ingredients are listed differently or the steps are described in a unique way.
The Judge (The Reward): Instead of a strict teacher checking every letter, the chef gets a "taste test" from a smart food critic (an embedding model). The critic says, "This dish captures the essence of the original recipe," even if the chef used different words to describe it.
The Safety Net: To make sure the chef doesn't accidentally start cooking in French or mixing languages, there is a simple rule: "You must write the recipe in Tibetan script."

How They Did It (The Two-Stage Plan)

The researchers didn't just jump into the new method; they used a two-step process:

Step 1: The Warm-Up (Cold-Start): First, they gave the chef a tiny bit of the old-school word-for-word training. This just taught the chef how to hold a pen in Tibetan and write basic sentences. It wasn't about being perfect; it was just to get them started so they didn't get lost.
Step 2: The Taste Test (Reinforcement Learning): Once the chef could write, they switched to the new method. The chef tried many different ways to write the recipe. Every time they got the meaning right (according to the smart critic), they got a reward. If they got the meaning wrong or mixed languages, they got no reward.

What Happened? (The Results)

The researchers tested this on translating between Tibetan and Chinese and writing Tibetan news headlines.

The Old Way (SFT): The chef became very good at copying the exact words of the Tibetan recipes. However, they forgot a lot of their French cooking skills (the "tax" was high).
The New Way (Semantic RL):
- Better Memory: The chef kept their French skills almost perfectly intact. They didn't forget their old capabilities.
- Better Meaning: Even though the new chef's Tibetan recipes didn't match the reference book word-for-word (lower "n-gram overlap"), the smart food critic (LLM judges) preferred the new chef's dishes because they captured the true flavor and meaning better.
- More Flexible: The new chef learned to express the same idea in many different ways, rather than just one rigid way.

The Takeaway

The paper argues that when teaching AI a rare language, we shouldn't force it to memorize a dictionary. Instead, we should reward it for understanding the meaning.

By focusing on "semantic space" (the world of ideas and meanings) rather than "token-level" (the world of specific letters and words), we can teach AI new languages without making it forget everything else it already knows. It's like teaching someone to speak a new language by having them tell stories, rather than making them recite a dictionary, ensuring they remain a good storyteller in all languages.

Technical Summary: Reinforcement Learning with Semantic Rewards for Low-Resource Language Expansion

1. Problem Statement

The expansion of Large Language Models (LLMs) to low-resource languages often incurs an "alignment tax": improvements in the target language come at the cost of catastrophic forgetting in general capabilities (e.g., performance in high-resource languages). The authors argue this trade-off stems from the rigidity of Supervised Fine-Tuning (SFT). SFT relies on teacher-forced, token-level likelihood maximization on narrow, biased data distributions. This enforces surface-form imitation, leading to overly confident parameter updates that aggressively reallocate representational capacity, thereby interfering with pre-trained knowledge and degrading general competence.

2. Methodology: Semantic-Space Alignment

The paper proposes a paradigm shift from token-level distribution matching to semantic-space alignment, prioritizing meaning preservation over rigid surface-form imitation. This is operationalized through a two-stage training framework powered by Group Relative Policy Optimization (GRPO).

Two-Stage Training Paradigm:

Cold-Start SFT: A lightweight supervised fine-tuning step (5k instances, 2 epochs) is performed to bootstrap minimal output competence in the target language (e.g., correct script usage and basic consistency). This ensures the model can generate non-degenerate outputs before reinforcement learning begins.
Reinforcement Learning with Semantic Rewards: Starting from the cold-start policy, the model undergoes GRPO optimization. Instead of token-level likelihood, the model is optimized using embedding-level semantic rewards.
- Objective: The model explores diverse linguistic realizations that maintain semantic equivalence to reference texts, rather than mimicking specific token sequences.
- Stability: GRPO incorporates trust-region constraints (via relative comparisons within a sampled group of candidates) to limit policy drift, preventing destructive interference with pre-trained representations.

Reward Design:
The reward function $R(y, y^*)$ combines two components:

Semantic Similarity ( $R_{sim}$ ): Calculated using cosine similarity between sentence embeddings of the generated output $y$ and the reference $y^*$ , derived from a bilingual embedding model (adapted from CINO/XLM-R). A threshold-and-rescale shaping function is applied to focus learning on meaningful improvements beyond minimal adequacy.
Language Consistency ( $R_{lang}$ ): A rule-based hard constraint (using Unicode ranges and regex) that penalizes mixed-language or off-target outputs, ensuring optimization remains strictly within the target low-resource language space.

3. Key Contributions

Semantic-Space Alignment Paradigm: The authors propose decoupling meaning preservation from surface-form imitation by utilizing GRPO with embedding-level rewards.
Mitigation of Alignment Tax: The approach demonstrates that significant low-resource gains can be achieved while virtually eliminating the degradation of general capabilities (alignment tax) typically seen in SFT.
Superior Semantic Quality: The method produces outputs that are semantically superior and preferred by LLM judges over SFT baselines, even when n-gram overlap (BLEU/ROUGE) is lower.
Transferable Representations: The paper validates that semantic RL yields more robust and transferable representations, evidenced by stronger few-shot generalization to downstream tasks compared to supervised methods.

4. Experimental Results

The approach was evaluated on Tibetan–Chinese Machine Translation (MT) and Tibetan Headline Generation (HG).

Task Performance vs. Alignment Tax:
- Machine Translation: While Strong SFT achieved higher reference-based BLEU scores (0.6006 vs. 0.4519 for RL), it suffered a significant alignment tax, dropping CMRC (Chinese reading comprehension) scores by ~5.15 points. In contrast, the RL method preserved general capabilities (CMRC scores improved or remained stable) while achieving high semantic similarity.
- Headline Generation: RL achieved comparable general capability preservation to SFT but was preferred by LLM judges in 51.2% of pairwise comparisons versus 35.1% for SFT, despite having lower ROUGE-L scores.
Few-Shot Transfer: In a transfer test from MT to HG (using only 1,000 HG samples), the RL-initialized model achieved higher semantic similarity (0.5690 vs. 0.5456) than the SFT-initialized model, indicating that RL learns more generalizable, semantically grounded features rather than task-specific surface patterns.
Reward Ablation: Experiments confirmed that combining embedding similarity with language consistency is critical. Adding BLEU-based rewards weakened performance, reintroducing the rigidity of token-level matching.

5. Significance and Claims

The paper claims that Reinforcement Learning with Semantic Rewards provides a safer and more reliable pathway for inclusive low-resource language expansion. By shifting the alignment objective from distribution matching to meaning-centered alignment, the method allows models to acquire new linguistic capabilities without the destructive interference typical of unconstrained likelihood maximization.

The authors conclude that this paradigm offers a scalable solution for extending LLMs to weakly supported languages under data scarcity, effectively resolving the tension between task adaptation and the preservation of general competence. The study suggests that reference-based metrics often fail to capture true semantic quality in low-resource settings, and that semantic-space alignment better captures human preferences and transferability.