Technical Summary: Reference-Free Reinforcement Learning Fine-Tuning for MT

Problem Statement

Production machine translation (MT) is dominated by encoder-decoder Sequence-to-Sequence (Seq2Seq) models due to their efficiency and alignment capabilities. However, recent advancements in Reinforcement Learning (RL) for fine-tuning, specifically Group Relative Policy Optimization (GRPO), have been almost exclusively applied to large decoder-only Large Language Models (LLMs) with $\ge$7B parameters. These models are often impractical for production deployment across the world's long-tail of language pairs due to computational constraints. Furthermore, existing RL approaches for MT typically rely on parallel data or target-language references for reward signals, limiting their applicability in low-resource scenarios. There is a lack of systematic study regarding the efficacy of GRPO on efficient encoder-decoder architectures using reference-free reward signals.

Methodology

The authors propose a reference-free RL fine-tuning pipeline for the NLLB-200 encoder-decoder model (600M and 1.3B parameters) using Group Relative Policy Optimization (GRPO).

1. Policy and Optimization:

• Model: The base model is NLLB-200. Training utilizes 4-bit quantization (NF4) and Low-Rank Adaptation (LoRA) applied to query and value projections, allowing training on a single NVIDIA A10G GPU.
• Algorithm: GRPO is employed to eliminate the need for a separate value model, reducing memory overhead. The advantage $A_i$ for each hypothesis is estimated from intra-group reward variation:
  $$A_i = \frac{r_i - \text{mean}(r)}{\text{std}(r) + \epsilon}$$
  where $\epsilon = 10^{-4}$ prevents noise amplification. The policy is updated via a PPO-clipped surrogate objective with a KL penalty to prevent deviation from the reference model. The reference model ($\pi_{ref}$) is obtained by disabling LoRA adapters on the same model, avoiding the need for a separate reference model.

2. Hybrid Reference-Free Reward:
The reward signal requires no parallel data at fine-tuning time, relying instead on monolingual source text. It combines two pre-trained quality estimators:

• LaBSE: Provides cross-lingual semantic adequacy via cosine similarity between source and hypothesis embeddings.
• COMET-Kiwi: Provides a learned quality estimate based on direct assessment annotations.
• Hybrid Reward: The final reward is an equal-weighted average ($0.5/0.5$) of LaBSE and COMET-Kiwi scores. This combination is designed to balance fluency and faithfulness while mitigating the limitations of individual estimators on underrepresented languages.

3. Experimental Design:

• Experiment A (Systematic Evaluation): Trained on FLORES-200 development splits (source side only) across 13 typologically diverse languages. Evaluated against an unfine-tuned baseline and Supervised Fine-Tuning (SFT) baselines (1-epoch and 3-epoch).
• Experiment B (Cross-Domain): Trained on 10,000 monolingual English sentences from CCNews. Evaluated on FLORES-200 (out-of-domain) and NTREX-128 (news domain).
• Ablations: Investigated KL regularisation strength ($\beta$) and training data size ($N$).

Key Results

1. Consistent Reference-Free Gains:
GRPO yielded consistent improvements over the baseline across all 13 languages and both model scales (600M and 1.3B) without any target-language data.

• Maximum Gain: Traditional Chinese showed the largest improvement (+5.03 chrF++ on the 1.3B model).
• Competitiveness with SFT: On morphologically complex languages with weak baselines (e.g., Basque, Bengali, Swahili), GRPO outperformed or matched 1-epoch SFT and remained competitive with 3-epoch SFT, despite the latter having access to parallel target data during training.

2. Empirical Gain Pattern:
The study identified a consistent pattern where the magnitude of gains is largest where:

• The baseline performance is weakest.
• The reward discriminability (the ability of the reward model to distinguish quality differences) is highest.
  This pattern held true across both English and Spanish source languages, suggesting the gains are driven by target-language properties rather than source-specific training dynamics.

3. Model Scaling and Failure Modes:

• Scaling: The 1.3B model generally outperformed the 600M model, though some languages (e.g., Tibetan, Yoruba) showed "reversed scaling" where the smaller model performed better, potentially acting as an implicit regularizer against noisy reward signals.
• Arabic: Consistently failed to improve, attributed to a strong baseline (leaving little headroom), root-pattern morphology that COMET-Kiwi struggles to score, and domain mismatch.
• Tibetan: Exhibited reward variance collapse in extended training (Experiment B), where hypotheses converged to similar scores, causing numerical instability. This required best-checkpoint selection to mitigate.

4. Practical Accessibility:
The full pipeline runs on a single 24GB GPU. A reliability threshold for training data was identified at approximately 500 source sentences; below this, neither GRPO nor SFT produced significant gains. The approach exhibited zero catastrophic forgetting on held-out languages (French, Hindi, Russian).

Significance and Claims

The paper claims to demonstrate a practical path for improving production-grade Seq2Seq MT models without parallel data. Its primary contributions are:

1. Feasibility of Reference-Free RL on Efficient Architectures: Proving that GRPO can effectively fine-tune small encoder-decoder models (600M/1.3B) using only monolingual source text, making RL accessible for low-resource language pairs where large decoder-only LLMs are impractical.
2. Identification of a Predictive Pattern: Establishing that the utility of reference-free RL is not uniform but is predictable; it is most effective for languages with weak baselines and high reward discriminability. This offers a heuristic for practitioners to select languages where this approach is likely to succeed.
3. Competitiveness with Supervised Methods: Demonstrating that reference-free GRPO can match or exceed the performance of supervised fine-tuning (SFT) on specific language families, even when SFT has access to parallel data, highlighting the potential of reward-guided exploration in data-scarce regimes.

The authors remain modest, noting that the reward models themselves were pre-trained on parallel data, limiting the approach to "zero-resource" only in the fine-tuning stage. They also acknowledge that the observed patterns require validation on a larger language sample and that the current reward discriminability limits the quality ceiling for certain morphological types (e.g., Arabic).