Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation

Technical Summary: RLSR (Reinforcement Learning for Source Rewriting)

Problem Statement
Source rewriting (or pre-editing) aims to improve Machine Translation (MT) quality by modifying the source text before it enters the MT model. While Large Language Models (LLMs) can perform this task via prompting, current approaches suffer from two critical limitations:

1. Heuristic Dependency: Prompt design is highly empirical and inconsistent; a strategy effective for one MT model often fails or degrades performance for another.
2. Resource Intensity: Achieving consistent results requires manual, per-model prompt tuning, which is time-consuming and resource-heavy.
   Existing methods that use iterative inference or feedback loops are computationally expensive. The authors ask whether a universal rewriting method can be discovered that performs consistently across diverse MT models without manual prompt tuning.

Methodology: RLSR Framework
The authors propose RLSR (Reinforcement Learning for Source Rewriting), a framework that treats source rewriting as a reinforcement learning problem where the reward is the direct improvement in downstream translation quality.

• Reward Definition: The reward $R$ is defined as the difference in automatic evaluation metric scores between the translation of the rewritten source ($\tilde{s}$) and the original source ($s$):
  $$R(s, \tilde{s}, r) = Q(s, M(\tilde{s}), r) - Q(s, M(s), r)$$
  where $M$ is the fixed downstream MT model, $r$ is the reference translation, and $Q$ is an automatic metric (specifically xCOMET-XL in the experiments).
• Optimization Objective: Since the MT model and metric are non-differentiable, standard Supervised Fine-Tuning (SFT) cannot directly optimize this objective. Instead, RLSR employs an on-policy RL algorithm, DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization).
• KL Regularization: To prevent "reward hacking" (e.g., generating nonsensical text that exploits metric vulnerabilities), the total reward includes a Kullback-Leibler (KL) divergence penalty against a reference policy ($R_{ref}$), typically an instruction-following LLM with a standard prompt:
  $$R_{total} = R(s, \tilde{s}, r) - \beta \log \frac{R_\theta(\tilde{s}|s)}{R_{ref}(\tilde{s}|s)}$$
• Training Process: The rewriting model samples a rewrite $\tilde{s}$, the MT model translates both $s$ and $\tilde{s}$, the metric $Q$ evaluates them, and the resulting reward updates the rewriting policy.

Key Contributions

1. Prompt-Free Optimization: RLSR eliminates the need for manual prompt tuning by directly optimizing the rewriting model toward the downstream translation objective.
2. Efficiency: Unlike iterative feedback methods, RLSR requires only a single forward pass of the rewriting model and the MT model per training step, making it computationally efficient at inference time.
3. Superiority over SFT: The paper demonstrates that SFT, when applied to source rewriting, often leads to "degeneration" where the model simply copies the source text (to minimize loss on tokens that don't change) rather than learning to make meaningful edits. RL forces the model to explore diverse strategies to maximize the reward.
4. Generalization: Models trained via RLSR for a specific MT model generalize well to other, unseen MT models, suggesting the learned strategies are robust rather than overfitted to a single model's idiosyncrasies.

Experimental Results
The authors evaluated RLSR on the WMT2025 General MT Shared Task (16 language pairs) across six different downstream MT models (ranging from 4B to 31B parameters, including Gemma and Qwen variants).

• Performance vs. Baselines:
  • RLSR-trained 4B models significantly outperformed the "no-rewriting" baseline and existing same-scale (4B) prompt-based baselines (Simplification, Paraphrase, Easy Translate) across all metrics (xCOMET, MetricX, GEMBA).
  • The 4B RLSR models achieved performance competitive with prompt-based baselines using a massive 235B LLM, demonstrating high parameter efficiency.
• Robustness: While prompt-based methods showed high variance (e.g., "Paraphrase" often degraded performance), RLSR provided consistent improvements across all tested MT models without any prompt adjustment.
• Metric Bias Mitigation: Although trained using xCOMET, the models also improved on GEMBA (a prompt-based, reference-free metric), indicating the gains were not merely overfitting to the training metric.
• Qualitative Analysis: Case studies revealed that RLSR learns to make localized, length-preserving edits targeting specific translation obstacles (disfluencies, ambiguities), whereas prompt-based methods often rewrite the entire text indiscriminately or hallucinate content.

Significance and Claims
The paper claims that RLSR provides a streamlined, automated approach to source rewriting that adapts seamlessly to diverse MT models. By shifting from heuristic prompt engineering to direct reward optimization, the method significantly improves translation quality without the computational overhead of iterative inference or the manual labor of prompt tuning. The authors position this as a more effective paradigm than SFT for rewriting tasks, as RL successfully avoids the "copying" degeneration common in supervised approaches.

Limitations Acknowledged

• Training Cost: RLSR training is more expensive than SFT because it requires repeatedly running the downstream MT model and evaluation metric to compute rewards.
• Metric Reliance: The approach assumes the automatic metric (xCOMET) faithfully reflects translation quality. While GEMBA results mitigate concerns of metric bias, the authors note that human evaluation would be definitive but was not conducted due to cost.
• Scope: Experiments were limited to six MT models and 16 language pairs; broader validation is suggested for future work.