Beyond Scalar Scores: Reinforcement Learning for Error-Aware Quality Estimation of Machine Translation

Technical Summary: Beyond Scalar Scores: Reinforcement Learning for Error-Aware Quality Estimation of Machine Translation

Problem Statement
Quality Estimation (QE) is critical for evaluating Machine Translation (MT) outputs without reference translations, a necessity for large-scale real-world deployment. While Large Language Models (LLMs) have advanced MT evaluation, they face significant challenges in low-resource settings. Current LLM-based evaluators often exhibit lower correlation with human judgments for low-resource languages compared to high-resource ones. Furthermore, existing QE approaches predominantly rely on scalar quality scores (e.g., Direct Assessment or DA scores). These scalar signals lack explicit contextual information regarding the specific translation errors driving the judgment, limiting the model's ability to reason about why a translation is poor. Traditional fine-grained annotation methods like Multi-Dimensional Quality Metrics (MQM) or Error Span Annotation (ESA) provide error context but are costly, require extensive annotator expertise, and are difficult to scale. There is a gap in the literature for a QE framework that effectively utilizes lightweight, error-aware signals to improve performance in resource-scarce environments.

Methodology
The authors propose ALOPE-RL (Adaptive Learning and Optimization for Policy-based quality Estimation with Reinforcement Learning), a policy-based reinforcement learning framework designed to train efficient adapters for QE.

1. Dataset Creation (En→Malayalam):

   • The authors introduce the first segment-level QE dataset for English→Malayalam (En→Ml), a severely resource-scarce language pair.
   • The dataset comprises 5,000 instances across Finance, News, and Legal domains.
   • Crucially, it includes Translation Quality Remarks (TQR): short, free-form natural language comments from human annotators describing error types (e.g., Untranslated, Addition, Mistranslation, Fluency Error) and erroneous spans. This serves as a weak supervision signal that is more flexible and less cognitively demanding than MQM-style span marking.
   • Synthetic Explanations: Using TQR, source, and target segments, the authors generated synthetic detailed explanations (error categories and descriptions) using strong reasoning models (DEEPSEEK-V3 and GEMINI-2.5-pro) to serve as ground-truth references for reward calculation.

2. ALOPE-RL Framework:

   • Policy Optimization: The framework treats QE as a policy optimization problem using Group Relative Policy Optimization (GRPO). The policy ($\pi_\theta$) generates structured responses containing a DA score, error categories, and a natural language explanation.
   • Multi-Component Reward System: The model is trained using a weighted sum of six reward components:
     • Exact Score Reward ($R_{exact}$): Penalizes the distance between predicted and gold DA scores.
     • Score Bin Reward ($R_{bin}$): Encourages predictions to fall within the correct score range (bin).
     • Error Type Reward ($R_{err}$): Uses the Jaccard index to measure similarity between predicted and gold error categories.
     • Description Reward ($R_{desc}$): Uses BERTScore F1 to assess semantic similarity between predicted and synthetic descriptions.
     • Length Reward ($R_{len}$): Rewards descriptions within a target word count (60–140 words).
     • Format Reward ($R_{fmt}$): Ensures adherence to a strict XML schema.
   • Efficiency: The system is designed for low-resource constraints, utilizing compact LLMs (≤4B parameters: QWEN3-4B, GEMMA-3-4B, LLAMA-3.2-3B) fine-tuned with LoRA (Low-Rank Adaptation) and 4-bit quantization. Training is performed on small datasets (≤5K instances).

3. Extension to Other Languages:

   • The framework was also tested on English→Tamil, English→Marathi, and English→Hindi. Since TQR was unavailable for these pairs, the authors utilized Word-level QE Tags (WTAG) (binary OK/BAD labels derived from post-editing alignments) as an alternative weak supervision signal to test generalizability.

Key Contributions

• New Dataset: Introduction of the first segment-level QE dataset for English→Malayalam, featuring human-annotated DA scores and Translation Quality Remarks (TQR).
• Novel Framework: Development of ALOPE-RL, a policy-based RL framework that integrates error-aware weak supervision (TQR) to train compact LLMs.
• Performance: Demonstration that incorporating TQR as a contextual input significantly improves LLM-based QE performance, enabling small models to outperform larger baselines.
• Efficiency: Achievement of state-of-the-art results using compact models (≤4B parameters) and limited data, proving the viability of error-aware policy learning under constrained compute and data budgets.

Results

• Performance on En→Ml: ALOPE-RL achieved a Spearman correlation ($\rho$) of 0.571 on the En→Ml dataset using GEMMA-3-4B with "Core rewards" (focusing on score and error type). This outperformed:
  • Strong encoder-based baselines (COMETKIWI: $\rho$=0.474; XCOMET-XL: $\rho$=0.352).
  • Larger LLM baselines (DEEPSEEK-V3: $\rho$=0.474).
  • Instruction Fine-Tuned (IFT) variants of the same model ($\rho$=0.525).
• Impact of TQR: Models trained with TQR consistently achieved higher correlations than those trained with WTAG or no weak signals. TQR proved particularly effective for Dravidian languages (Malayalam, Tamil), whereas WTAG was less effective or introduced noise for Indo-Aryan languages (Marathi, Hindi), likely due to morphological differences.
• Reward Ablation: The "Core rewards" configuration (Score + Error Type + Format) yielded better results than the "All rewards" configuration, suggesting that complex semantic rewards (Description/Length) can introduce noise that diffuses the reward signal for DA prediction.

Significance and Claims
The paper claims that error-aware, policy-based learning is a viable and efficient pathway for QE in low-resource settings. By moving beyond scalar scores to incorporate free-form, contextual error remarks (TQR), the authors demonstrate that:

1. Context Matters: Providing LLMs with explicit error context allows them to reason about translation quality more effectively than relying solely on numeric scores.
2. Efficiency: High-performance QE does not necessarily require massive datasets or large models; compact, quantized models fine-tuned with RL can surpass larger, data-hungry baselines.
3. Scalability: TQR offers a practical middle ground between the simplicity of scalar scores and the complexity of fine-grained error annotation, making it a scalable signal for improving QE systems.

The authors conclude that their approach successfully addresses the limitations of current LLM evaluators in low-resource scenarios by leveraging weak, natural-language supervision to drive policy optimization.