Technical Summary: MTQE.en-he

Problem Statement

Machine Translation Quality Estimation (MTQE) is a critical component of MT pipelines, enabling systems to accept high-quality outputs and flag low-quality ones for human review. While MTQE has been extensively studied for high-resource language pairs, Hebrew presents specific challenges. It is classified as a mid-resource language with significantly fewer annotated resources than English. Furthermore, Hebrew's rich morphology creates conditions akin to low-resource learning, complicating effective translation and quality estimation. Prior to this work, there was no publicly available benchmark specifically for English-to-Hebrew MTQE, despite the existence of Hebrew-to-English test sets in WMT shared tasks.

Methodology

Dataset Construction (MTQE.en-he)

The authors introduce MTQE.en-he, the first publicly available benchmark for English-Hebrew MTQE.

• Source: The dataset comprises 959 English segments derived from the WMT24++ corpus, covering four domains: literary, news, social, and speech.
• Translation: Hebrew translations were generated using Google Translate.
• Annotation: Three human experts with native-level proficiency in both languages and extensive translation workflow experience annotated the segments.
• Scoring: Annotations were performed using Direct Assessment (DA) on a scale of 0 to 100, following standard WMT QE guidelines. The ground truth for evaluation is the mean of the three annotators' scores.
• Statistics: The dataset exhibits a high score distribution (73% of segments scored 70 or above) and short segment lengths (59% under 30 words). Inter-annotator agreement (Pearson correlation of one annotator vs. the mean of the others) averaged 0.5337.

Baseline Benchmarks

The paper evaluates three distinct approaches on the full dataset:

1. ChatGPT Prompting: Two prompting strategies were tested: "freestyle" (open-ended judgment) and "guidelines" (incorporating full annotation instructions).
2. TransQuest: Two versions were evaluated: a multilingual (any-to-any) model and an English-to-any (en-any) model.
3. CometKiwi: A model capable of simultaneous sentence-level and word-level quality prediction.

Fine-Tuning Experiments

The dataset was partitioned into training (300), validation (100), and test (559) slices using stratified sampling to maintain score distribution. The authors fine-tuned TransQuest and CometKiwi for 50 steps (5 epochs) using four methods:

• Full Fine-Tuning (FullFT): Updating all model parameters.
• LoRA (Low-Rank Adaptation): Freezing the base model and learning low-rank updates for attention and feed-forward layers.
• BitFit: Fine-tuning only bias terms and the head classifier layer (approx. 0.2% of parameters).
• FTHead: Fine-tuning only the classification head.

All methods used identical hyperparameters (batch size 32, learning rate 3e-5, 10% warmup with linear decay).

Ensembling

The authors also tested an ensemble approach, averaging the predicted scores of the models. They selected the "freestyle" ChatGPT prompt for the ensemble based on Occam's Razor (preferring simpler solutions).

Key Results

Baseline Performance

On the full dataset (n=959), CometKiwi emerged as the strongest single baseline with a Pearson correlation of 0.4828 and Spearman of 0.5456. TransQuest (en-any) followed with 0.4327 Pearson, while ChatGPT (freestyle) achieved 0.4266 Pearson.

• Limitations: Both TransQuest and CometKiwi failed to predict scores of 0.91 or higher, even when ground truth scores were in the "excellent or perfect" range (31% of the dataset). ChatGPT exhibited a tendency to repeat specific scores, creating horizontal bands in scatter plots.

Ensembling Improvements

Ensembling the three models (ChatGPT-freestyle, TransQuest, and CometKiwi) yielded the best single-model performance, outperforming the best individual baseline (CometKiwi) by 6.4 percentage points in Pearson (0.5472 vs. 0.4828) and 5.6 percentage points in Spearman (0.6014 vs. 0.5456).

Fine-Tuning Outcomes

• Full Fine-Tuning: FullFT failed to improve TransQuest and degraded CometKiwi's performance by 2–3 percentage points. Analysis revealed signs of overfitting and distribution collapse (where the model's output distribution became inconsistent or flat).
• Parameter-Efficient Methods: LoRA, BitFit, and FTHead successfully improved both TransQuest and CometKiwi by 2–3 percentage points in both Pearson and Spearman correlations on the test set.
  • CometKiwi + LoRA: Achieved 0.4670 Pearson (up from 0.4495).
  • TransQuest + LoRA: Achieved 0.4445 Pearson (up from 0.4205).
• Stability: Unlike FullFT, parameter-efficient methods demonstrated stable learning curves and modest, consistent distribution shifts.

Significance and Claims

The paper claims three primary novelties:

1. First Public Dataset: It is the first publicly released dataset for English-Hebrew MTQE, addressing a gap in resources for this mid-resource language pair.
2. Benchmarking New Paradigms: It establishes baselines using ChatGPT prompting and model ensembling for this specific language pair.
3. Parameter-Efficient Fine-Tuning: It demonstrates that for this under-resourced scenario, full-model updates are prone to overfitting, whereas parameter-efficient methods (LoRA, BitFit, FTHead) train stably and yield measurable improvements.

The authors position MTQE.en-he as a tool to enable future research on low-resource language systems. They explicitly note limitations, including the use of a single translation engine (Google Translate), the reliance on DA scores rather than the superior MQM metrics due to budget constraints, and the modest dataset size which limits the number of low-score samples. The work aims to inspire further research into synthetic data generation, cross-lingual transfer, and calibration to improve prediction quality for English-Hebrew and similar language pairs.