💬 NLP

Source-Free MT Evaluation Is Not MT Evaluation

This paper argues that machine translation evaluation must be fundamentally source-grounded to ensure adequacy, calling for a paradigm shift where quality estimation is treated as a primary evaluation method and references are used only as auxiliary evidence rather than the dominant standard.

Original authors: Baban Gain, Ramakrishna Appicharla, Asif Ekbal

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Baban Gain, Ramakrishna Appicharla, Asif Ekbal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computers that translate languages, there is a constant need to know if the machine is doing a good job. Imagine a translator who speaks two languages; we need a way to check if what they wrote in the second language actually means the same thing as what was said in the first. For decades, the standard way to check this has been to compare the machine's output against a human-written example of what the translation should look like. This human example is called a reference. If the computer's words match the human's words closely, the system gets a high score. This method works well for checking if the translation sounds natural and grammatically correct in the new language. However, it has a blind spot: it cannot easily tell if the translation is faithful to the original meaning if the computer chose different words than the human did. A translation can be perfect in meaning but look very different from the human example, and under the old rules, it would be punished for that difference.

A team of researchers from India set out to investigate whether the most advanced tools used today are actually looking at the original meaning, or if they are just obsessed with matching the human example. They focused on a specific type of evaluation tool that is supposed to look at three things at once: the original sentence, the machine's translation, and the human reference. The researchers wanted to know if these tools were truly using the original sentence as the main guide for judging accuracy, or if they were secretly letting the human reference dictate the score. To find out, they designed a clever test where they took a perfect translation and swapped out either the original sentence or the human reference with something that didn't fit. If a tool is truly grounded in the original meaning, changing the original sentence should hurt its score much more than changing the human reference. If the tool is biased toward the reference, it will panic when the reference changes, even if the translation still makes sense with the original.

The results of this investigation revealed a surprising and systematic flaw in how these advanced tools work. When the researchers tested a popular tool called COMET, they found that changing the human reference caused the score to drop dramatically, while changing the original sentence barely made a dent. In fact, for nearly every single example they tested, the tool was more than ten times more sensitive to changes in the human reference than to changes in the original text. This means the tool was treating the human reference as the absolute truth, rather than treating the original sentence as the authority. Even when the machine's translation was perfectly correct according to the original sentence, the tool gave it a terrible score simply because it didn't match the human reference. The researchers tested other sophisticated tools as well. One of them, XCOMET-XXL, did pay more attention to the original sentence than COMET did, but it still reacted much more strongly to changes in the human reference. Another tool, MetricX, showed a mixed picture: it was heavily biased toward the reference when translating into English, but behaved more fairly when translating from English into other languages.

The study also looked at how large language models, the powerful artificial intelligence systems that can write and reason, act as judges. When these models are asked to grade translations, they often perform better when they are allowed to compare the machine's output against a human reference, rather than just looking at the original sentence. This suggests that the models are finding it easier to spot similarities between two texts in the same language than to figure out if a text in one language truly matches a text in another. The researchers found that when the original sentence and the human reference disagreed, the models tended to side with the reference, ignoring the original meaning. This is a critical problem because the original sentence is the only thing that defines what the translation is supposed to mean. If a translation preserves the original meaning but differs from the human reference, it is a good translation. If a translation matches the human reference but changes the original meaning, it is a bad one.

The authors argue that the entire field has been looking at this problem backward. Currently, researchers treat the human reference as the gold standard and view the original sentence as just a backup plan for when a reference is missing. The paper suggests flipping this relationship. The original sentence should be the primary authority for judging whether a translation is accurate, and the human reference should be treated as just one possible example of how that meaning could be expressed. The researchers point out that a single human reference can never capture every way a sentence could be translated. It might make choices about grammar or style that are not required by the original text. By relying too heavily on the reference, evaluation tools are punishing translations that are faithful to the source but different from the human example.

This research does not say that human references are useless. They are still very helpful for checking if a translation sounds natural and flows well. However, the study concludes that any system that removes the original sentence from the evaluation process, or allows the human reference to override the original meaning, is fundamentally incomplete. The researchers call for a new approach where evaluation tools are explicitly designed to prioritize the relationship between the original sentence and the machine's output. They suggest that future tools should be tested to ensure they react strongly when the original meaning is changed, and that they treat the human reference as a helpful hint rather than the final word. Until this shift happens, the scores we see for machine translation systems may be measuring how well a computer mimics a specific human writer, rather than how well it understands and preserves the original message.

Technical Summary: Source-Free MT Evaluation Is Not MT Evaluation

Problem Statement

Machine Translation (MT) evaluation has historically relied on reference-based metrics, which compare system outputs against human-written reference translations. While these metrics (e.g., BLEU, COMET) are standard, the paper argues that they fundamentally misrepresent the definition of translation adequacy. Adequacy is defined as the extent to which a hypothesis preserves the meaning of the source sentence.

The core problem identified is that current evaluation protocols often treat the reference as the primary standard of truth rather than as auxiliary evidence. This creates two specific issues:

Source-Free Evaluation: Protocols that remove the source (relying only on hypothesis-reference similarity) cannot assess adequacy because they lack the ground truth of what was to be translated. They measure target-side agreement, not faithfulness to the source.
Reference Dominance in Hybrid Metrics: Even "hybrid" metrics that accept the source, hypothesis, and reference as inputs often fail to ground their judgments in the source. Instead, they may rely predominantly on the compatibility between the hypothesis and the reference, effectively treating the source as a secondary or ignored input.

The paper posits that an evaluation protocol is structurally incomplete for adequacy if it removes the source or allows the reference to dominate the source when they provide conflicting evidence.

Methodology

To empirically test whether hybrid metrics are truly source-grounded, the authors introduce a counterfactual replacement diagnostic. This method isolates the sensitivity of a metric to the source versus the reference while holding the "good" hypothesis constant.

The Diagnostic Procedure:

Setup: For a given example $i$ , the authors use a consistent triplet: source ( $s_i$ ), a known good translation/hypothesis ( $g_i$ ), and a reference ( $r_i$ ).
Corruption Conditions:
- Source Replacement: The metric scores the pair $(s_j, g_i, r_i)$ , where $s_j$ is a source from a different example $j$ that is incompatible with $g_i$ .
- Reference Replacement: The metric scores the pair $(s_i, g_i, r_j)$ , where $r_j$ is a reference from example $j$ that is incompatible with $g_i$ .
Measurement: The authors calculate the sensitivity (score change) for both conditions.
- For "higher-is-better" metrics (e.g., COMET, XCOMET-XXL), sensitivity is the reduction in score.
- For "lower-is-better" metrics (e.g., MetricX-24-Hybrid-XL), sensitivity is the increase in score.
Analysis: They compute the Ref/Src gap ratio (Reference Sensitivity / Source Sensitivity) and the Reference Dominance Rate (the percentage of examples where reference sensitivity exceeds source sensitivity).
- A metric is considered source-grounded if the Ref/Src ratio is substantially below 1 (indicating source corruption is more damaging).
- A metric is reference-dominated if the ratio is above 1.

The authors applied this diagnostic to three prominent hybrid metrics: COMET, XCOMET-XXL, and MetricX-24-Hybrid-XL, across 6,902 examples from 14 translation directions (including both into-English and out-of-English pairs). They also reviewed existing literature on LLM-as-a-Judge evaluation to assess similar biases.

Key Results

1. COMET

Extreme Reference Dominance: COMET is highly sensitive to reference corruption but relatively insensitive to source corruption.
Quantitative Findings: Replacing the reference caused an average score drop 10.43 times larger than replacing the source.
Consistency: Reference sensitivity exceeded source sensitivity for 99.54% of all examples. This pattern held consistently across all language pairs and directions (both into and out of English).
Interpretation: While COMET is not blind to the source (source replacement does lower scores), its judgment is overwhelmingly governed by the reference.

2. XCOMET-XXL

Improved Source Use, Persistent Bias: XCOMET-XXL utilizes the source significantly more than COMET, showing a large score drop when the source is corrupted.
Quantitative Findings: Reference replacement remained 1.50 times more damaging than source replacement.
Consistency: Reference dominance persisted in 96.18% of examples.
Interpretation: Despite stronger source modeling, XCOMET-XXL remains systematically reference-dominated. The reference still acts as the stronger signal for adequacy judgments.

3. MetricX-24-Hybrid-XL

Direction-Dependent Behavior: Unlike the other two metrics, MetricX exhibits a sharp asymmetry based on translation direction.
Into-English: Strongly reference-dominated (Ref/Src ratio of 1.68×; reference dominance in 95.59% of examples).
Out-of-English: Approximately balanced or slightly source-dominated (Ref/Src ratio of 0.99×; reference dominance in only 46.86% of examples).
Interpretation: MetricX's behavior suggests a reliance on English-centric features or training data composition, as the bias disappears when English is not the target language.

4. LLM-as-a-Judge

The paper reviews findings (e.g., Huang et al., 2024) indicating that current LLM judges also exhibit reference bias. When provided with source, reference, and hypothesis, LLMs often rely more heavily on the reference (R-T configuration) than the source (S-T configuration) to achieve higher correlation with human scores.
Removing the source to optimize correlation is argued to be a failure to perform source-grounded adequacy assessment, not a valid solution.

Key Contributions

Conceptual Reframing: The paper argues that "source-free" evaluation is not merely a fallback for missing references but is structurally incapable of measuring adequacy. It distinguishes between source access (having the input) and source grounding (the source governing the decision).
Diagnostic Tool: The introduction of the counterfactual replacement diagnostic provides a rigorous method to measure whether a metric's decision boundary is governed by the source or the reference.
Empirical Evidence: The study demonstrates that even state-of-the-art hybrid metrics (COMET, XCOMET-XXL, MetricX) are predominantly reference-sensitive, with COMET showing the most extreme bias.
Critique of Current Practices: The paper challenges the community's reliance on reference-based metrics as the primary standard for adequacy, noting that they often conflate "similarity to a human rendering" with "faithfulness to the source."

Significance and Claims

The paper claims that the current standard for MT evaluation is structurally incomplete for assessing adequacy. Its primary significance lies in shifting the paradigm of how metrics should be designed and evaluated:

Reframing Quality Estimation (QE): The authors call for QE (typically used when references are missing) to be reframed as the primary approach for source-grounded adequacy evaluation, rather than a fallback.
Design Requirements: Future hybrid metrics must be designed to explicitly prioritize source–hypothesis faithfulness, using references only as complementary evidence. This requires training strategies that prevent the reference from overriding the source (e.g., counterfactual training, random reference removal).
Evaluation Standards: Metric evaluation should move beyond simple correlation with human scores. Developers must report source grounding via ablation studies (source vs. reference corruption) and performance on source-dependent challenge sets.
Reporting Norms: MT papers should not present source-free scores as complete measures of translation quality. Claims about adequacy must be supported by evaluators whose use of the source has been empirically demonstrated.

The authors conclude that while references remain valuable as auxiliary evidence, they must never override a valid source–hypothesis relationship. The source must remain the primary authority in adequacy evaluation.