💬 NLP

Contrastive ESA: Human Evaluation of Multiple Translations at Once

The paper introduces Contrastive Error Span Annotation (cESA), a human evaluation protocol that assesses multiple translations simultaneously to reduce annotator noise and time while producing absolute quality scores for interpretable model rankings.

Original authors: Vilém Zouhar, Roman Grundkiewicz, Sara Rajaee, Parker Riley, Martin Popel, Rachel Bawden, Philipp Koehn, Marine Carpuat, Tom Kocmi

Published 2026-07-30

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Vilém Zouhar, Roman Grundkiewicz, Sara Rajaee, Parker Riley, Martin Popel, Rachel Bawden, Philipp Koehn, Marine Carpuat, Tom Kocmi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a massive talent show, but instead of singers, you are judging translations of stories from one language to another. For years, the standard way to do this was to hand the judge one single translation, ask them to grade it in isolation, and then move to the next. It's like asking someone to rate a single apple without seeing any other apples to compare it to. The problem? It's exhausting, expensive, and surprisingly messy. When you look at just one thing alone, your brain gets tired, your mood changes, and your grades can be all over the place. This is the world of "Machine Translation Evaluation," a field dedicated to figuring out how well computers can translate human languages. The goal is simple: find the best AI translator so we can trust it with real-world tasks. But to do that, humans have to act as the referees, and the old rules of refereeing were making the game slow and the scores unreliable.

Enter a team of researchers who decided to shake up the rules of the game. They introduced a new protocol called Contrastive Error Span Annotation (cESA). Instead of showing the judge one translation at a time, they put three (or more) translations side-by-side on the same screen. It's like giving the judge a basket of apples instead of a single fruit. Suddenly, spotting a bruised apple becomes much easier because you can see the shiny, perfect ones right next to it. The researchers found that this "group viewing" method didn't just make the judges faster; it made them more consistent and accurate. They discovered that showing three translations at once was the sweet spot, saving about 31% of the time while actually improving the quality of the scores. They also checked to make sure that seeing the other translations didn't trick the judges into being unfair, and found that the bias was so tiny it didn't matter. In short, they proved that when you let judges compare options together, everyone gets a better, faster, and fairer result.

The Old Way: The Lonely Apple

For a long time, the standard way to test AI translators was "pointwise evaluation." Imagine you are grading a stack of essays. In the old method, you would pick up one essay, read it, give it a score, put it down, and then pick up the next one without ever looking back. You might give the first essay an 85 because it seemed good, but then you might give the second one a 90, even if it was actually worse, just because you were tired or the first one was really bad. This is called "annotation noise." It's like trying to judge the temperature of a cup of coffee by touching it once, in a cold room, without a thermometer. The paper notes that this method suffers from high "annotator noise" and is very costly because you need so many judges to get a reliable average.

The New Way: The Fruit Basket

The authors, led by Vilém Zouhar and colleagues, proposed a new method called cESA. Instead of looking at one translation in a vacuum, the annotator sees multiple translations of the same source text at the same time. Think of it as a "fruit basket" approach. If you see a slightly bruised apple next to a perfect one, you notice the bruise much faster than if you were looking at the bruised apple alone.

In this new system, the judge looks at a source text (like a sentence from a story) and sees, say, three different AI models' translations of it side-by-side. The judge then:

Marks specific "error spans" (the exact words that are wrong).
Decides if the error is "major" (a big mistake) or "minor" (a small slip).
Gives a score from 0% to 100% based on a clear scale.

The paper explains that this works because of a psychological concept called "joint vs. separate evaluation." When you see things together, you can spot mistakes in one translation by comparing them to the others. It also helps you understand the "space of possible translations," making your judgment more objective. Plus, you don't have to re-read the original source text over and over again for every single translation, which saves a huge amount of time.

The Experiment: Testing the Fruit Basket

To see if this new method actually worked, the researchers ran a massive experiment using English-to-Japanese translations. They tested 12 different AI models (including big names like Gemini, Claude, and DeepSeek) using a dataset of 97 segments from news, social media, and videos.

They compared the old way (showing one translation at a time) with the new way (showing 1, 2, 3, or 4 translations at a time). Here is what they found:

Speed: The new method was significantly faster. When showing three models at once (the "sweet spot"), the average time to annotate a segment dropped from 77.8 seconds to 53.7 seconds. That is a 31% reduction in time per item.
Quality: The scores were more stable. The researchers measured "inter-annotator agreement" (how much two different judges agreed) and "intra-annotator agreement" (how much the same judge agreed with themselves on a second try). The new method showed higher agreement, meaning the judges were more consistent.
The Magic Number: They tested showing 1, 2, 3, and 4 models at once. They found that showing 3 models was the best balance. Going from 1 to 2 models saved a lot of time, but adding a 4th model didn't save much more time and actually made the screen a bit too crowded. Three was the perfect number to keep the judges happy and efficient.

Addressing the "What Ifs": Are the Judges Biased?

The researchers were worried that maybe seeing the other translations would trick the judges. They asked: "If a really bad translation is sitting next to a good one, will the good one look too good? Or if a great one is next to a bad one, will the bad one look too bad?"

They ran tests to check for two types of bias:

Position Bias: Does the translation on the left get a better score than the one on the right? The data showed almost no difference based on position.
Surrounding Bias: Does seeing a "decoy" (a very bad or very good translation) change the score of the others? They found a tiny effect: if a model was next to a much stronger model, its score dropped by about 0.5 points on average. However, the authors note this is so small compared to the actual differences between models (which can be 10 or 20 points) that it doesn't really matter. It's like if a runner finishes 10 seconds behind the winner, it doesn't matter if they finished 0.5 seconds slower because they were standing next to a world champion.

The Verdict

The paper concludes that cESA is a better way to judge AI translators. It is faster, cheaper, and produces more reliable scores. The authors suggest that anyone doing human evaluation of translations should switch to showing three models at a time. They even made a simple tool (called Pearmut) that anyone can use to run this new protocol, complete with tutorials and guidelines.

In the end, the paper suggests that by letting judges compare apples in a basket rather than one by one, we get a much clearer picture of which AI is truly the best. It's a simple change in how we look at the work, but it makes the whole process of judging machines much smarter and more human-friendly.

Technical Summary: Contrastive ESA: Human Evaluation of Multiple Translations at Once

Problem Statement

Current human evaluation of machine translation (MT) predominantly relies on pointwise evaluation, where annotators assess a single translation in isolation against a source input. This paradigm suffers from high annotator noise and significant costs, as the lack of comparative context makes it difficult to detect subtle errors or calibrate absolute scores consistently. While contrastive methods (ranking or side-by-side comparison) exist, they often fail to scale to document-level evaluation or require complex post-hoc statistical corrections (e.g., Bayesian modeling) to derive absolute quality metrics, making the final model rankings latent and difficult to interpret.

Methodology: Contrastive Error Span Annotation (cESA)

The authors propose Contrastive Error Span Annotation (cESA), a protocol designed to present multiple translations of the same source document simultaneously to annotators.

Core Protocol

Simultaneous Presentation: Annotators view $k$ translations of the same source document side-by-side on a single screen.
Error Marking: Annotators mark error spans within each translation, classifying them as minor or major.
Absolute Scoring: Unlike pure ranking systems, annotators assign an absolute quality score from 0% to 100% to each translation.
Score Anchors: The protocol utilizes detailed score anchors (e.g., 85–100% for "Very Good," 0–20% for "Not acceptable") and a discretized scale (steps of 5%) to guide judgment.
Iterative Review: Annotators can navigate back to previous screens to update annotations if new error patterns are discovered, addressing a limitation in standard tools.

Experimental Design

The authors validated cESA through two human evaluation campaigns using English→Japanese translation data from the WMT25 shared task (97 segments, 38 documents, diverse domains including news, social media, and video):

Small-scale: Compared standard ESA against cESA with $k=1$ (single output) to isolate the effect of improved guidelines and interface.
Large-scale: Evaluated 12 models (including 10 MT systems and 1 human reference) with varying numbers of simultaneous outputs ( $k \in \{1, 2, 3, 4\}$ ).
Metrics: The study measured annotation time, intra-annotator agreement (consistency of the same annotator), inter-annotator agreement (consistency between different annotators), and stability (ranking robustness on data subsets).

Key Results

Efficiency and Quality

Time Reduction: The optimal configuration, cESA with $k=3$ , reduced annotation time per segment by 31% compared to standard pointwise ESA (53.7s vs. 77.8s).
Noise Reduction: cESA ( $k=3$ ) demonstrated lower inter-annotator disagreement (Mean Absolute Error of 17.0 vs. 20.0 for standard ESA) and higher stability in model rankings.
Diminishing Returns: Increasing $k$ beyond 3 did not yield further time savings and slightly degraded quality metrics, suggesting $k=3$ is the optimal balance.

Bias Analysis

The authors investigated potential biases introduced by showing multiple outputs:

Position Bias: Shuffling the order of models on the screen effectively mitigated position bias; no significant distributional differences were found based on screen position.
Similarity Bias: The presence of similar model outputs did not significantly skew scores (effect size $\approx -0.2$ points).
Quality Bias (Decoy Effect): The presence of a stronger model on the same screen caused a slight, consistent downward bias in the scores of weaker models (approx. -0.5 points on average). However, the authors note this effect is negligible compared to the actual performance gaps between models and is alleviated by randomizing model selection per screen.

Calibration

The study found that showing multiple models together improves within-screen calibration. The score distribution remains consistent across configurations, but the variance in scores for the same model across different screens decreases, indicating that the shared context helps annotators apply the scoring rubric more consistently.

Key Contributions

Protocol Definition: Introduction of cESA, a standardized protocol that combines the depth of error span marking with the efficiency of contrastive evaluation while retaining absolute scores.
Scalability: Demonstration that contrastive evaluation can be scaled to document-level tasks with multiple models without requiring complex post-hoc ranking corrections.
Empirical Validation: Large-scale evidence showing that cESA reduces annotation costs and noise while improving agreement and stability compared to standard pointwise ESA.
Tooling: Release of an implementation in the Pearmut annotation platform, allowing for minimal setup deployment with tutorials in 18 language pairs.

Significance and Claims

The paper claims that cESA represents an advancement in the state-of-the-art for human MT evaluation. By leveraging the psychological benefits of joint evaluation (finding errors in one translation aids in finding them in others), the protocol enables more consistent and efficient judgments.

Crucially, the authors assert that cESA yields absolute quality judgments that allow for simple, interpretable, non-parametric model rankings (e.g., via average scores) without the need for latent parametric models like TrueSkill. The paper concludes that for evaluation practitioners, showing three model outputs ( $k=3$ ) simultaneously offers the best trade-off between speed, quality, and annotator workload. The collected scores can be used directly as averages for model comparison, provided the minor biases identified are understood to be negligible in the context of typical model performance gaps.