💬 NLP

Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?

This paper investigates the reliability of existing automatic evaluation metrics for Classical Chinese to English translations, revealing that while all metrics have blind spots, MetricX24 performs best, thereby highlighting the urgent need for more robust and interpretable metrics tailored to historically and culturally distinct translation settings.

Original authors: Osvaldo Quinjica, Eric Bennett, Xinchen Yang, Andrew Schonebaum, Marine Carpuat

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Osvaldo Quinjica, Eric Bennett, Xinchen Yang, Andrew Schonebaum, Marine Carpuat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the only clues you have are written in a language that hasn't been spoken for two thousand years. This is the daily reality for scholars studying Classical Chinese, the ancient language of emperors, philosophers, and poets. Today, we have super-smart computer brains called Large Language Models (LLMs) that can read these ancient texts and translate them into modern English. It's like having a magical time machine that turns dusty scrolls into readable stories. But here's the catch: how do we know the machine isn't making things up? If the computer translates a word for "crow" as "pigeon," or changes a king into a duke, the whole story could be ruined.

To check if the computer is doing a good job, scientists use "evaluation metrics." Think of these metrics as automated judges or spell-checkers that give the translation a score. Usually, these judges are trained on modern languages like French or Spanish. They look for things like matching words or similar sentence structures. But Classical Chinese is a wild card. It's incredibly short, relies heavily on context, and often leaves out subjects or objects that modern languages require. A translation that sounds perfect to a modern English speaker might actually be a complete lie about what the ancient text said. The big question is: Can our current automated judges spot these ancient-specific mistakes, or are they too busy looking for modern grammar rules to notice the historical errors?

This paper puts those automated judges to the test. The researchers created a special "exam" for translation metrics using a clever trick called "minimal pairs." Imagine you have a perfect translation of an ancient sentence. Now, you take that sentence and make one tiny, specific change—like swapping a "crow" for a "pigeon" or removing the name of the person speaking. This creates a "broken" version that is almost identical to the original but contains a specific error. The researchers then fed both the perfect version and the broken version to various translation metrics to see if the metrics would give the broken one a lower score.

The results were a bit of a mixed bag, revealing that even the smartest automated judges have "blind spots." The study found that while some metrics are good at catching obvious disasters (like translating the whole text into modern Chinese instead of English), they often fail to catch the subtle, dangerous errors that matter most to historians. For instance, many metrics didn't notice when a translation got the tense wrong (changing "is" to "was") or when a specific title like "Duke" was swapped for "King." These are the kinds of mistakes that could lead a scholar to the wrong conclusion about history.

Among all the judges tested, one called MetricX24 performed the best overall. It was the most sensitive to errors, correctly penalizing many of the broken translations. However, even MetricX24 wasn't perfect; it still missed certain types of errors, particularly those involving modern versus ancient meanings of words. On the flip side, the metrics were generally very good at one thing: they didn't punish harmless changes. If the translation used a slightly different way to format a name or broke a sentence in a different but valid place, the metrics mostly gave it a pass. This is good news, as it means the judges aren't too strict about style.

Ultimately, the paper suggests that we can't just rely on the current tools to evaluate ancient translations. The metrics are like a pair of glasses designed for modern streets; they work well there, but they get blurry when you try to navigate the narrow, winding alleys of ancient history. The authors propose that to get reliable translations for digital humanities, we need new, smarter evaluation tools that are specifically trained to understand the unique quirks of Classical Chinese, rather than just applying modern rules to ancient texts. Until then, human experts will still need to double-check the work of our AI time machines.

Technical Summary: Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?

Problem Statement
While Large Language Models (LLMs) have demonstrated surprising proficiency in translating historical languages like Classical Chinese (wenyanwen), their integration into digital humanities workflows is hindered by a lack of reliable evaluation methods. Current automatic evaluation metrics are primarily validated on modern languages and may not generalize to the unique linguistic and cultural characteristics of Classical Chinese. These characteristics include extreme conciseness, reliance on context, pervasive zero anaphora (omitted subjects/objects), and polysemy where meaning is determined by discourse rather than morphology. Consequently, existing metrics risk overlooking subtle, context-dependent errors that distort historical meaning while appearing fluent, or conversely, penalizing legitimate paraphrastic variations essential to scholarly interpretation.

Methodology
The authors introduce a diagnostic framework to probe the sensitivity of machine translation (MT) evaluation metrics to specific error types and acceptable variations in Classical Chinese-to-English translation.

Data Construction: Source texts were drawn from the Chinese Text Project (CTP), with James Legge's translations serving as high-quality human references. Baseline machine translations were generated using GPT-4o-mini. A memorization analysis confirmed that results were not driven by training data overlap.
Error Taxonomy: Through expert philological inspection of LLM outputs, the authors defined eight error categories relevant to scholarly use:
1. Unintended translation into Modern Chinese.
2. Omitted object.
3. Omitted subject.
4. Incorrect lexical translation.
5. Pronoun substitution/misattribution.
6. Incorrect tense realization.
7. Title or rank substitution.
8. Modern-sense substitution (replacing a classical sense with a modern one).
  Additionally, three categories of acceptable variations were defined: name formatting, name annotation, and sentence segmentation/minor paraphrasing.
Perturbation Generation: A semi-automatic pipeline generated "minimal pairs" for each baseline translation. These pairs consisted of the original translation and a minimally edited variant introducing exactly one targeted error or variation. This process involved rule-based edits, LLM-based rewrites, and manual vetting by experts trained in both modern and Classical Chinese.
Evaluation Setup: The study evaluated a suite of metrics across three families:
- Surface Overlap: BLEU, chrF.
- Learned Neural Metrics: COMET-22, XCOMET, MetricX24 (and their Quality Estimation variants).
- LLM-as-a-Judge: GEMBA (using DA and MQM instructions).
  Metrics were tested in reference-based (src+ref+hyp, ref+hyp) and reference-free (src+hyp) settings.
Statistical Analysis: The authors employed a linear mixed-effects (LME) model to analyze standardized score deltas ( $\delta^*$ ) between original and perturbed translations. This approach allowed for the quantification of metric sensitivity across different error types while accounting for repeated observations from the same source segment.

Key Results

General Validity: All metrics successfully distinguished reasonable translations from catastrophic errors (e.g., word-shuffled or unrelated translations).
Error Sensitivity:
- Language Mismatch: Most reference-based metrics correctly penalized translations that inadvertently output Modern Chinese instead of English. However, MetricX24 and XCOMET-QE failed to detect this in reference-free settings.
- Omissions: Neural metrics (COMET, XCOMET, MetricX24) adequately detected omitted subjects and objects. In contrast, surface metrics (BLEU, chrF) and GEMBA variants often failed to penalize these omissions, sometimes scoring the erroneous translation higher than the original.
- Semantic Substitutions: No metric reliably detected incorrect tense realization or title/rank substitution. For modern-sense substitution, only MetricX24-Ref reliably detected the error. For incorrect lexical translation, XCOMET and MetricX24 variants performed well, while COMET and surface metrics failed.
- Pronoun Errors: MetricX24 variants showed the strongest sensitivity to pronoun substitution, while GEMBA showed only moderate sensitivity.
Acceptable Variations: Nearly all metrics tolerated acceptable variations (name formatting, annotation, segmentation), with the exception of BLEU, which penalized name formatting. Sentence segmentation was the most tolerated variation type.
Top Performer: MetricX24 emerged as the best overall performer, exhibiting the highest sensitivity to errors across most categories. However, it still exhibited blind spots regarding modern-sense substitution (in non-reference settings) and tense errors.
Temporal Analysis: A significant interaction was found between metric performance and the historical period of the source text (Pre-Qin/Han vs. Post-Han). However, the authors note that the Post-Han subset was small (6.4% of data), leading to larger variance and reduced statistical power for that specific group.

Significance and Claims
The paper claims that while neural metrics outperform surface-overlap baselines, no single metric reliably captures the full range of distinctions relevant to scholarly evaluation of Classical Chinese. The findings highlight that current metrics have specific blind spots regarding errors that distort historical meaning (e.g., tense, titles, and modern-sense substitutions), which are critical for scholars lacking source language proficiency.

The authors position their diagnostic framework as a primary contribution. By using scholar-grounded minimal pairs, they provide a scalable method to convert expert judgment into evaluation signals without requiring massive manual annotation. They argue that progress in this domain depends not only on better metrics but also on better evaluation resources that systematically target subtle semantic distinctions often underrepresented in standard benchmarks. The work aims to support reliable AI-assisted scholarly translation and advance the understanding of how evaluation metrics generalize to historically and culturally distinct settings.

Technical Summary: Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?

More like this