💬 NLP

SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages

This paper introduces SiniticMTError, a novel fine-grained dataset featuring error span, type, and severity annotations for machine-translated examples across several Sinitic languages (Mandarin, Cantonese, Wu, and Hokkien), designed to advance translation quality estimation and error-aware generation while highlighting the current limitations of large language models in detecting such errors.

Original authors: Hannah Liu, Junghyun Min, En-Shiun Annie Lee, Ethan Yue Heng Cheung, Shou-Yi Hung, Elsie Chan, Shiyao Qian, Runtong Liang, Kimlan Huynh, Wing Yu Yip, York Hay Ng, TSZ Fung Yau, Ka Ieng Charlotte Lo, Y

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hannah Liu, Junghyun Min, En-Shiun Annie Lee, Ethan Yue Heng Cheung, Shou-Yi Hung, Elsie Chan, Shiyao Qian, Runtong Liang, Kimlan Huynh, Wing Yu Yip, York Hay Ng, TSZ Fung Yau, Ka Ieng Charlotte Lo, You-Wei Wu, Richard Tzong-Han Tsai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, multilingual robot translator. It's great at translating between big, popular languages like English and Mandarin. But when you ask it to translate into smaller, regional languages like Cantonese, Wu (Shanghai dialect), or Hokkien, it starts to stumble. It might get the general idea right, but it messes up the details, sounding awkward or even making up words that don't belong.

This paper introduces a new tool called SINITICMTERROR. Think of it as a "Mistake Map" or a "Training Manual for Errors" designed specifically to teach AI how to spot and fix its own translation mistakes in these under-served Chinese languages.

Here is a breakdown of what the researchers did, using some everyday analogies:

1. The Problem: The "Overconfident Tourist"

Imagine a tourist who knows a little bit of a local language. They can order food and ask for directions, but if you ask them to describe the weather, they might say, "The sky is very beautiful" (which is grammatically okay but sounds weird in that culture, where you'd just say "The weather is nice").

Current AI models are like that tourist. They are fluent enough to sound natural, but they often make subtle, "silly" mistakes:

Adding things that aren't there: Like the tourist adding "I think" when the original sentence didn't include it.
Using the wrong word: Translating "beautiful" for a weather report when the local language expects "good."
Missing the point: Leaving out crucial words entirely.

For languages like Cantonese, Wu, and Hokkien, these mistakes are common because there isn't enough high-quality data to train the AI properly.

2. The Solution: The "Mistake Map" (SINITICMTERROR)

The researchers built a massive dataset where they took thousands of sentences, had an AI translate them, and then hired native speakers (the "local experts") to go through the translations with a red pen.

They didn't just say "this is wrong." They created a detailed log for every single error:

Where is the error? (Pinpointing the exact words).
What kind of error is it? (Did the AI add a word? Did it forget a word? Did it use the wrong grammar?).
How bad is it? (Is it a minor typo, or did it completely change the meaning?).

The Analogy: Imagine a teacher grading a student's essay. Instead of just giving a "C," the teacher highlights every specific sentence, writes "Grammar error here," "Too formal here," and "You missed a key detail there." SINITICMTERROR is that detailed grading sheet, but for AI.

3. The Languages: The "Family Reunion"

The paper focuses on four specific branches of the "Sinitic" (Chinese) language family:

Mandarin: The big, popular cousin that everyone knows.
Cantonese: The rich, expressive cousin spoken in Guangdong and Hong Kong.
Wu (Shanghainese): The sophisticated cousin from Shanghai.
Hokkien: The ancient, dialect-heavy cousin spoken in Fujian, Taiwan, and Southeast Asia.

The researchers found that while Mandarin is getting better, the other three are still struggling. The AI often treats them like "broken" versions of Mandarin, rather than unique languages with their own rules. For example, Cantonese has many tiny "particles" at the end of sentences to show attitude, and the AI keeps dropping them or using the wrong ones.

4. The Test: "Can the AI Grade Itself?"

The researchers didn't just make the map; they tested if current super-smart AI models (like GPT-4 or Gemini) could use this map to find errors.

The Result: It was a bit of a disaster. Even the smartest AI models only got about 30% to 40% of the errors right.

The Metaphor: It's like asking a brilliant math student to grade a test on a subject they've never studied. They can see something is wrong, but they can't pinpoint exactly what or how bad it is. They often miss the subtle cultural nuances that a native speaker would catch immediately.

5. Why This Matters

This dataset is a game-changer for two reasons:

It's a Training Gym: Now, developers can use this "Mistake Map" to train their AI models to become better at spotting errors, much like a coach using game tape to show a player where they missed the ball.
It Gives a Voice to the "Small" Languages: By treating these languages with the same rigorous attention as English or Mandarin, the paper argues that these languages deserve better technology. It highlights that current AI is biased toward "big" languages and needs to learn to respect the nuances of the "small" ones.

In a Nutshell

The paper says: "AI is getting better at translation, but it's still clumsy with regional Chinese languages. We've built a detailed 'error atlas' to show exactly where it trips up, and we've proven that even the smartest AI can't fix these mistakes without human help. Now, we have the map to teach them how to walk properly."

1. Problem Statement

Despite significant advancements in Machine Translation (MT) driven by Large Language Models (LLMs) and supervised systems, progress remains limited for low-resource Sinitic languages. While Mandarin Chinese is well-supported, other major Sinitic variants—specifically Cantonese (Yue), Wu Chinese, and Hokkien (Min Nan)—suffer from:

Data Scarcity: A lack of large-scale, high-quality parallel corpora.
Lack of Standardization: Inconsistent writing systems and orthographies (e.g., written Cantonese is often informal).
Poor Model Performance: Fine-tuned models often perform poorly, and LLMs frequently output noise or "translationese" (unnatural phrasing) for these languages.
Evaluation Gap: There is a critical absence of fine-grained, human-annotated datasets to benchmark translation quality estimation (QE) and error detection for these specific language pairs.

2. Methodology

The authors introduce SINITICMTERROR, a novel fine-grained dataset designed to bridge these gaps. The methodology involves four key stages:

A. Data Construction & Model Selection

The dataset consists of triplets: Source (English/Mandarin) $\to$ Machine Translation (MT) $\to$ Reference.

Source Data: Primarily derived from FLORES+ (an extension of FLORES-200) for English-to-Mandarin, Cantonese, and Wu. For Hokkien, data was sourced from the Ministry of Education Dictionary (Mandarin-to-Hokkien).
MT Generation: Different models were selected based on capability and reproducibility for each language:
- Mandarin & Cantonese: NLLB-200 (600M parameters). Selected after benchmarking multiple models (including Llama 3, Qwen, Aya) for its ability to generate Cantonese with the lowest (most error-prone) SacreBLEU/ChrF++ scores while maintaining fluency.
- Wu Chinese: Qwen2.5 Max (the only publicly accessible LLM with stable Wu capabilities).
- Hokkien: Taigi-LLaMA-2-Chat-7B (a specialized model trained on Taiwanese Hokkien).

B. Annotation Framework

Annotations follow the Multidimensional Quality Metrics (MQM) framework and the AfriCOMET adaptation.

Granularity: Annotators identify specific error spans within the MT output.
Labels:
- Error Types (9 categories): Mistranslation, Omission, Addition, Untranslated, Grammar, Spelling, Typography, Unintelligible, and Register.
- Severity (2 levels): Major (significant meaning change) and Minor (fluency/readability issues).
Process: Native speakers proficient in English conducted the annotation using the TRANSLATIONCORRECT tool. The process included multi-stage pilot studies (4 rounds for Cantonese/Wu) to ensure inter-annotator agreement and refine guidelines.

C. Baseline Evaluation

The authors evaluated the performance of various LLMs (both open-source like Gemma, Phi, Qwen, and closed-source like GPT-4o, Gemini) on the task of automatic error detection.

Metrics: Span Precision (SP), Major Precision (MP), and correlation metrics (Pearson's $r$ , Spearman's $\rho$ , Kendall's $\tau$ ) between LLM-derived MQM scores and human references.
Protocol: A "tag-to-span" generation protocol was used to ensure precise alignment between model-declared error regions and character offsets.

3. Key Contributions

SINITICMTERROR Dataset: The first large-scale, human-annotated resource providing span-level error types and severity for Cantonese, Wu Chinese, and Hokkien, complementing existing Mandarin resources.
- Scale: 2,009 Mandarin sentences, 1,402 Cantonese sentences, 68 Wu sentences (pilot), and 154 Hokkien sentences (pilot).
Error Analysis: Detailed linguistic insights into the specific failure modes of MT in Sinitic languages, including:
- Cantonese: High frequency of typography and omission errors; struggles with functional particles and serial verb constructions.
- Mandarin: Dominated by mistranslation and grammar errors.
- Translationese: MT outputs often retain English clause structures, leading to unnatural phrasing in Chinese languages.
- Language Confusion: Instances where models output Japanese or Mandarin words in Hokkien/Cantonese contexts.
Benchmarking: Established baseline results showing that current state-of-the-art LLMs (even GPT-4o and Gemini) struggle with fine-grained error detection in low-resource Sinitic languages, achieving low span precision (0.25–0.45) and weak correlation with human judgments.

4. Key Results & Statistics

Error Distribution:
- Cantonese exhibits a much higher error span density (6.38 spans/sentence) compared to Mandarin (1.93 spans/sentence), indicating significantly worse model performance.
- Mandarin: Most common errors are Mistranslation (61.2%), Omission (14.2%), and Grammar (7.1%).
- Cantonese: Most common errors are Mistranslation (33.8%), Typography (26.3%), and Omission (21.6%).
- Wu/Hokkien: Pilots show high rates of Mistranslation and Register errors.
Severity: Minor errors are more frequent than Major errors across all languages, suggesting models often produce fluent but semantically slightly off outputs.
LLM Baselines:
- Span Precision: Ranged from 28.1% to 59.7% across models.
- Correlation: Segment-level MQM score correlations with human judgments remained below 0.5 (e.g., GPT-4o achieved $r=0.43$ for English-Mandarin but only $r=0.19$ for English-Cantonese).
- Conclusion: Current LLMs lack the specialized training required for fine-grained error analysis in these low-resource settings.

5. Significance

Resource for Low-Resource NLP: SINITICMTERROR fills a critical void in the NLP ecosystem for Sinitic languages, moving beyond Mandarin to include varieties spoken by over 170 million people combined.
Training & Evaluation: The dataset enables the fine-tuning of models for error-aware generation and translation quality estimation (QE), specifically for languages where data is scarce.
Linguistic Insights: The paper highlights that "low-resource" challenges in Sinitic languages are not just about data volume but also about orthographic standardization and dialectal variation, requiring specialized annotation guidelines.
Future Directions: The dataset serves as a foundation for transfer learning, allowing models trained on high-resource languages to be adapted more effectively to Sinitic varieties, and supports downstream tasks like language detection and linguistic acceptability judgment.

In summary, SINITICMTERROR provides the necessary infrastructure to diagnose, understand, and improve machine translation for underrepresented Chinese dialects, demonstrating that current AI models still struggle significantly with the nuances of these languages without targeted, human-annotated supervision.