💻 computer science

IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages

This paper introduces IndicQE-APE, a consolidated benchmark aggregating 126,754 instances across nine Indic language pairs with multi-faceted annotations to evaluate and compare the performance of LLMs and COMET metrics on quality estimation and automatic post-editing tasks.

Original authors: Diptesh Kanojia, Archchana Sindhujan, Sourabh Deoghare, Daria Sokova, Shenbin Qian, Girish Koushik, Tharindu Ranasinghe, Constantin Orăsan, Chrysoula Zerva, Ricardo Rei, Frédéric Blain, André F. T. Ma

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Diptesh Kanojia, Archchana Sindhujan, Sourabh Deoghare, Daria Sokova, Shenbin Qian, Girish Koushik, Tharindu Ranasinghe, Constantin Orăsan, Chrysoula Zerva, Ricardo Rei, Frédéric Blain, André F. T. Martins, Marco Turchi, Matteo Negri, Rajen Chatterjee, Anoop Kunchukuttan, Mitesh M. Khapra, Pushpak Bhattacharyya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

When a computer translates a sentence from one language to another, how do we know if the result is any good? For decades, the standard answer was to compare the machine's output against a human-written version of the same text. If the two matched closely, the machine was doing well. But this method hits a wall when dealing with languages that have fewer digital resources, such as the many languages spoken across the Indian subcontinent. In these cases, there often is no perfect human translation to use as a yardstick. Instead, researchers must rely on "quality estimation," a way of scoring a translation based only on the original text and the machine's guess, without a reference. Even more challenging is "automatic post-editing," where a computer tries to fix its own mistakes to bring the text closer to human standards. For years, data for these tasks in Indic languages has been scattered across different projects, making it difficult to train systems that work well across the board or to compare them fairly.

A new study brings order to this chaos by creating a single, massive resource called INDICQE-APE. The researchers gathered and unified data from several major international translation competitions spanning four years, combining them with a newly created dataset for English to Malayalam. The result is a collection of nearly 127,000 examples covering nine different language pairs. What makes this collection unique is that every single example carries multiple types of human feedback at once: a score for how good the translation is, a version of the text that a human has edited to fix errors, and even notes explaining what went wrong. This allows researchers to test how well computers can judge quality and how well they can fix their own work using the exact same set of sentences.

The team used this new benchmark to test a variety of modern artificial intelligence systems, including large language models and specialized scoring tools. They discovered a surprising flaw in how these systems are currently evaluated. While many models are excellent at ranking translations within a single language pair—telling a Hindi translation from a bad one—they often fail to be comparable across different languages. A score of 80 for a Hindi translation does not mean the same thing as a score of 80 for a Tamil translation. In fact, for some of the most powerful models tested, the scores actually move in the wrong direction when comparing different languages, making it impossible to create a single leaderboard that ranks performance fairly across all pairs.

The researchers also investigated what makes a translation particularly difficult for a computer to evaluate. They categorized the data into four types of difficulty, such as segments where human annotators disagreed on the score, or where the translation required a lot of editing. They found that most of these categories did not actually make the task harder for the computer once they accounted for the language and the general quality of the text. However, one specific type of difficulty stood out: segments where the overall impression of the translation seemed fine, but the individual words were clearly wrong, or vice versa. When a translation sent a mixed signal—looking good at a glance but failing on the details—every single system tested struggled to rank it correctly. This suggests that the biggest challenge for current technology is not just the complexity of the language, but the contradiction between the big picture and the small details.

In addition to testing existing tools, the researchers tried to build new, lightweight systems to see if they could improve on the results. They found that simply reading the internal mathematical states of a frozen large language model could predict quality almost as well as asking the model to write a score. They also trained a small add-on to a standard translation evaluator, which performed just as well as the much larger, pre-made tools. These findings suggest that the key to better quality estimation lies not in building bigger models, but in better understanding how to read the signals already present in the data.

The study also shed light on the practice of automatic post-editing. When the researchers asked computers to fix machine-translated text, the results were counterintuitive. In three out of four language pairs, the original, unedited machine translation was actually closer to the human standard than the version the computer tried to fix. This happened because the computers tended to make changes that moved the text further away from the specific style of the human editors, even if the new version was grammatically correct. The human editors had often made very minimal changes, and the computers, trying to be helpful, over-corrected. This reveals a critical gap: current systems are not yet ready to replace human editors because they do not yet understand the subtle, conservative nature of human editing in these languages.

Ultimately, this work provides a much clearer map of the landscape for machine translation in Indic languages. It shows that while technology has made great strides in understanding individual languages, it still lacks the ability to judge quality fairly across different tongues. The new benchmark serves as a rigorous testing ground that exposes these weaknesses, particularly the confusion caused by mixed signals in translations and the difficulty of matching human editing styles. By consolidating years of scattered data into one coherent resource, the researchers have given the field a solid foundation to build upon, ensuring that future progress is measured against a standard that is both comprehensive and honest about where the technology currently stands.

Technical Summary: INDICQE-APE

Problem Statement

Reference-free evaluation of machine translation (MT) for Indic and other low-resource languages faces two compounding challenges: data fragmentation and inconsistent evaluation metrics.

Data Fragmentation: Quality Estimation (QE) and Automatic Post-Editing (APE) data for Indic languages are scattered across separate WMT shared-task releases (2020–2024) and other resources. These datasets possess incompatible schemas, splits, and provenance, preventing unified training and evaluation across tasks and language pairs.
Metric Inconsistency: Existing evaluations often rely on a single learned scalar metric. The behavior of these metrics across different languages is rarely checked, leading to the assumption that within-language accuracy implies cross-lingual comparability, which this paper challenges.

Methodology: The INDICQE-APE Benchmark

The authors introduce INDICQE-APE, a consolidated benchmark designed to unify these fragmented resources and enable rigorous measurement.

Data Construction

Scope: The benchmark consolidates 126,754 instances across nine directional language pairs (English to {Hindi, Marathi, Tamil, Telugu, Gujarati, Malayalam} and {Estonian, Nepali, Sinhala} to English).
Alignment: Unlike previous releases where label types were distributed across different files, INDICQE-APE aligns up to four label types on the same segment:
1. Direct Assessment (DA): Z-normalized 0–100 scores based on FLORES guidelines.
2. Human Post-Edit (PE): Human corrections used as the reference for APE.
3. Word-level Tags: OK/BAD tags derived from MT-to-PE alignment (supplemented by derived tags where real tags are missing).
4. Explanations: Human Translation Quality Remarks (for English-Malayalam) and model-generated error descriptions.
Provenance & Hygiene: Every instance is tagged with sources and cadence_id. The dataset undergoes deduplication via content-hash UIDs and strict source-disjoint splitting to prevent train/test leakage.
Difficulty-Stratified Test Set: Instead of sampling from a training tail, the test set is constructed to be stratified across four difficulty axes:
- A1 (Annotator Disagreement): High standard deviation in annotator scores.
- A2 (Edit Effort): High Token Error Rate (TER) between MT and post-edit.
- A3 (Word Error): High density of BAD tags or specific error categories.
- A4 (Signal Conflict): Segments where holistic DA scores and token-level error signals (e.g., low TER) conflict (e.g., a low score with minimal edits, or a high score with dense errors).

Evaluation Protocol

The paper benchmarks:

QE: Six prompted Large Language Models (LLMs) and three COMET family metrics (CometKiwi-DA, CometKiwi-XL, XCOMET-XL) on segment-level correlation with human DA.
APE: Three systems (sarvam-m, sarvam-t, tiny-aya) on post-editing performance, measured against human post-edits using surface metrics (char-TER, chrF++) and reference-free metrics.
Modelling Studies:
1. Layer Probing: Training a regression head on frozen hidden states of instruction-tuned LLMs to predict quality.
2. COMET Regression: Training a lightweight regression head on a COMET encoder to predict DA, analyzing the impact of encoder initialization, features, and loss functions.

Key Results

1. Cross-Lingual Comparability vs. Within-Language Accuracy

The study finds that within-language accuracy does not guarantee cross-lingual comparability.

Per-Language Offsets: Metrics exhibit per-language offsets ( $a_\ell$ in $m(x) = a_\ell + b_\ell q(x)$ ). For CometKiwi-XL, the best-performing metric within languages, this offset correlates negatively with quality ($-0.442$).
Pooling Artifacts: Pooling language pairs into a single correlation metric inflates scores artificially by capturing inter-pair mean differences rather than intra-pair ordering. CometKiwi-XL loses 0.117 in correlation when pairs are pooled, whereas other metrics remain stable.
Conclusion: Raw scores from such metrics are not comparable across different language pairs; leaderboards should average within-pair correlations rather than pooling segments.

2. Difficulty Axes and System Performance

Signal Conflict (A4): When compared against a control group of segments with the same human score distribution from the same language pair, only the "Signal Conflict" axis (A4) consistently degrades system performance. Segments where holistic and token-level signals conflict are ranked worse than equally-scored segments without this conflict.
Other Axes: Axes defined by annotator disagreement (A1) or edit effort (A2) do not show a significant effect once the score range compression is controlled for.
Prompting Costs: Few-shot prompting (4-shot) degrades performance for smaller models ( $\le$ 3.4B parameters), reducing both correlation and output-format compliance. Larger models (e.g., GPT-5.5, aya-expanse-32B) are more robust.

3. Automatic Post-Editing (APE) Findings

The "Do-Nothing" Baseline: On three of the four evaluated pairs, the unedited Machine Translation (MT) outperforms all post-editing systems when measured by surface metrics (char-TER, chrF++) against the human post-edit.
Reference Bias: This occurs because post-edits are conservative repairs of the MT. Surface metrics against post-edits measure "residual repair effort" rather than absolute quality. Systems that retranslate from scratch (e.g., sarvam-t) depart further from the conservative post-edit and score worse, despite potentially higher quality.
Reference-Free View: Reference-free metrics (CometKiwi) rank the post-editing systems above the unedited MT, highlighting the discrepancy between reference-based and reference-free evaluation in APE.

4. Modelling Insights

Layer Probing: Quality signals in frozen LLMs are best read from intermediate layers (e.g., layer -20 in a 24-layer model), outperforming prompting the same model.
Regression Heads: For training a regression head on a COMET encoder, the encoder initialization is the dominant factor. Reference-free pretraining (CometKiwi) significantly outperforms reference-based pretraining or no pretraining. Feature engineering (attention features, length ratios) provides negligible gains compared to the encoder choice.

Significance and Claims

The paper claims to provide the first unified resource for Indic QE and APE that supports training and evaluation across tasks and pairs on a single footing. Its primary contributions are:

INDICQE-APE Benchmark: A dataset of 126,754 instances with aligned multi-modal labels, verified provenance, and a difficulty-stratified test set.
Measurement of Reference-Free QE: Demonstrating that within-language correlation and cross-lingual comparability are distinct properties. It warns that raw scores from current metrics cannot be compared across language pairs due to per-language offsets.
Identification of Hard Cases: Showing that "signal conflict" (disagreement between holistic scores and token-level errors) is the primary difficulty axis that consistently challenges systems, whereas annotator disagreement and edit effort are less critical when controlled for score distribution.
APE Baselines: Establishing that unedited MT often beats post-editing systems under reference-based metrics due to the conservative nature of post-edits, necessitating a "do-nothing" baseline for valid APE evaluation.

The authors conclude that while the benchmark enables new directions (probing hidden states, lightweight regression heads), significant challenges remain, particularly in handling agglutinative languages with word-level tagging and resolving cases where human scores and surface evidence disagree.