💬 NLP

Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages

This paper introduces ITEM, a large-scale benchmark evaluating 29 automatic metrics across six major Indian languages, revealing that LLM-based evaluators align best with human judgments while highlighting distinct metric behaviors in summarization versus translation and the significant impact of outliers on evaluation reliability.

Original authors: Amir Hossein Yari, Kalmit Kulkarni, Ahmad Raza Khan, Fajri Koto

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amir Hossein Yari, Kalmit Kulkarni, Ahmad Raza Khan, Fajri Koto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading essays written by students in six different Indian languages. You want to know if your automated grading machine (the "metric") is doing a good job. Usually, these machines were trained and tested mostly on English essays. This paper asks: Do these machines work just as well when grading Hindi, Bengali, Tamil, and other Indian languages?

The authors built a massive testing ground called ITEM (Indian Text Evaluation Metrics Testbed) to find out. Here is what they discovered, explained through simple analogies:

1. The Problem: The "English-Only" Ruler

Imagine trying to measure the length of a snake using a ruler designed only for measuring snakes in a specific zoo in Europe. It might work okay, but if the snake is a different species or lives in a different jungle (like India), the ruler might give you the wrong number.

For years, the tools used to grade AI translations and summaries were built and tested almost exclusively on English. The authors realized that for over 1.5 billion people speaking Indian languages, we were using rulers that might not fit. They wanted to see if these tools were actually reliable or if they were just guessing.

2. The Experiment: The "Human vs. Machine" Taste Test

To test this, the researchers created a giant dataset called ITEM.

The Ingredients: They gathered 150 news articles and sentences for six major Indian languages (Hindi, Bengali, Marathi, Gujarati, Tamil, and Telugu).
The Cooks: They asked different AI chefs (like GPT-4, Llama, and specialized Indian models) to either translate the text or summarize it.
The Tasters: Real human experts tasted (read) the results and gave them a score from 1 to 5, judging things like:
- Translation: Did it say the right thing? (Adequacy) Did it sound natural? (Fluency)
- Summarization: Was it true? (Faithfulness) Did it keep the main points? (Focus/Coverage) Did it flow well? (Coherence)

Then, they ran the automated "grading machines" on the same text to see if the machines agreed with the human tasters.

3. The Big Discoveries

🏆 The New Champion: The "Super-Reader" (LLMs)

The study found that the old-school grading tools (like counting how many words match) were often out of touch.

The Analogy: Think of old metrics like a word-counting robot. It just checks if you used the right words, even if the sentence makes no sense.
The Winner: The new champions are Large Language Models (LLMs). These are like super-readers who actually understand the story, the tone, and the nuance. They agreed with the human tasters much more than any other tool. In fact, they were the most reliable judges across the board.

🎯 Different Jobs, Different Skills

The researchers found that the tools are good at different things depending on the task:

For Summarization (Condensing a story): The tools are great at checking if the content is there (did you keep the main facts?). However, they struggle to tell if the summary flows logically (coherence). It's like a tool that checks if you included all the ingredients in a cake but can't tell if the cake tastes good.
For Translation (Changing languages): The tools are excellent at checking if the sentence sounds natural (fluency). They are okay at checking if the meaning is right, but they aren't perfect at catching deep semantic errors.

🌪️ The "Outlier" Problem: One Bad Apple

The study looked at what happens when the data has "outliers"—weird, extreme examples that don't fit the norm.

The Analogy: Imagine you are measuring the height of a group of people. If you accidentally include a 10-foot-tall giant in the group, your average height calculation goes haywire.
The Finding: The researchers found that these "weird" examples can trick the grading machines. When they removed these outliers, the agreement between the machines and humans changed significantly. Some tools that looked great suddenly looked bad, and vice versa. This means previous studies that didn't check for these "weird apples" might have been wrong.

🧱 The "Robustness" Test: Can the Tool Handle Noise?

Finally, they tested if the tools could handle "noise"—like scrambling words, changing a word to its opposite, or swapping names.

The Analogy: Imagine a security guard (the metric) checking a list of names. If you scramble the letters in a name, does the guard still know it's the same person?
The Finding: Some tools were very fragile. If you shuffled the words in a sentence, their scores crashed. Others, like the "Super-Readers" (LLMs), were more robust and understood that the meaning was still there despite the jumbled words. Interestingly, the tools reacted differently depending on the language; for example, Tamil and Telugu were more sensitive to certain changes than Hindi.

4. The Conclusion

The paper concludes that we can no longer rely on the old, simple tools (like counting matching words) to grade AI in Indian languages.

The Takeaway: We need to use the "Super-Readers" (LLMs) because they understand the language better.
The Warning: We must be careful about "weird" data points (outliers) that can skew results, and we need to test our tools against "noise" to make sure they are truly reliable.

In short, the paper built a new, fairer testing ground for Indian languages and proved that to grade AI correctly in these languages, we need smarter, more human-like evaluators, not just word-counters.

Technical Summary: Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages

Problem Statement

While automatic metrics are the primary drivers of progress in Machine Translation (MT) and Text Summarization (TS), their development and validation have been almost exclusively focused on high-resource languages like English and German. This creates a significant gap for Indian languages, spoken by over 1.5 billion people, where the reliability and universality of current evaluation practices remain unverified. Existing research on Indian languages often relies on narrow sets of metrics, lacks fine-grained analysis across multiple quality dimensions, and frequently overlooks critical factors such as the impact of outliers and metric robustness under controlled perturbations. Consequently, the deployment of MT and TS systems for Indian languages often proceeds without empirical evidence of metric efficacy.

Methodology

To address these gaps, the authors introduce ITEM (Indian Text Evaluation Metrics Testbed), a large-scale benchmark designed to systematically evaluate the alignment of 29 automatic metrics with human judgments.

Dataset Construction

The benchmark covers six major Indian languages: Hindi, Bengali, Marathi, and Gujarati (Indo-Aryan family), and Tamil and Telugu (Dravidian family).

Sources: Summarization data was sampled from the XLSum corpus (BBC news), while translation data was drawn from FLORES-200.
Generation: Outputs were generated using three complementary models: two proprietary state-of-the-art LLMs (Cohere Command R+, GPT-4o mini) and two open-source models focused on Indian languages (IndicBART, IndicTrans2).
Annotation: Native speakers evaluated samples across fine-grained dimensions. For MT, the dimensions were Adequacy and Fluency. For TS, the dimensions were Faithfulness, Focus, Coverage, and Coherency.
Quality Control: A rigorous process involving expert annotators and a Median Absolute Deviation (MAD) based outlier detection method ensured data reliability. The final dataset comprises 5,175 samples (2,571 for summarization, 2,604 for translation) after filtering for high annotator agreement.

Evaluation Framework

The study evaluates 29 metrics across four categories:

Lexical/N-gram Overlap: BLEU, ROUGE (48 variants), METEOR, ChrF++, etc.
Embedding-based: BERTScore (144 variants using IndicBERT, mBERT, XLM-R), MoverScore, LaBSE, LASER.
Neural Learned: BLEURT, COMET, xCOMET.
LLM-based: GPTScore, MetricX-24, GEMBA, and direct evaluations using GPT-4.1, LLaMA-3.3, DeepSeek V3, and Gemini 2.5.

The evaluation covers:

Alignment: Pearson correlations between metric scores and human judgments at segment and system levels.
Outlier Sensitivity: Analysis of how removing outliers (defined as $|z| > 3.5$ using MAD) affects correlations.
Robustness: Testing metric resilience against controlled perturbations, including paraphrasing, entity substitution/deletion, negation, syntactic reordering (shuffling), and lexical noise.

Key Results

1. Metric Alignment with Human Judgments

LLM Dominance: LLM-based evaluators (specifically DeepSeek-V3 and GPT-4.1) demonstrated the strongest alignment with human judgments at both segment and system levels, outperforming traditional and neural-learned metrics.
Task-Specific Strengths:
- In Text Summarization, metrics were more effective at capturing content fidelity (Faithfulness, Focus, Coverage) but struggled with Coherency.
- In Machine Translation, metrics better reflected Fluency, with Adequacy being reasonably captured but finer semantic properties remaining less reliable.
Optimal Configurations: Among traditional metrics, ROUGE-2 (recall, no stopwords) performed best for TS, while BLEU-3 led for MT. For embedding-based metrics, IndicBERT (layer 8) was optimal for TS, and LaBSE for MT.

2. Impact of Outliers

Outliers significantly influence metric-human correlations. Removing outliers generally reduced correlations for Faithfulness and Focus in TS, suggesting previous inflation, while improving correlations for Coverage in some cases.
Embedding-based metrics (e.g., LASER, BARTScore) showed high sensitivity to outliers, whereas neural and LLM-based evaluators demonstrated greater resilience.

3. Language-Specific Variations

Telugu consistently achieved the highest correlation scores across tasks.
Marathi (in TS) and Hindi (in MT) ranked lowest. Notably, Hindi, despite being a high-resource language, was evaluated less reliably by LLMs than lower-resource languages like Tamil and Telugu, contradicting assumptions of superior performance in high-resource settings.

4. Robustness to Perturbations

Structural Noise: Shuffling (word order perturbation) was the most disruptive factor for MT metrics, causing sharp drops in Adequacy and Fluency correlations.
Semantic Noise: Truncating key sentences had the greatest negative impact on TS metrics regarding Faithfulness and Coverage.
Language Sensitivity: Tamil and Telugu showed high sensitivity to negation perturbations, while Hindi was the most vulnerable overall and Gujarati the most resilient.
Metric Robustness: COMET showed robustness to meaning-preserving variations (paraphrasing, synonyms) but limitations under structural and meaning-altering perturbations.

Significance and Contributions

The paper claims to provide the first unified, high-resolution evaluation framework for both MT and TS in Indian languages that goes beyond single overall scores. Its primary contributions are:

ITEM Benchmark: A publicly released dataset covering six major Indian languages with fine-grained annotations for MT and TS.
Comprehensive Metric Analysis: A systematic evaluation of 29 metrics across diverse configurations, identifying that LLM-based evaluators currently offer the most reliable alignment with human judgment.
Outlier and Robustness Insights: The study highlights the previously overlooked impact of outliers on correlation measures and provides empirical evidence of metric vulnerabilities under specific linguistic perturbations (e.g., shuffling, negation) in Indian languages.
Guidance for Future Design: The findings suggest that future metric design for Indian languages must account for the trade-off between content fidelity and linguistic quality, and prioritize robustness against structural noise, particularly for languages with flexible word orders and complex morphology.

The authors conclude that while LLM-based evaluators are currently the most reliable, their reliance on proprietary, resource-heavy systems necessitates the development of transparent, efficient, and robust alternatives for the Indian language ecosystem.