💬 NLP

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

This paper introduces LQM, a linguistically motivated, hierarchical error taxonomy designed to diagnose machine translation errors across sociolinguistic, pragmatic, and linguistic dimensions, which is validated through a large-scale, dialect-specific Arabic corpus and expert human annotation to address the limitations of existing language-agnostic evaluation frameworks.

Original authors: Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki, Wesam El-Sayed, Muhammad Abdul-Mageed

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki, Wesam El-Sayed, Muhammad Abdul-Mageed

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a translator to help you communicate with a friend who speaks a very specific, local version of a language—like a street-smart cousin from a specific neighborhood, rather than a professor from a university.

For a long time, we've had tools to check if a translator did a "good job." But these tools were like spellcheckers. They only looked at whether the words were spelled right and if the sentence structure looked okay. They didn't care if the translator sounded like a stiff robot, used the wrong slang, or accidentally insulted your friend by using a formal greeting when a casual "hey" was needed.

This paper introduces a new, much smarter way to check translations, called LQM (Linguistically Motivated Multidimensional Quality Metrics). Think of LQM not as a spellchecker, but as a cultural detective with a six-level magnifying glass.

Here is how LQM works, broken down into simple concepts:

1. The Problem: The "One-Size-Fits-All" Trap

The authors found that existing tools (like MQM) were too generic. They treated all languages the same. But in languages like Arabic, there is a huge gap between the "official" language (like Standard Arabic) and the many local dialects (like Egyptian, Moroccan, or Emirati Arabic).

If you ask a translator to speak like a Moroccan local, but they reply in formal, textbook Arabic, a spellchecker says, "Great! No spelling mistakes!" But a human Moroccan would say, "This sounds weird; you sound like a news anchor, not a neighbor."

2. The Solution: The Six-Layer Ladder

LQM climbs a six-step ladder to find exactly where the translation broke down. Instead of just saying "This is wrong," it tells you why.

Level 1: The Social Suit (Sociolinguistics)
- The Analogy: Imagine wearing a tuxedo to a beach party. You are dressed correctly, but you are in the wrong context.
- The Error: The translator used the "official" language instead of the requested local dialect, or used a formal tone when a casual one was needed.
Level 2: The Intent (Pragmatics)
- The Analogy: Someone says, "It's cold in here." They aren't asking for a temperature report; they are hinting, "Please close the window."
- The Error: The translator understood the words but missed the point. They didn't catch the joke, the polite request, or the cultural nuance.
Level 3: The Meaning (Semantics)
- The Analogy: You ask for a "car," and they bring you a "Ferrari." Technically correct, but maybe not what you wanted. Or they bring you a "bicycle."
- The Error: The translator got the facts wrong, missed a name, or used the wrong word for a specific object.
Level 4: The Grammar (Morphosyntax)
- The Analogy: Building a house with the bricks in the wrong order. The walls might stand, but the door is on the roof.
- The Error: The sentence structure is broken, or the verb tenses don't match.
Level 5: The Spelling (Orthography)
- The Analogy: Typos and messy handwriting.
- The Error: Misspelled words or weird punctuation.
Level 6: The Glitch (Graphetics)
- The Analogy: The text is garbled, like a fax machine that ate the paper.
- The Error: The computer code broke, and the letters look like random symbols (e.g., Ã± instead of ñ).

3. The Experiment: Testing the "Detectives"

The researchers built a massive test set using 3,850 sentences from seven different Arabic dialects (like Egyptian, Moroccan, and Yemeni). They asked six different AI models (the "translators") to translate these sentences.

Then, they hired human experts to act as the "cultural detectives" and grade the translations using the LQM ladder.

What did they find?

The "Direction" Matters:
- When translating from Dialect to English, the AI mostly failed at Level 3 (Meaning). It didn't understand the local slang or idioms.
- When translating from English to Dialect, the AI mostly failed at Level 1 (Social Suit). It kept defaulting to the "official" language instead of the requested local dialect. It was like a tourist trying to speak local slang but sounding like a textbook.
Old Tools Missed the Point: The old "spellcheck" style metrics (like BLEU) were only weakly related to how good the translation actually felt to a human. They couldn't see the cultural mismatches.

4. Why This Matters

This paper is a wake-up call for AI developers. It says: "Don't just make your AI smart at grammar; make it smart at culture."

If you want an AI to translate for real people, it needs to know the difference between a formal business meeting and a casual chat with friends. LQM provides the map to find those mistakes so we can fix them, ensuring that future translators don't just sound correct, but sound right.

1. Problem Statement

Existing Machine Translation (MT) evaluation frameworks, including automatic metrics (e.g., BLEU, COMET) and human evaluation standards like Multidimensional Quality Metrics (MQM), are largely language-agnostic. While effective for standard languages, they fail to capture dialect- and culture-specific errors in diglossic languages like Arabic.

In Arabic, translation failures often stem not from surface-level semantic mismatches, but from:

Variety mismatches: Confusing Modern Standard Arabic (MSA) with specific dialects (e.g., Egyptian, Moroccan).
Pragmatic failures: Incorrect handling of speech acts, honorifics, and cultural context.
Register issues: Failure to maintain appropriate tone or formality.

Current MQM taxonomies often collapse these distinct phenomena into broad "Mistranslation" labels, making it difficult to diagnose why a model failed or to provide targeted corrective signals for improvement.

2. Methodology

A. The LQM Framework

The authors propose LQM (Linguistically Motivated Multidimensional Quality Metrics), a hierarchical error taxonomy designed to diagnose MT errors through six linguistically grounded levels. Unlike MQM, which lists error types, LQM organizes them by linguistic depth:

Sociolinguistics: Focuses on communicative competence and variety choice.
- Subcategories: Standardization interference (using MSA instead of dialect), Wrong dialect (horizontal mismatch), and Register mismatch (tone/formality).
Pragmatics: Captures failures in communicative intent and implied meaning.
- Subcategories: Speech acts/illocutionary force, Code switching, Multi-word expressions (MWEs)/proverbs, Discourse markers, and Vocatives/honorifics.
Semantics: Evaluates meaning transfer and propositional content.
- Subcategories: Lexical semantics (named entities, wrong terms, polysemy), Propositional semantics (addition, omission, hallucination), and Discourse semantics (pronouns, cohesion).
Morphosyntax: Captures violations of structural constraints.
- Subcategories: Grammar (verbal features like tense/aspect, nominal features like gender/number) and Constituent order (locale-specific reordering).
Orthography/Writing Conventions: Evaluates mechanical correctness.
- Subcategories: Spelling (typos, inconsistencies), Surface mechanics (numbers, currency, time formats), and Punctuation.
Graphetics: The lowest level, capturing technical realization failures.
- Subcategory: Character encoding errors.

The framework supports two annotation modes: a Lightweight version for broad categorization and a Diagnostic version for fine-grained subcategory labeling.

B. Dataset Construction

The authors constructed a new bidirectional parallel corpus named Casablanca:

Size: 3,850 sentences (550 per variety).
Languages: English ↔ 7 Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, Yemeni).
Source: Derived from conversational, culturally rich TV dialogue transcripts, professionally translated to ensure high-quality references.

C. Experimental Setup

Models Evaluated: Six Arabic-aware Large Language Models (LLMs) in a zero-shot setting:
- Closed-source: Gemini-2.5-Pro, Gemini-2.5-Flash.
- Open-weight: Fanar-9B, Gemma-27B, Command-A (111B), Command-R7B.
Annotation: Expert human annotators (linguists and native speakers) labeled 6,113 error spans across 3,495 erroneous sentences using LQM.
Scoring: A severity-weighted score equation (0–100) was derived, assigning weights of 1 (minor), 5 (major), and 25 (critical) to errors.

3. Key Contributions

LQM Taxonomy: A novel, language-agnostic framework that explicitly separates sociolinguistic and pragmatic failures from semantic and form-level errors, addressing the "catch-all" limitation of MQM in diglossic settings.
Casablanca Dataset: A high-quality, culturally rich, bidirectional parallel corpus covering seven Arabic dialects, specifically designed to stress-test dialect sensitivity.
Comprehensive Evaluation: The first large-scale evaluation of six diverse LLMs on Arabic dialects using expert span-level human annotations and severity-weighted scoring.
Public Resources: The annotated dataset, prompts, and guidelines are publicly available to facilitate future research.

4. Results

A. Error Distribution & Model Performance

Directional Shift:
- Dialect → English (DA→EN): Errors are dominated by Semantics (67.7%–92.3%), particularly named entities and lexical coverage. Models struggle to map dialect-specific idioms to English.
- English → Dialect (EN→DA): Errors shift dramatically to Sociolinguistics (up to 81.5% in Mauritanian). Models frequently default to MSA (standardization interference) or mix dialects (wrong dialect) rather than maintaining the target dialect's identity.
Model Rankings:
- Gemini-2.5-Pro achieved the highest performance across 9 of 12 directions.
- Command-R7B and Fanar-9B showed the weakest performance, with Fanar exhibiting high variance (strong on UAE, weak on Moroccan).
- EN→Moroccan was the most challenging direction, with most models scoring below 40.

B. Correlation with Automatic Metrics

The correlation between the automatic metric spBLEU and human LQM scores was weak (Pearson $r = 0.289$ ).
This confirms that n-gram overlap metrics fail to capture the sociolinguistic and pragmatic nuances (e.g., dialect identity, register) that are critical for high-quality dialectal translation.

C. Inter-Annotator Agreement (IAA)

Agreement was strong for error detection (overlap-based F1 = 0.821) and coarse categorization (F1 = 0.662).
Agreement dropped for fine-grained subcategories and strict span boundaries, indicating that while the framework is robust for diagnosis, precise boundary labeling remains challenging.

5. Significance and Conclusion

Diagnostic Precision: LQM provides a structured way to identify where models fail (e.g., "Is it a vocabulary issue or a cultural register issue?"), enabling targeted improvements in dialectal MT.
Beyond Arabic: While validated on Arabic, the framework is language-agnostic and applicable to any language with significant dialectal variation or sociolinguistic complexity.
LLM Limitations: The study reveals that current LLMs, even large ones, suffer from an "identity crisis" in dialectal generation, often defaulting to standardized forms or failing to grasp deep pragmatic intent.
Future Directions: The paper argues that improving dialectal MT requires optimizing not just semantic adequacy but also sociolinguistic fidelity, moving beyond surface-level metrics to linguistically motivated evaluation.