💬 NLP

NormEval: A Unified Multi-Metric Framework for Evaluating Semantic Fidelity in Text Normalization

This paper introduces NormEval, a unified multi-metric framework that combines five complementary metrics to holistically evaluate text normalization quality across efficiency, downstream utility, and morphological fidelity, thereby preventing misleading rankings caused by the limitations of isolated evaluation methods.

Original authors: Md Abdullah Al Kafi, Raka Moni, Walayat Hussain

Published 2026-06-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Abdullah Al Kafi, Raka Moni, Walayat Hussain

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive library. You have millions of books, but the words are all over the place: some are capitalized, some are in different tenses (run, ran, running), and some have long, fancy suffixes. To find things quickly, you decide to "normalize" the library. You strip away the extra bits so that "running," "ran," and "runs" all become just "run." This saves space and makes searching faster.

But here's the problem: What if you strip away too much?

If you are too aggressive, you might turn "running" into "run" (good), but you might also accidentally turn "runner" (a person) into "run" (an action), or worse, turn two completely different words into the same nonsense string. You've saved space, but you've lost the meaning.

This paper, NormEval, is like a new, super-strict quality inspector for libraries. It says, "Stop just checking if you saved space or if your search engine works faster. We need to check if you actually broke the meaning of the words while you were cleaning up."

The Problem: The "One-Number" Trap

The authors explain that for a long time, people evaluating these cleaning tools only looked at one or two things:

The Compression Score: "Did we shrink the dictionary?" (Great for saving space, but doesn't tell us if we deleted important words).
The Downstream Score: "Did the computer get better at guessing the topic?" (Great, but sometimes a computer gets better just by luck, or because it's ignoring the messy parts, not because the cleaning was perfect).

The paper argues that relying on just these numbers is like judging a chef only by how fast they chop vegetables. They might be fast, but they might have chopped the onions and the gold rings you were wearing.

The Solution: The "NormEval" Five-Point Checkup

The authors created a unified framework called NormEval that uses five different "sensors" to check the cleaning process. Think of it as a car safety inspection that checks the engine, the brakes, the tires, the airbags, and the fuel efficiency all at once.

Here are the five metrics, explained simply:

Compression Ratio (CR): The "Space Saver"
- What it does: Measures how much the vocabulary shrank.
- Analogy: Did the librarian successfully fit all the books into a smaller room?
- The Catch: A high score here is good only if the books still make sense.
Model Performance Delta (MPD): The "Test Drive"
- What it does: Checks if the cleaning helped or hurt the computer's ability to do a task (like sorting reviews as positive or negative).
- Analogy: After the library is reorganized, can you still find the book you need quickly?
- The Catch: The paper found that sometimes the computer does worse after cleaning, even if the cleaning looked "efficient." This metric acts as a "safety gate" to stop bad cleaning methods.
Information Retention Score (IRS): The "Meaning Check"
- What it does: Uses advanced AI to compare the meaning of the original text vs. the cleaned text.
- Analogy: If you read the original sentence and the cleaned sentence, do they tell the same story?
- The Catch: Sometimes the AI says "Yes, the meaning is the same," even if the words look weirdly chopped up.
Algorithm Effectiveness Score (AES): The "Balanced Scorecard"
- What it does: Combines the "Space Saver" (CR) and the "Meaning Check" (IRS) into one number.
- Analogy: It's a grade that says, "You saved space, AND you kept the meaning. Good job." If you saved space but lost meaning, your grade drops.
Average Normalized Levenshtein Distance (ANLD): The "Microscope" (The Safety Gate)
- What it does: This is the paper's big innovation. It looks at the letters themselves. It measures exactly how many characters were changed, added, or deleted.
- Analogy: Imagine a surgeon. The "Meaning Check" (IRS) might say, "The patient is alive." But the "Microscope" (ANLD) looks at the incision and says, "Wait, you cut off the wrong finger!"
- Why it matters: The paper found that sometimes the "Meaning Check" is fooled. It thinks the meaning is safe, but the "Microscope" sees that the words have been mutilated. This metric acts as a Safety Gate to stop tools that are too aggressive.

The Experiments: What Did They Find?

The authors tested this new framework on two languages: English and Bangla (a language spoken in Bangladesh and India).

The "Safety Gate" Discovery: They found that some popular cleaning tools (like one called BNLTK for Bangla) looked great on paper. They saved a lot of space and the AI thought the meaning was preserved. But, when they used the "Microscope" (ANLD), they realized these tools were chopping words into meaningless fragments.
The Verdict: If you only looked at the old methods, you would have picked the "bad" tool. But with NormEval, they could see that the "bad" tool was actually destroying the language, and they could pick the "good" tool (BanLemma) that kept the words safe.
Cross-Language Test: They also tested it on six different languages (like Arabic, German, Spanish). They found that languages with complex word structures (like Arabic) are very hard to clean without breaking them. The framework successfully showed which languages were suffering the most from aggressive cleaning.

The Bottom Line

The paper concludes that we can no longer trust just one number to judge how well we are cleaning up text. We need a multi-dimensional checklist.

Old Way: "Did we save space? Yes? Great!"
NormEval Way: "Did we save space? Yes. Did we keep the meaning? Yes. Did we accidentally chop the words into gibberish? No. Did the computer still work? Yes."

The authors have released this as an open-source tool (a Python package) so that anyone building language systems can use this "five-point checkup" to ensure they aren't accidentally breaking their own software. It's about making sure that in the rush to make things smaller and faster, we don't lose the soul of the language.

Technical Summary: NormEval – A Unified Multi-Metric Framework for Evaluating Semantic Fidelity in Text Normalization

1. Problem Statement

Text normalization techniques, such as stemming and lemmatization, are foundational to Natural Language Processing (NLP) pipelines, reducing vocabulary size and enabling models to generalize across morphological variants. However, the evaluation of these methods suffers from a critical methodological gap: existing approaches rely on isolated, single-metric paradigms that fail to distinguish between beneficial vocabulary reduction and harmful semantic distortion.

The literature currently relies on three insufficient paradigms:

Intrinsic Compression: Measuring vocabulary reduction (e.g., Compression Ratio). This fails to detect if the reduction collapses semantically unrelated words.
Extrinsic Downstream Accuracy: Measuring classifier performance on normalized text. This is a coarse signal that conflates normalization quality with model architecture and dataset characteristics, often masking errors.
Sequence Prediction Scores: Using metrics like BLEU against gold-standard references. This requires scarce annotated resources for low-resource languages and fails to capture downstream utility or semantic fidelity.

Consequently, there is no standardized framework to detect semantic degradation—the gradual loss or distortion of meaning during processing—which is particularly problematic in morphologically rich languages (e.g., Bangla, Arabic) where aggressive stemming can merge distinct concepts, leading to precision/recall failures in high-stakes domains like clinical decision support and legal analysis.

2. Methodology: The NormEval Framework

The paper proposes NormEval, a unified, data-driven evaluation framework that integrates five complementary metrics across three orthogonal dimensions: Macro-level Efficiency, Downstream Utility, and Micro-level Morphological Fidelity.

The Five Metrics

Compression Ratio (CR): Measures the percentage reduction in vocabulary size.
- $CR = \frac{\# \text{Unique Words Before}}{\# \text{Unique Words After}}$
- Indicates macro-level efficiency but does not assess semantic validity.
Model Performance Delta (MPD): Quantifies the change in downstream task performance (e.g., macro F1-score) between normalized and original text.
- $MPD = \mathcal{P}_{norm} - \mathcal{P}_{orig}$
- Acts as a "hygiene check" rather than a primary performance metric; it detects if normalization harms task utility.
Information Retention Score (IRS): Measures semantic preservation at the macro level using contextual transformer embeddings (e.g., BanglaBERT, DistilBERT).
- Calculated as the cosine similarity between aggregated embedding vectors of original and normalized documents.
- High IRS indicates the normalized text retains the original meaning.
Algorithm Effectiveness Score (AES): A unified harmonic mean of Vocabulary Reduction Gain (VRG) (a bounded transformation of CR) and IRS.
- $AES = \frac{2 \cdot IRS \cdot VRG}{IRS + VRG}$
- Ensures a high score requires both efficiency and semantic fidelity, penalizing algorithms that excel in only one dimension.
Average Normalized Levenshtein Distance (ANLD): A character-level diagnostic metric acting as a Safety Gate.
- Calculates the average normalized edit distance between original tokens and their normalized forms.
- Detects "destructive over-stemming" (aggressive mutations) that macro-level embeddings (IRS) and downstream tasks (MPD) may mask.

Experimental Design

The framework was validated across three experimental settings:

Multilingual Generalizability: Evaluated on the XNLI benchmark across six typologically diverse languages (Arabic, German, English, Spanish, French, Russian) using a zero-shot cross-lingual transfer protocol.
Ablation Study: Conducted on English and Bangla sentence complexity datasets (BTSD) to test the necessity of each metric by systematically removing them.
Robustness Evaluation: Tested on a Bangla sentiment classification corpus with class imbalance and domain shift to verify the framework's ability to detect morphological risks in realistic scenarios.

3. Key Contributions

Unified Framework: Introduces NormEval, the first framework to simultaneously evaluate intrinsic compression, downstream utility, semantic preservation, micro-level morphological distortion, and statistical robustness.
Safety Gate Formalism: Proposes the use of ANLD thresholds to detect harmful over-stemming that is invisible to embedding-based metrics (IRS) or accuracy scores.
Re-evaluation of MPD: Provides empirical evidence that downstream accuracy (F1-score) is an unreliable proxy for normalization quality on its own, as gains are often statistically insignificant ( $p > 0.05$ ). It redefines MPD as a "hygiene" check to prevent degradation.
Cross-Linguistic Consistency: Demonstrates that tools with identical AES scores can differ significantly in morphological risk, a distinction only visible through the multi-dimensional framework.
Open-Source Ecosystem: Releases a reproducible Python package (normeval) containing all datasets, benchmarks, and evaluation interfaces.

4. Key Results

Multilingual Findings (XNLI): Morphologically rich languages (Arabic, Spanish, Russian) exhibited high compression but also high ANLD scores (>0.32) and significant downstream performance degradation (negative MPD). Crucially, IRS remained moderate (0.70–0.81), confirming that embedding similarity alone fails to flag severe character-level distortion.
Ablation Findings: Removing any single metric from the framework led to incorrect algorithm rankings.
- Without CR, efficient algorithms could not be distinguished from conservative ones.
- Without MPD, harmful normalization that degrades downstream performance could not be detected.
- Without IRS/AES, the balance between efficiency and fidelity was lost.
- Without ANLD, the framework failed to detect "destructive over-stemming" (e.g., BNLTK in Bangla), incorrectly ranking it as superior to safer alternatives based on IRS and AES alone.
Robustness Findings: In the Bangla sentiment task, the algorithm with the highest AES (BNLTK) was flagged as unsafe due to high ANLD (0.26) and negative MPD. Conversely, a more conservative algorithm (BanLemma) with lower AES but safe ANLD (0.18) yielded a 7.31 percentage point improvement in SVM macro F1-score. This validates the Safety Gate Hypothesis: ANLD is necessary to identify morphological risks that compromise downstream utility.

5. Significance and Claims

The paper claims that NormEval addresses a systemic failure in current evaluation methodologies, which often allow "silent corruption" of meaning to go undetected. By operationalizing the Safety Gate Hypothesis, the framework provides a principled, linguistically informed method for selecting normalization algorithms.

The authors assert that NormEval is not merely a diagnostic tool but a decision-support system for high-stakes NLP applications (e.g., clinical, legal). It enables practitioners to:

Avoid selecting algorithms that appear efficient (high CR/AES) but are semantically destructive (high ANLD).
Calibrate ANLD thresholds based on specific task requirements (e.g., stricter thresholds for sentiment analysis vs. relaxed for information retrieval).
Ensure that normalization improves, rather than covertly degrades, intelligent systems.

The paper concludes that while the current study focuses on stemming and lemmatization, the framework lays the groundwork for more reliable, linguistically informed NLP pipelines across languages and tasks.