💬 NLP

SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation

The paper proposes Script-Normalized WER (SN-WER), a training-free evaluation metric that transliterates text into a canonical script before scoring, effectively reducing artificial error inflation caused by script mismatches in Indic multilingual ASR systems while maintaining sensitivity to genuine recognition errors.

Original authors: Priyaranjan Pattnayak

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Priyaranjan Pattnayak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are judging a spelling bee, but there's a twist: half the contestants are writing their answers in English, and the other half are writing the exact same words in a different script, like Hindi or Arabic.

If you strictly count every letter that doesn't match the judge's "English" answer key, you might think the second group is failing miserably. But in reality, they spelled the words perfectly; they just used a different alphabet. This is the problem the paper addresses.

Here is a simple breakdown of the paper "SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation":

The Problem: The "Alphabet Penalty"

When computers try to listen to human speech and turn it into text (called ASR), we measure how good they are using a score called WER (Word Error Rate). Think of WER as a "mistake counter."

The Issue: In many languages (especially in India, where the paper focuses), people often type or speak in their native script (like Devanagari), but the computer sometimes outputs the same words written in the Latin alphabet (like English letters, known as "romanized" text).
The Result: If the computer says "Namaste" (in English letters) and the correct answer is "नमस्ते" (in Hindi letters), a standard WER counter screams, "Total failure! These are completely different words!" It inflates the error rate, making the computer look worse than it actually is. It's like giving a student an 'F' just because they wrote their essay in French instead of English, even though the story was perfect.

The Solution: The "Universal Translator" (SN-WER)

The authors propose a new scoring method called SN-WER.

How it works: Before the computer gets its score, SN-WER acts like a universal translator. It takes both the "correct answer" and the "computer's guess" and converts them both into the same, standard script (the native script of that language).
The Magic: Once everything is in the same script, the computer can compare the actual words rather than the shape of the letters.
The Rule: This doesn't change how the computer works or how it was trained. It only changes how we grade the results. It's a "report card" upgrade, not a "student" upgrade.

What They Found (The Experiments)

The researchers tested this on 5 different Indian languages, 2 different datasets (one very clean, one very noisy), and 3 different AI models.

On Clean Data (FLEURS):
- Analogy: Imagine a quiet library where the computer made very few mistakes, but many were just "wrong script" errors.
- Result: SN-WER showed that the computer was actually much better than the old score suggested. It reduced the "error gap" between different models by up to 12%. It proved that some of the "failures" were just formatting issues, not real listening problems.
On Noisy Data (Common Voice):
- Analogy: Imagine a busy, noisy street. Here, the computer is actually making real mistakes (hearing "cat" instead of "bat").
- Result: SN-WER didn't magically fix these scores. The errors stayed high. This is good news! It proves that SN-WER isn't just hiding bad performance; it's smart enough to know the difference between a "script mistake" and a "real listening mistake."
The Stress Tests:
- The researchers tried to trick the system by forcing the computer to output random "wrong script" text. SN-WER successfully ignored the script confusion and only penalized the actual nonsense words.
- They also checked if SN-WER would accidentally hide real errors. It didn't. If the computer got a word wrong, SN-WER still gave it a bad score.

Why Does This Matter?

The paper argues that we should report SN-WER alongside the traditional WER score, like showing both a "Raw Score" and a "Normalized Score."

When to use it: If you are building a search engine, a voice assistant for a database, or a tool that feeds speech into a large AI model, you often don't care if the text is in Hindi or English letters; you just want the meaning to be right. SN-WER tells you how well the AI understands the meaning.
When NOT to use it: If you are making subtitles for a movie or a textbook, the script does matter. In those cases, you still need the traditional WER to ensure the computer is getting the letters right, not just the sounds.

The Bottom Line

The paper introduces a new way to grade speech-to-text AI that stops punishing it for using the "wrong alphabet." It helps researchers see the true listening ability of the AI, separating "bad handwriting" from "bad hearing." It's a simple, free tool that makes evaluation fairer for languages that use diverse scripts.

Technical Summary: Script-Normalized WER (SN-WER) for Multi-Script Indic ASR Evaluation

1. Problem Statement

Word Error Rate (WER) is the standard metric for evaluating Automatic Speech Recognition (ASR) systems. However, in multilingual settings—particularly for Indic languages—WER can significantly overestimate errors when the reference transcripts are in native scripts (e.g., Devanagari, Bengali, Tamil) while the model hypotheses are emitted in romanized text.

Current evaluation practices treat these script differences as lexical substitutions, artificially inflating error rates and complicating cross-language model comparisons. Prior work has shown that normalization choices and orthographic variations substantially alter reported error rates, yet existing metrics like WERd (for dialectal Arabic) or lenient CER (for Japanese) address spelling variants or character-level inconsistencies rather than cross-script canonicalization in monolingual, multi-script settings.

2. Methodology: Script-Normalized WER (SN-WER)

The authors propose SN-WER, a training-free, evaluation-only scoring method designed to isolate script-mismatch penalties from genuine lexical recognition errors.

Core Principle

SN-WER computes WER after transliterating both the reference ( $R$ ) and hypothesis ( $H$ ) tokens into a language-specific canonical script ( $C$ ) before alignment.
$SN\text{-}WER(R, H) = WER(T(R), T(H))$
Where $T(\cdot)$ is a deterministic, boundary-preserving transliteration map.

Implementation Details

Canonical Script: The benchmark reference script (native script) is used as the target canonical form.
Transliteration: Romanized tokens are detected via Unicode-block heuristics and transliterated into the native script using standard libraries (ICU, IAST-style, ITRANS-style).
Normalization: Standard Unicode, punctuation, and digit normalization are applied.
Constraints: The method assumes the transliteration map is deterministic and preserves token boundaries. It does not claim to be a replacement for standard WER/CER but serves as a diagnostic companion.

Diagnostic Properties

The authors define three properties for SN-WER:

Identity: If reference and hypothesis share the same script, $SN\text{-}WER \approx WER$ .
Conservativeness: Under ideal conditions, $SN\text{-}WER \le WER$ , as script-only mismatches are removed without increasing edit distance. The authors empirically bound deviations caused by transliteration collisions or token splitting.
Lexical Sensitivity: SN-WER must not reduce errors caused by incorrect words, deletions, insertions, or word-order changes.

3. Experimental Setup

The evaluation was conducted across:

Languages: 5 Indic languages (Hindi, Bengali, Tamil, Odia, Gujarati) and 2 additional languages (Arabic, Urdu) for cross-script validation.
Datasets: Curated (FLEURS) and Noisy (Common Voice v17).
Models: Whisper-large-v3, Whisper-small, and Meta's MMS.
Controls: Stress tests involving artificial romanization injection (0–50% token replacement) and lexical corruption (20–30% token substitution) were performed to validate robustness.

4. Key Results

4.1 Reduction of Inflated Gaps

Curated Data (FLEURS): SN-WER narrowed inflated model gaps by up to 12% (e.g., on Gujarati). For Odia on Common Voice, the reduction was even higher (26%), suggesting significant script-mismatch components in the baseline WER.
Noisy Data (Common Voice): Reductions were smaller or inconsistent (e.g., -4.3% for Whisper-large), indicating that noisy conditions contain genuine recognition errors that SN-WER correctly preserves.

4.2 Robustness and Stability

Transliterator Invariance: Comparing IAST, ITRANS, and ICU mappings yielded negligible differences ( $\Delta < 0.002$ ).
Collision Rates: Transliteration collision rates (distinct tokens mapping to identical forms) remained below 0.1%.
Normalization Stability: Ablations on digit and punctuation normalization shifted SN-WER by at most $\Delta < 0.05$ .

4.3 Stress Tests and Sensitivity

Artificial Script Mismatch: When 50% of hypothesis tokens were artificially romanized, WER inflated by $\Delta = 0.234$ , while SN-WER increased by only $\Delta = 0.158$ . This represents a 67% attenuation of script-driven inflation.
Lexical Corruption: When 20–30% of tokens were lexically corrupted (without script change), SN-WER tracked WER nearly identically ( $\Delta SN / \Delta WER \approx 1.09$ ), confirming it does not mask genuine recognition errors.
Adversarial Checks: Random token shuffling or substitution resulted in SN-WER scores approaching 1.0, confirming sensitivity to word-order and lexical errors.

4.4 Cross-Script Extension

In Arabic and Urdu, SN-WER showed moderate reductions (4.9–9.0%), validating the approach's applicability beyond Indic scripts when orthographic conventions are respected.

5. Significance and Claims

The paper positions SN-WER not as a replacement for WER/CER, but as a companion score essential for specific evaluation contexts:

Diagnostic Utility: It quantifies how much of the measured error is attributable to script mismatch versus lexical failure.
Downstream Relevance: SN-WER is particularly valuable when the output script is orthogonal to the downstream task, such as in keyword search, indexing, retrieval, intent classification, or multilingual LLM pipelines where the semantic content matters more than the orthography.
User-Facing Limitations: For applications where the script choice is part of the output quality (e.g., user-facing captions, subtitles, educational tools), standard WER/CER remains necessary, and SN-WER should be reported alongside it.

The authors conclude that SN-WER provides a rigorous, training-free method to disentangle script bias from model performance, offering a more accurate basis for comparing ASR models in multi-script Indic environments.