💬 NLP

Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages

This paper proposes an orthographically-informed evaluation framework (OIWER) leveraging large language models to address the challenges of spelling variations in Indian languages, demonstrating that it provides a more accurate and human-aligned assessment of speech recognition performance compared to traditional metrics like Word Error Rate.

Original authors: Kaushal Santosh Bhogale, Tahir Javed, Greeshma Susan John, Dhruv Rathi, Akshayasree Padmanaban, Niharika Parasa, Mitesh M. Khapra

Published 2026-03-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kaushal Santosh Bhogale, Tahir Javed, Greeshma Susan John, Dhruv Rathi, Akshayasree Padmanaban, Niharika Parasa, Mitesh M. Khapra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a student's spelling test. The student writes a sentence perfectly, but you mark it wrong because they used a slightly different way to spell a word that is actually acceptable in their dialect. You might say, "You missed the 'u' in 'colour'!" when the student wrote "color," or perhaps they split a compound word differently.

In the world of Automatic Speech Recognition (ASR) for Indian languages, this is exactly what's happening. Computers are trying to "listen" to human speech and turn it into text. But because Indian languages are incredibly rich, flexible, and diverse, the computers are getting unfairly harsh grades.

Here is a simple breakdown of the paper's story, using some everyday analogies:

1. The Problem: The "Rigid Grader"

For years, researchers have used a metric called WER (Word Error Rate) to grade how good these speech-to-text systems are. Think of WER as a strict, by-the-book grader.

The Issue: Indian languages are like a giant, flexible puzzle. You can often spell a word in five different ways, split a long word into two smaller ones, or merge two words into one, and all of them are correct.
The Result: The computer says, "I heard 'passbook'." The human grader wrote "pass book." The strict grader marks it as an error.
The Consequence: The systems look terrible on paper (high error rates), even though a human listening to the audio would say, "Hey, that sounds perfect!" It's like getting an 'F' on a test because you used a synonym instead of the exact word the teacher had in mind.

2. The Solution: The "Flexible Dictionary"

The authors (a team from IIT Madras and Sarvam AI) realized they needed a new way to grade. They created a framework called OIWER (Orthographically-Informed Word Error Rate).

Think of this as giving the grader a super-powered, flexible dictionary before they start grading.

Instead of just checking if the student wrote "Passbook," the dictionary tells the grader: "Wait! 'Pass book', 'pass-book', and 'passbook' are all valid. If the student wrote any of these, give them full credit."
They also account for tricky things like:
- Loan words: Words borrowed from English that get spelled differently in Hindi or Tamil.
- Sandhi: The magical way Indian languages glue words together (like how "at" + "home" becomes "athome" in speech).
- Diacritics: The little dots and lines above or below letters that change the sound.

3. The Magic Tool: The "AI Assistant"

You might ask, "How do you make a dictionary that knows every possible way to spell a word in 22 different languages? That sounds like a million hours of work!"

The authors used Large Language Models (LLMs)—the same kind of AI that powers chatbots—to do the heavy lifting.

The Analogy: Imagine you need to list every possible nickname for a friend named "Robert." You could spend years asking people, or you could ask a super-smart AI: "Hey, list every way 'Robert' can be written or spoken." The AI instantly gives you "Rob, Bob, Bobby, Robbie," and even regional variations.
The team used this AI to generate lists of valid spellings for every word in their test data. Then, human experts just did a quick "spot check" to make sure the AI didn't get crazy.

4. The Results: A Fairer Game

When they re-graded the speech systems using this new "Flexible Dictionary" (OIWER), the results were eye-opening:

The "Pessimism" Vanished: The error rates dropped significantly (by an average of 6.3 points). The systems weren't actually worse; they were just being graded too harshly.
The Gap Closed: Previously, one model (Gemini) looked much worse than another (Canary). But when graded fairly, the gap between them shrank. It turns out they were actually much closer in performance than we thought.
Human Feel: The new scores matched what actual humans felt when listening to the recordings. The "computer score" finally matched the "human experience."

5. The Takeaway

This paper is essentially saying: "Stop grading Indian languages like they are rigid, standardized English."

By acknowledging that Indian languages are fluid and flexible, and by using AI to help us understand those rules, we can finally see the true potential of these speech systems. It's like finally realizing that the student didn't fail the spelling test; the test itself was just too rigid for the language being spoken.

In short: They built a smarter way to grade speech-to-text that respects the beautiful chaos of Indian languages, making the technology look as good as it actually feels to our ears.

1. Problem Statement

Automatic Speech Recognition (ASR) evaluation for Indian languages faces a critical challenge: inflated Word Error Rates (WER) that do not reflect actual system performance or human perception.

Root Causes: Indian languages exhibit high linguistic richness characterized by:
- Flexible spelling variations for common suffixes and morphemes.
- Permissibility in splitting or merging compound words (agglutination).
- Non-standard spellings in code-mixed contexts (e.g., English words written in Indic scripts).
- Lack of standardized orthographic guidelines compared to high-resource languages like English.
Consequence: Traditional WER metrics penalize valid orthographic variations as errors. This creates a "pessimistic" view of ASR performance, exaggerates performance gaps between models, and misaligns with how human users perceive transcription quality.
Limitations of Existing Solutions:
- Multi-reference WER: Requires multiple human transcriptions, which is resource-intensive and costly.
- Rule-based normalization (e.g., WER-SN): Relies on manual curation or heuristic rules, often failing to capture the full spectrum of variations, especially for loanwords and colloquialisms in low-resource languages.

2. Methodology

The authors propose a framework to create Orthographically-Informed (OI) Benchmarks and a new evaluation metric, OIWER.

A. Framework for Generating Variations

Instead of relying solely on manual curation, the authors use a Human-in-the-Loop (HITL) approach augmented by Large Language Models (LLMs):

Linguistic Categorization: They identified seven specific categories of orthographic variations common in Indian languages:
- Matra/Diacritic Variations: Alternative vowel signs and diacritical marks (e.g., nuktas in Hindi, pulli in Tamil).
- Spelling Variations for Loanwords: Transliterations and localized versions of English/Sanskrit/Persian words.
- Splitting/Merging: Decomposing or combining compound words.
- Phonetic Variations: Dialectal and accent-based differences.
- Ligature Variations: Consonant clusters joined or separate.
- Sandhi: Rules governing sound merging at word boundaries.
- Inverse Text Normalization (ITN): Variations in numbers, dates, and currencies.
LLM-Assisted Generation:
- An LLM (Gemini-2.5-Pro) is prompted with the full audio transcript and the predefined variation categories to generate context-aware candidate variations for every word.
- The output is structured as a sequence of sets, where each set contains valid variations for a specific word or subsequence (allowing for word merging).
Human Post-Editing:
- Native-speaking experts review the LLM-generated variations using a custom LabelStudio interface.
- They refine the lists by adding missing variants or removing incorrect ones.
- Efficiency: This process takes ~~1.2 minutes per utterance, significantly faster than full manual transcription (~~2.4 mins).

B. The OIWER Metric

The authors introduce Orthographically-Informed Word Error Rate (OIWER):

Algorithm: It modifies the standard Dynamic Programming (DP) approach used in WER (similar to the SCLITE toolkit).
Mechanism: Instead of comparing the predicted transcript ( $P$ ) against a single reference ( $T$ ), it compares $P$ against a set of reference variations ( $V$ ).
Calculation: The algorithm computes the minimum edit distance between the prediction and any valid variation in the reference set, finding the optimal alignment path.

3. Experimental Setup

Dataset: The framework was applied to the IndicVoices benchmark, covering 22 Indian languages across four language families.
Models Evaluated:
- Canary-C: Fine-tuned Canary-1B-flash model on MahaDhwani data.
- IndicConformer-IC: Open-source multilingual Conformer by AI4Bharat.
- Saarika:v2: SarvamAI's public API.
- Meta MMS-1B-all: Meta's multilingual model.
- Azure: Microsoft's batch transcription API.
- Gemini-2.5-Pro: Google's model used via prompting.
Scale: Generated variations for ~40.2K words across 22 languages, resulting in an average of 1.3 to 3.2 variations per word.

4. Key Results

Reduction in Pessimistic Error Rates:
- OIWER consistently yields lower error rates than standard WER.
- Average Improvement: A 6.3-point reduction in error rates across all languages for the state-of-the-art Canary model.
Narrowing Model Gaps:
- Standard WER suggested a 18.1-point performance gap between Gemini and Canary.
- OIWER reduced this gap to 11.5 points, indicating that WER was exaggerating the difference.
- Example: In Tamil, the gap dropped from 17.4 to 4.3 points.
Alignment with Human Perception:
- OIWER aligns 4.9 points closer to "Perceived WER" (human post-edited transcripts) compared to the previous best method, WER-SN.
- WER-SN failed to capture many loanword variations and colloquial forms, whereas OIWER captured the full orthographic diversity.
Error Type Analysis:
- The introduction of variations primarily reduced substitution errors by 28,500 instances across all languages, while insertion and deletion rates remained similar. This confirms that many "errors" were actually valid spelling variations.
LLM as a Proxy:
- A strong correlation ( $R^2 = 0.89$ ) was found between OIWER scores calculated using only LLM-generated variations and those using human-verified variations. This suggests LLMs can serve as a reliable, low-cost proxy for human annotation.

5. Significance and Contributions

New Evaluation Standard: The paper establishes OIWER as a superior metric for Indian languages, moving beyond rigid string matching to account for linguistic reality.
Scalable Benchmark Creation: It demonstrates a scalable, cost-effective pipeline using LLMs to generate orthographic variants, solving the resource bottleneck of multi-reference WER.
Realistic Progress Tracking: By correcting inflated error rates, OIWER provides a "truer sense of progress," preventing the false narrative that ASR models for Indian languages are significantly worse than they actually are.
Community Resource: The authors released the Orthographically-Informed IndicVoices Benchmark and the associated code, enabling future research to evaluate ASR systems more fairly and accurately.

In conclusion, this work argues that for morphologically rich and low-resource languages, evaluation metrics must evolve from strict orthographic matching to orthographically-informed approaches that respect the fluidity of human language usage.