💬 NLP

Dialect Matters: Cross-Lingual ASR Transfer for Low-Resource Indic Language Varieties

This paper empirically demonstrates that for low-resource Indic dialects, fine-tuning on small amounts of dialect-specific data often yields ASR performance comparable to using larger datasets from phylogenetically related high-resource languages, challenging the assumption that linguistic proximity alone dictates transfer success.

Original authors: Akriti Dhasmana, Aarohi Srivastava, David Chiang

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Akriti Dhasmana, Aarohi Srivastava, David Chiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand human speech. This robot is a master of language, but it has only ever listened to people speaking in perfect, polished, "standard" classrooms. It knows the rules of grammar and the exact pronunciation of words from the big, famous languages. But in the real world, people don't speak like that. They talk fast, they mumble, they mix languages together in the same sentence, and they use local slang that changes from one village to the next. This is the world of Automatic Speech Recognition (ASR): the technology that turns spoken words into text. The big question scientists are asking is: If you teach a robot a "standard" language, can it automatically understand a local dialect that sounds similar? Or does the robot need to actually listen to the messy, real-world dialects to learn them? This paper dives into that exact puzzle, exploring whether a robot's "family tree" of languages is enough to help it understand its cousins, or if it needs to hang out with the cousins directly to get the job done.

The researchers behind this study decided to test this idea using the incredibly diverse languages of India, specifically those written in the Devanagari script (like Hindi, Marathi, and many local dialects). They wanted to see if a speech model, which had been trained on a massive amount of clean, standard speech, could be "fine-tuned" (a bit like giving it a quick crash course) to understand low-resource dialects. The team had a hunch that the old way of thinking might be wrong. Usually, scientists assume that if two languages are closely related on a family tree—like cousins—they will be easy to transfer knowledge between. You'd think teaching a robot standard Hindi would make it great at understanding a Hindi dialect. But the authors suspected that the messy reality of how people actually speak might make that assumption fail.

To test this, they used a giant dataset called VAANI, which contains over 150,000 hours of spontaneous, noisy, and code-mixed speech from all over India. It's like a massive library of real conversations rather than a textbook. They took a top-tier speech model and tried to teach it various dialects in two different ways: first, by feeding it huge amounts of data from a "standard" language that was closely related to the target dialect, and second, by feeding it much smaller amounts of data from the actual dialect itself.

The results were a surprise to the usual rules of the game. The study suggests that simply being a "cousin" on the language family tree isn't enough. In fact, they found that fine-tuning a model on a tiny amount of dialect-specific data often worked just as well as, or even better than, fine-tuning it on a huge amount of data from a closely related standard language. It's as if teaching a robot a few hours of a local village's slang was more effective than teaching it a whole library of the "official" version of the language. The researchers discovered that the "standard" data often confused the model because it didn't capture the unique quirks, spellings, and sounds of the dialect.

They also dug deep into a specific, rarely studied language variety called Garhwali, spoken in the Himalayas. They tested several different AI models to see which one could handle this tricky dialect best. They found that even the best model struggled, making mistakes like turning local words into standard Hindi words or messing up the spacing between words. This happened because the models had been pre-trained on standard Hindi, so they had a built-in bias to "correct" the dialect back to the standard version, even when it was wrong.

The paper concludes that for speech recognition to work well on dialects, we can't just rely on the idea that "related languages are easy to learn." Instead, we need to treat dialects as their own unique things. The study suggests that giving models even a small taste of the actual dialect they need to understand is far more powerful than feeding them a diet of related, but different, standard languages. It's a reminder that in the world of language, the messy, real-world variations matter just as much as the polished, textbook rules.

Technical Summary: Dialect Matters: Cross-Lingual ASR Transfer for Low-Resource Indic Language Varieties

Problem Statement

Automatic Speech Recognition (ASR) systems for Indic languages have advanced significantly through large-scale multilingual pre-training on high-resource, standardized languages (e.g., Hindi, Marathi, Bengali). However, these systems often fail to address the linguistic reality of the region, where speech is characterized by extensive dialectal variation, spontaneous and noisy recording conditions, and frequent code-switching.

Current approaches typically assume that phylogenetic similarity (genetic relatedness) between a pre-training language and a target dialect is sufficient to facilitate transfer. This paper challenges that assumption, noting that dialects and non-standard varieties often possess distinct phonological, lexical, and orthographic properties not captured by standard language data. Consequently, state-of-the-art models frequently underperform on low-resource dialects, even those closely related to pre-training languages. Furthermore, there is a lack of systematic analysis regarding cross-dialectal transfer in zero-shot settings and the specific challenges posed by non-standardized speech like Garhwali.

Methodology

The authors conduct a comprehensive empirical study using the VAANI dataset, which contains over 150,000 hours of spontaneous speech from 156,534 speakers across 773 districts in India. The study focuses on 30 Devanagari-script Indic varieties.

Experimental Design

Baseline Assessment (RQ1): The authors evaluated IndicWav2Vec-Hindi (a Wav2Vec 2.0 model pre-trained on 40 Indic languages and fine-tuned on Hindi) on test samples from 30 Devanagari varieties to establish a baseline for zero-shot performance.
Orthographic Analysis (RQ2): To investigate the impact of spelling consistency, the authors measured the frequency of hapax legomena (words appearing only once) and type-to-token ratios in the transcripts. They correlated these metrics with Word Error Rate (WER) and Character Error Rate (CER).
Cross-Lingual Transfer (RQ3): The authors fine-tuned w2v-bert-2.0 on 1 to 7 hours of speech from various languages (both standard and dialectal) and evaluated them in a zero-shot setting across all 30 varieties. They analyzed the correlation between phylogenetic distance and performance, and specifically compared the effectiveness of fine-tuning on dialectal data versus closely related high-resource standardized data.
Case Study (Garhwali): A detailed evaluation was performed on Garhwali, a low-resource Pahari language variety. Multiple self-supervised models (Wav2Vec2, HuBERT, XLS-R, Whisper, w2vBERT) were fine-tuned and compared.
Error Analysis: The study included a diagnostic approach to quantify bias toward pre-training languages. Using a Hindi spell-checker, the authors categorized non-Hindi words in ground truth transcripts to determine if models preserved them, converted them to Hindi, or distorted them.

Key Contributions

Empirical Analysis of Cross-Lingual Transfer: The paper provides a large-scale analysis showing that fine-tuning on other dialects is often more effective for low-resource Indic dialects than relying on closely related, high-resource standardized languages.
First Detailed ASR Study on Garhwali: The authors present the first comprehensive ASR evaluation for Garhwali, including model selection, quantitative metrics, and qualitative error analysis.
Diagnostic Approach for Bias: The study introduces a method to quantify model bias toward pre-training languages in dialectal ASR, revealing systematic normalization of dialectal and code-mixed forms into standard Hindi.

Results

Baseline and Orthography

Baseline Performance: Even for languages seen during pre-training, IndicWav2Vec-Hindi exhibited high error rates (e.g., 50.4% WER on Hindi test speech), indicating that pre-training alone is insufficient for spontaneous, noisy speech.
Orthographic Variability: There is a strong positive correlation (Pearson's $\rho = 0.705$ ) between the percentage of unique words (hapax legomena) in a test set and the WER. Languages with inconsistent orthographic conventions (e.g., Thethi, Surjapuri) suffer significantly higher error rates.

Cross-Lingual Transfer

Phylogenetic Distance: While a general trend exists where closer phylogenetic distance correlates with lower error rates (Spearman's $\rho = 0.333$ ), this factor alone does not fully explain performance in dialectal settings.
Dialectal Fine-Tuning Superiority: Models fine-tuned on non-standard varieties frequently outperformed models fine-tuned on phylogenetically closer but standardized, high-resource languages.
- Example: Fine-tuning on Marwari (a Rajasthani dialect) yielded better zero-shot performance on Kumaoni (a Pahari variety) than fine-tuning on standard Hindi or Rajasthani, despite the greater phylogenetic distance.
- Controlled Experiments: Within phylogenetic subgroups (Rajasthani, Western Hindi, Bihari), fine-tuning on small amounts of dialectal data (e.g., 1.7 hours of Bundeli + Khariboli) yielded performance comparable to or better than fine-tuning on larger amounts of standardized data (e.g., 101.7 hours of Hindi + Bundeli + Khariboli).
Statistical Significance: A statistically significant trend was found where fine-tuning on dialectal speech is associated with lower WER on dialectal evaluation sets ( $r_{pb} = -0.196$ ), even when the dialectal training data volume was smaller than that of standardized languages.

Garhwali Case Study

Model Selection: Among the evaluated architectures, w2vBERT fine-tuned on Garhwali achieved the lowest error rates (WER: 0.493, CER: 0.193).
Persistent Challenges: Despite dialect-specific fine-tuning, the WER remained high (49.3%), indicating that Garhwali ASR remains a significant challenge.
Bias Analysis:
- The w2vBERT model preserved approximately 34.8% of non-Hindi words but converted 23.3% into valid Hindi words.
- In contrast, the IndicWav2Vec-Hindi baseline preserved fewer than 5% of non-Hindi words, converting the majority to Hindi or incorrect forms.
- Error Patterns: The most frequent errors involved word-boundary issues (space insertion/deletion), vowel length confusions (e.g., short vs. long vowels), and aspiration errors. The model tended to regularize Garhwali forms toward Hindi orthographic norms (e.g., deleting the halant ◌् at word endings).

Significance and Claims

The paper argues that the prevailing assumption—that phylogenetic similarity is the primary driver of successful cross-lingual transfer in Indic ASR—is insufficient for dialectal settings. The authors claim that the dialectal status of the fine-tuning data is a more informative predictor of performance than phylogenetic proximity alone.

Key implications drawn by the authors include:

Dialects as Distinct Entities: Dialects should be treated as distinct acoustic and linguistic entities rather than minor variants of standardized languages.
Data Selection Strategy: For low-resource dialects, incorporating even limited amounts of dialectal speech during fine-tuning is often more effective than relying on larger volumes of standardized language data.
Bias Mitigation: Current ASR systems exhibit substantial bias toward pre-training languages (specifically Hindi), leading to the systematic normalization of dialectal and code-mixed speech. Future systems require dialect-aware data selection and modeling strategies to mitigate this bias.

The study concludes that effective ASR for low-resource dialects requires moving beyond default assumptions of phylogenetic similarity toward evaluation and modeling practices that explicitly account for dialectal variation.