💬 NLP

Jointly Improving Dialect Identification and ASR in Indian Languages using Multimodal Feature Fusion

This paper proposes a multimodal framework that jointly optimizes Automatic Speech Recognition and Dialect Identification for low-resource Indian languages by fusing Conformer-based speech features with RoBERTa-processed ASR embeddings through a bottleneck encoder and gating mechanism, achieving significant performance improvements across eight languages and thirty-three dialects.

Original authors: Saurabh Kumar, Amartyaveer, Prasanta Kumar Ghosh

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Saurabh Kumar, Amartyaveer, Prasanta Kumar Ghosh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a conversation in a bustling Indian marketplace. The speakers are all speaking the same language (like Hindi or Bengali), but they are using different local "accents" or dialects. Some words sound slightly different, and the rhythm of speech varies from village to village.

For a computer to understand this, it needs to do two things at once:

Transcribe the words (Automatic Speech Recognition, or ASR).
Identify the specific dialect being spoken (Dialect Identification, or DID).

The Problem:
In the past, researchers tried to teach computers to do these two jobs separately, or they tried to combine them in a way that created a "tug-of-war." If the computer focused too hard on guessing the dialect, it might get the words wrong. If it focused too hard on the words, it might miss the dialect clues. It was like trying to juggle two balls while riding a unicycle; often, you'd drop one.

The Solution: The "Smart Translator" Team
The authors of this paper built a new system that acts like a highly coordinated team of experts working together, rather than two people fighting over the microphone. Here is how their system works, using simple analogies:

1. The Two Specialists (The Encoders)

Instead of just listening to the audio, the system uses two different "ears" to gather information:

The Audio Specialist (Bottleneck Encoder): This part listens to the raw sound waves. It's like a musician who can hear the subtle differences in how a drum is hit or a note is sung. It focuses on the acoustic quirks of the dialect.
The Text Specialist (RoBERTa Encoder): This part looks at the words the computer thinks it heard (based on the audio). It's like a linguist who knows that if someone says "water" in a specific way, they are likely from a specific region. It focuses on the linguistic clues.

2. The Manager (The Gating Mechanism)

Now you have two different opinions: one from the Audio Specialist and one from the Text Specialist. How do you combine them?
The system uses a "Gating Mechanism," which acts like a smart manager. Instead of just averaging their opinions, the manager decides: "For this specific sentence, the sound is very clear, so I'll trust the Audio Specialist more. For that next sentence, the words are tricky, so I'll lean on the Text Specialist." It dynamically blends the two sources of information to get the best possible picture.

3. The Refiner (Attention Encoder)

Once the manager has blended the information, the system passes it through an "Attention Encoder." Think of this as a spotlight. It scans the combined information and says, "Wait, this specific part of the sentence is the most important clue for identifying the dialect," or "This part is crucial for getting the word right." It sharpens the focus before making a final decision.

4. The Safety Net (No "Pre-pending")

Previous methods tried to help the computer by forcing it to read a "dialect tag" (like a label saying "This is a Bhojpuri speaker") at the very start of every sentence.

The Flaw: If the computer guessed the wrong dialect at the start, it would carry that wrong label through the whole sentence, confusing itself and making transcription errors.
The Fix: The new system does not force these tags at the start. It learns the dialect naturally from the sound and text while it works. This means even if the system isn't 100% sure about the dialect, it doesn't get confused and ruin the transcription of the words.

The Results: A Winning Team

The researchers tested this system on 8 different Indian languages with 33 different dialects. Here is what they found:

Better Dialect Guessing: The new system correctly identified the dialect about 81.6% of the time, which is better than previous methods.
Better Word Transcription: Because the system didn't get confused by wrong dialect tags, it also got the words right more often. It reduced the error rate in transcribing words significantly.
The "Safety Net" Effect: In older systems, if the computer guessed the dialect wrong, the transcription got much worse. In this new system, even when the dialect guess was wrong, the transcription stayed strong. It proved that you don't have to sacrifice one skill to improve the other; you can actually improve both at the same time.

In Summary:
This paper presents a smarter way for computers to handle the complex mix of Indian languages. By using a "team approach" that listens to both sound and text clues, and by avoiding the trap of forcing labels onto sentences, the system becomes more accurate at both understanding what is being said and who (or which region) is saying it.

Technical Summary: Jointly Improving Dialect Identification and ASR in Indian Languages using Multimodal Feature Fusion

1. Problem Statement

Automatic Speech Recognition (ASR) and Dialect Identification (DID) are critical for advancing speech technology in linguistically diverse regions like India, where many languages are low-resource and exhibit significant dialectal variations. While accurate DID can enhance ASR performance by mitigating errors caused by pronunciation, vocabulary, and grammatical differences, these two tasks have traditionally been studied separately or in ways that create performance trade-offs.

Existing joint ASR-DID approaches often suffer from suboptimal results in one or both tasks. For instance, methods that treat DID as an auxiliary task or prepend dialect IDs to training text may improve DID accuracy but degrade ASR performance, particularly when the dialect prediction is incorrect. Furthermore, many prior works rely solely on audio or text features, failing to leverage the complementary nature of multimodal data. There is a specific need for a framework that jointly optimizes both tasks without compromising the accuracy of the primary ASR task, especially in the context of Indian languages.

2. Methodology

The authors propose a novel multimodal framework that jointly improves ASR and DID through feature fusion, avoiding the practice of prepending dialect IDs to input text. The architecture consists of two primary blocks: an ASR Block and a DID Block.

2.1. Core Architecture

Input Processing: Input speech is processed by a Self-Supervised Learning (SSL) encoder (IndicWav2Vec) and refined by a Conformer Encoder to capture local speech patterns and long-range dependencies.
ASR Block: The Conformer output is fed into a decoder comprising both an Attention Decoder and a CTC Decoder for generating the final transcription.
DID Block: This block extracts dialectal features from the Conformer output using two parallel pathways:
1. Bottleneck Encoder: Processes speech-based features. It utilizes a convolutional block followed by a Bottleneck Transformer. Unlike previous designs, this implementation replaces 2D convolutions with 1D convolutions and removes average pooling to better preserve temporal dynamics.
2. RoBERTa Encoder: Processes text-based features derived from the ASR task. The Conformer output is projected to the vocabulary size and passed through a softmax (CTC) to generate embeddings, which are then processed by a RoBERTa encoder to extract contextual dialectal cues.

2.2. Multimodal Fusion Mechanism

The core innovation lies in how these representations are combined:

Gating Mechanism: An adaptive gating mechanism dynamically balances the contributions of the speech-based features (from the Bottleneck Encoder) and text-based features (from the RoBERTa Encoder). It computes learnable weights to fuse the representations: $H_{fused} = G \odot H_{rob} + (1 - G) \odot H_{bottleneck}$ .
Attention Refinement: The fused representation is further refined by an Attention Encoder (using Multi-Head Self-Attention and Layer Normalization) to enhance contextual dependencies.
Integration with ASR: The resulting dialect embeddings are concatenated with the original Conformer encoder output in the ASR Block. Crucially, these embeddings are detached from the computational graph before concatenation. This prevents gradient interference from the DID task to the ASR task, ensuring stable optimization and preventing the ASR model from learning false context when dialect predictions are incorrect.

2.3. Objective Function

The model is trained using a weighted loss function that combines:

CTC Loss ( $L_{CTC}$ )
Attention Loss ( $L_{ATT}$ )
Cross-Entropy Loss for DID ( $L_{CE}$ )

The total loss is formulated as: $L = \lambda_{CTC}L_{CTC} + (1 - \lambda_{CTC})L_{ATT} + \gamma_{CE}L_{CE}$ .

3. Key Contributions

Multimodal Feature Fusion: The paper introduces a framework that effectively fuses acoustic features (via a Bottleneck Encoder) and linguistic features (via RoBERTa on CTC embeddings) for dialect identification.
Gradient Isolation: By detaching dialect embeddings before concatenating them with ASR features, the method prevents the ASR performance from degrading due to incorrect dialect predictions, a common issue in previous joint models.
No Dialect ID Prepending: Unlike many existing ASR-DID systems, this approach does not require prepending dialect IDs to the training text, allowing the model to learn robust representations without relying on potentially erroneous explicit tokens.
Specialized Encoders: The design includes a modified Bottleneck Encoder with 1D convolutions to preserve temporal information and a RoBERTa encoder to leverage text-based cues from the ASR process.

4. Experimental Results

The proposed method, ASR-BN-ROB, was evaluated on the RESPIN dataset covering eight Indian languages (Bhojpuri, Bengali, Chhattisgarhi, Kannada, Magahi, Maithili, Marathi, and Telugu) with 33 dialects.

4.1. Dialect Identification (DID) Performance

The proposed ASR-BN-ROB model achieved an average DID accuracy of 81.63%, outperforming the baseline (ASR-DID-ROB) which scored 80.74%.
Significant improvements were observed across all languages, with notable gains in Chhattisgarhi (+2.38%) and Bhojpuri (+1.43%).
The model demonstrated improved generalization, evidenced by a 16.08% reduction in the standard deviation of dialect-wise accuracies compared to the baseline, indicating more consistent performance across different dialects.

4.2. ASR Performance

The method achieved an average Character Error Rate (CER) of 4.65% and a Word Error Rate (WER) of 17.73%.
These results surpass existing state-of-the-art ASR-DID methods and the Base-ASR model.
Impact of Incorrect Predictions: A critical finding is the model's robustness when dialect predictions are incorrect. In the baseline (ASR-DID-ROB), incorrect DID predictions led to significantly higher error rates (CER 6.01%, WER 21.85%). In contrast, the proposed model reduced the CER by 5.52% and WER by 5.17% in cases of incorrect DID predictions compared to the baseline, demonstrating that the ASR performance is not heavily penalized by DID errors.

4.3. Statistical Significance

Paired T-tests confirmed statistically significant differences (at a 95% confidence interval) in DID accuracy, CER, and WER between the proposed method and the baseline across the eight languages.

5. Significance and Claims

The authors claim that this work addresses the critical trade-off between ASR and DID performance often found in joint modeling. By leveraging multimodal attention fusion and a specific architectural design that isolates gradients, the framework achieves robust joint modeling.

The paper asserts that this approach contributes to the broader goal of developing more accurate and adaptable speech technologies for Indian languages, particularly in linguistically diverse settings. The results highlight that effective multimodal feature fusion can simultaneously enhance dialect classification and speech transcription without the performance degradation typically associated with incorrect auxiliary task predictions. The authors note that future work will explore the impact of this fusion on multilingual, multi-dialect ASR scenarios to further enhance adaptability.