Enhancing ASR Performance in the Medical Domain for Dravidian Languages

1. Problem Statement

Automatic Speech Recognition (ASR) for low-resource Dravidian languages (specifically Telugu and Kannada) faces critical challenges when applied to specialized medical domains:

• Data Scarcity: There is a severe lack of annotated medical speech data for these languages.
• Morphological Complexity: Dravidian languages have rich morphological structures and complex phonetics, making them difficult for general-purpose ASR models to handle.
• Domain Specificity: Medical terminology and pronunciation variability differ significantly from general speech, causing standard models to underperform.
• Synthetic Data Limitations: While Text-to-Speech (TTS) can generate synthetic data to augment training, naive mixing of real and synthetic data often degrades performance due to quality inconsistencies and domain mismatches.

2. Methodology

The authors propose a Confidence-Aware Training Framework designed to effectively integrate heterogeneous data sources (real recordings and TTS-generated synthetic speech) through a multi-stage pipeline.

A. Data Sources

• Real Data: In-house medical recordings (Telugu) and public datasets (Kannada), totaling ~30 hours per language, normalized to 16 kHz.
• Synthetic Data: Generated using IndicTTS and GlowTTS models.
  • Aligned Synthetic: Matches real transcripts exactly.
  • Unaligned Synthetic: Generated from additional medical sentences not present in the real dataset.

B. Hybrid Confidence Scoring Mechanism

The core innovation is a three-component confidence scoring system used to weight training samples:

1. Static Confidence Metrics (Pre-computed):
   • Perceptual Score: Extracts acoustic features (spectral centroid, rolloff, MFCC, pitch, energy) to assess audio quality.
   • Acoustic Similarity: Cosine similarity between MFCCs of real and aligned synthetic speech.
   • WER Score: Word Error Rate of synthetic audio when transcribed by a baseline model (IndicWav2Vec).
2. Dynamic Model Confidence:
   • Calculated using Entropy of the model's output probability distribution during training. High entropy indicates high uncertainty.
3. Aggregation Strategies:
   • Fixed Weights: A linear combination of static metrics ($\alpha S_{perceptual} + \beta S_{sim} + \gamma S_{wer}$).
   • Learnable Weights: The weights ($\alpha, \beta, \gamma$) are treated as learnable parameters optimized via softmax, allowing the model to adaptively determine the importance of each metric based on the language and data source.
   • Hybrid Integration: Combines static scores with dynamic model entropy ($C_{final} = \lambda C_{static} + (1-\lambda)C_{model}$). A curriculum learning approach anneals $\lambda$ from 1.0 (relying on static quality) to 0.5 (balancing with model uncertainty) during training.

C. Training and Post-Processing

• Weighted Loss Function: The Cross-Entropy loss is multiplied by the final confidence score ($L_{weighted} = C_{final} \cdot L_{CE}$), ensuring high-confidence samples contribute more to gradient updates.
• Models Evaluated: Wav2Vec2-Large (CTC-based) and Whisper-Medium (Seq2Seq).
• Post-Decoding Correction:
  • Statistical LM: A 5-gram KenLM model trained on domain-specific medical text.
  • Neural LM: IndicBART and mT5 for contextual refinement.

3. Key Contributions

1. Curriculum Learning Strategy: A dynamic training schedule that evolves confidence weighting, prioritizing static quality filtering in early stages and model uncertainty in later stages.
2. Novel Hybrid Confidence Mechanism: A unique combination of static perceptual/acoustic metrics and dynamic model entropy to assess data quality in mixed real/synthetic datasets.
3. Adaptive Weighting: Introduction of learnable weights for confidence aggregation, which outperforms fixed-weight strategies by automatically learning language-specific data quality patterns.
4. Comprehensive Pipeline: A two-stage approach combining adaptive training with statistical and neural post-processing for transcript correction.

4. Experimental Results

The framework was evaluated on Telugu and Kannada medical datasets using Wav2Vec2 and Whisper models.

Performance Improvements (Word Error Rate - WER):

Key Findings:

• Learnable vs. Fixed: The learnable weight strategy significantly outperformed fixed-weight static confidence, demonstrating the need for adaptive aggregation.
• Model Architecture: Wav2Vec2 consistently outperformed Whisper in this low-resource, domain-specific setting, likely due to its CTC-based decoding and smaller parameter count (317M vs. 769M), which reduced overfitting.
• Post-Processing: The KenLM (5-gram for Telugu, 4-gram for Kannada) provided the most effective post-processing correction, outperforming neural LMs (IndicBART/mT5) in terms of both accuracy and computational efficiency.
• Synergy: Confidence-aware training and language modeling address orthogonal error types (training noise vs. decoding linguistic constraints), yielding synergistic improvements.

5. Significance and Conclusion

This work demonstrates that adaptive confidence-aware training is a viable solution for building robust ASR systems in low-resource, morphologically complex languages within specialized domains.

• Scalability: The framework is designed to be extensible to other Indian languages and specialized domains beyond healthcare.
• Data Efficiency: It maximizes the utility of synthetic data without compromising model robustness, solving the critical bottleneck of data scarcity.
• Practical Deployment: By combining learnable confidence weighting with efficient statistical language models (KenLM), the system achieves a high balance of accuracy and inference speed, making it suitable for real-world medical applications in Dravidian-speaking regions.