Technical Summary: Bangla-WhisperDiar

Problem Statement

The paper addresses two critical challenges in Bangla (Bengali) spoken language understanding, specifically within the context of long-form audio:

1. Bangla Long-Form Automatic Speech Recognition (ASR): Transcribing extended, spontaneous, and multi-speaker Bangla recordings into text. This task is complicated by dialectal variations, conversational speech patterns, environmental noise, and the morphological complexity of the Bangla script.
2. Speaker Diarization: Determining "who spoke when" in long recordings, often containing overlapping speech and varying acoustic conditions.

Despite the global prevalence of Bangla (spoken by over 230 million people), the language remains under-resourced in speech technology. State-of-the-art systems are typically optimized for high-resource languages like English, resulting in suboptimal performance for Bangla, particularly in diverse, real-world acoustic environments.

Methodology

The authors propose a complete pipeline for both tasks, leveraging fine-tuning of pre-trained models with extensive data augmentation and domain-specific preprocessing.

1. Bangla Long-Form ASR Pipeline

The ASR system is built upon the Whisper-Medium architecture, specifically the tugstugi_bengaliai-regional-asr_whisper-medium variant, which was pre-trained on Bengali regional speech.

• Data Preprocessing & Chunking: Long-form audio (up to 1+ hours) is segmented into non-overlapping 25-second chunks, the practical limit for Whisper's encoder. To generate ground-truth labels for these chunks, the authors implemented a sequential fuzzy alignment algorithm. This uses bidirectional search windows and RapidFuzz ratio scoring to align the full ground-truth transcript with ASR hypotheses, preserving reading order while accommodating minor insertions or deletions.
• Text Normalization: A Bangla-specific normalization pipeline ensures consistency. This includes converting English numerals to Bangla words (handling calendar years 1000–2099), removing non-Bengali Unicode characters, and collapsing whitespace.
• Audio Augmentation: To improve robustness, a stochastic augmentation pipeline is applied to 30% of training clips. This includes:
  • Noise injection (pink/white noise).
  • Multi-tap echo and room reverberation (via synthetic Room Impulse Response convolution).
  • Clipping distortion, bandpass filtering (simulating telephone audio), pitch shifting, and time stretching.
• Training Strategy: The model undergoes full-weight fine-tuning (not parameter-efficient adaptation like LoRA) using the 8-bit AdamW optimizer to manage VRAM constraints. Training utilizes mixed precision (fp16), a cosine learning rate schedule, and gradient checkpointing.
• Inference & Post-Processing: Inference uses beam search with static KV caching. Post-processing includes Unicode normalization, removal of zero-width characters, and an iterative regex-based deduplication process to remove hallucinated repetitions (multi-word phrases, single words, and n-grams).

2. Speaker Diarization Pipeline

The diarization system utilizes the PyAnnote.Audio framework, specifically the pyannote/segmentation-3.0 model.

• Architecture: The pipeline consists of three stages: Segmentation (detecting speaker turns), Speaker Embedding (extracting speaker representations), and Clustering (grouping segments by identity).
• Fine-Tuning Strategy: The authors adopt a segmentation-swap approach. They fine-tune only the segmentation component (the PyanNet backbone) on the competition's annotated dataset using PyTorch Lightning. The pretrained speaker embedding and clustering components are retained without modification.
• Data Preparation: Annotations are converted into standard PyAnnote formats (RTTM, UEM, LST). The training set consists of 10 manually annotated recordings, with the last two reserved for validation.
• Post-Processing: A minimum-duration filter (0.3 seconds) is applied to remove spurious ultra-short segments likely caused by noise misclassification.

Key Contributions

1. Comprehensive Data Pipeline: Development of an automated preprocessing pipeline for Bangla long-form audio, featuring ASR-guided ground-truth alignment via fuzzy matching and Bangla-specific text normalization.
2. Robust Augmentation Strategy: Implementation of a diverse audio augmentation suite (noise, reverb, echo, distortion, pitch/time perturbation) tailored to improve model resilience against real-world acoustic degradations.
3. Fine-Tuning Protocols:
   • Full-weight fine-tuning of Whisper-Medium for ASR, demonstrating viability on consumer-grade GPUs using 8-bit optimizers.
   • Domain-adaptive fine-tuning of the PyAnnote segmentation model, integrated via a pipeline-swap strategy for speaker diarization.
4. Post-Processing Techniques: Introduction of hallucination removal for ASR and short-segment filtering for diarization to enhance output quality.

Results

The proposed systems were evaluated on the DL Sprint 4.0 benchmark (BengaliLoop dataset).

• ASR Performance: The fine-tuned system achieved a Word Error Rate (WER) of 24.41% on the test set. This represents a significant improvement over the pretrained Tugstugi baseline (34.07%) and a 28.4% relative reduction compared to the baseline. Post-processing contributed an additional reduction from 25.76% to 24.41%.
• Diarization Performance: The system achieved a Diarization Error Rate (DER) of 23.92%, a 40.3% relative reduction compared to the pretrained PyAnnote baseline (40.08%). This outperformed both classical VAD-based approaches and the pretrained neural baseline.
• Efficiency: The ASR system processed a 3-hour 38-minute recording with a Real-Time Factor (RTF) of 0.1659 on a single T4 GPU, improving to 0.0190 with dual-GPU optimization. The diarization module processed the same data with an RTF of 0.1054.
• Competition Scores: On the private test set, the system achieved a WER of 24.75% and a DER of 26.13%.

Significance and Claims

The paper claims that its primary significance lies in demonstrating that domain adaptation is critical for low-resource languages like Bangla, even when starting with strong multilingual pretrained models.

• Effectiveness of Fine-Tuning: The results validate that fine-tuning on domain-specific data (conversational Bangla) significantly bridges the gap between general pretrained models and specific application needs.
• Strategic Efficiency: The authors highlight that selective fine-tuning (only the segmentation component for diarization) is an effective strategy that balances performance gains with computational efficiency, as the speaker embedding component generalizes well across languages.
• Data Quality: The work emphasizes that Bangla-specific text normalization and rigorous data alignment are essential to reduce training noise and improve label consistency.
• Robustness: The study demonstrates that extensive data augmentation can effectively simulate real-world acoustic conditions, making the models robust to noise and distortion found in long-form recordings.

The authors conclude that while computational constraints and limited annotated data remain challenges, their approach provides a robust, scalable foundation for Bangla speech processing, with future work planned to explore joint ASR-diarization architectures and model compression for edge deployment.