⚡ electrical engineering

Make It Hard to Hear, Easy to Learn: Long-Form Bengali ASR and Speaker Diarization via Extreme Augmentation and Perfect Alignment

This paper addresses the scarcity of long-form Bengali speech resources by introducing the 882-hour Lipi-Ghor-882 dataset and demonstrating that targeted fine-tuning with synthetic degradation optimizes ASR performance, while heuristic post-processing of baseline models proves more effective than retraining for speaker diarization, collectively establishing a highly efficient dual pipeline with a ~0.019 Real-Time Factor.

Original authors: Sanjid Hasan, Risalat Labib, A H M Fuad, Bayazid Hasan

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sanjid Hasan, Risalat Labib, A H M Fuad, Bayazid Hasan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot two very difficult skills at once: listening to a long, messy conversation in Bengali and telling who is speaking when multiple people are talking over each other.

This paper is the story of a team called "Team Villagers" who entered a high-stakes competition to solve this problem. They discovered that the "standard textbook" ways of doing this didn't work. Instead, they had to get creative, almost like a chef who realizes that to make a perfect dish, they sometimes need to burn the ingredients slightly to bring out the right flavor.

Here is the story of their journey, broken down into simple concepts and analogies.

1. The Big Problem: The "Lost in Translation" Gap

In the world of AI, there are huge libraries of data for English. But for Bengali, the data is like a tiny, dusty attic compared to a massive library. The team needed to process long conversations (like a whole podcast or a movie) where many people talk.

They faced two main hurdles:

ASR (Automatic Speech Recognition): Turning speech into text.
Speaker Diarization: Figuring out "Who said what?" (e.g., Speaker A vs. Speaker B).

2. The First Lesson: Speed vs. Accuracy (The Race Car Dilemma)

At the start, they tested several pre-made AI models. It was like testing different cars for a race:

The Sports Car (Moonshine): It was incredibly fast but crashed immediately because it couldn't understand the language well.
The Tank (Whisper-Medium): It was slow and heavy, but it understood the language perfectly.
The Compromise (Titu-FastConformer): A good balance, but not the best.

The Fix: They realized they couldn't just pick a car; they had to tune the engine. They took the "Tank" (Whisper) and put it on a super-fast track (using special hardware and parallel processing). They managed to make it 100 times faster without losing its accuracy. It went from taking 4 hours to run to just 26 minutes.

3. The "Make It Hard to Hear" Trick (The Gym Analogy)

This is the most interesting part of their story. Usually, when you train an AI, you give it the clearest, perfect audio possible. You want it to learn from a clean voice.

But the team did the opposite. They took a small, perfect dataset and intentionally ruined it. They added static noise, echo, and background chatter to 20% of the audio.

The Analogy: Imagine training a weightlifter. If you only let them lift 10 lbs in a quiet room, they get strong for that specific room. But if you force them to lift 10 lbs while someone is screaming and shaking the floor, they develop "deep muscle memory."
The Result: When they tested this "noisy" AI on the real, messy competition data, it crushed the competition. By making the training hard to hear, they made the AI easy to learn from difficult real-world situations.

4. The Diarization Disaster (The Broken Compass)

While the speech-to-text part was a success, the "Who is speaking?" part was a nightmare.
They tried using the most famous, state-of-the-art AI models available (the "GPS" of the industry).

The Problem: These fancy models failed miserably on Bengali conversations. They got confused by accents, background noise, and people talking over each other.
The Failed Fix: They tried retraining these models, adding more data, and cleaning the audio. Nothing worked. It was like trying to fix a broken compass by polishing the glass; the needle still wouldn't point North.

The "Post-Processing" Solution:
Since the AI couldn't figure it out on its own, the team built a strict rulebook (a heuristic algorithm) to fix the AI's mistakes after it made them.

The Analogy: Imagine a student who is bad at math but good at following rules. The AI does the math (guesses who spoke), and then a strict teacher (the algorithm) comes in and says, "Wait, two people can't speak at the exact same time. Move this sentence here. Delete that tiny 0.1-second blip."
The Result: By ruthlessly editing the AI's output with these rules, they fixed the errors that the AI couldn't solve on its own.

5. The Gift to the World: Lipi-Ghor-882

The team realized that the biggest reason everyone was struggling was that there wasn't enough good data. So, they built a massive new library called Lipi-Ghor-882.

It contains 882 hours of Bengali audio.
It's like building a massive, well-organized library from scratch so that future students (researchers) don't have to start from zero.

The Final Takeaway

The paper concludes with a powerful lesson for anyone building AI for languages like Bengali:

For Transcription: Don't just feed the AI perfect data. Make it train on messy, noisy data so it becomes tough and adaptable.
For Speaker Identification: Sometimes the "smartest" AI model isn't the best. A simple, strict set of rules applied after the AI speaks can be more powerful than trying to make the AI smarter.
Data is King: If you don't have enough data, no amount of fancy math will save you. You have to build the data first.

In short: They turned a difficult, messy problem into a solution by making the training harder, the rules stricter, and the data bigger.

1. Problem Statement

The paper addresses two critical gaps in low-resource Bengali speech processing:

Long-Form Audio Processing: Existing Automatic Speech Recognition (ASR) systems struggle with long-duration, multi-speaker audio, often failing to balance inference speed (Real-Time Factor) with accuracy.
Speaker Diarization: Robust speaker separation in Bengali is hindered by a severe lack of large-scale, temporally aligned conversational datasets.
Resource Scarcity: There is a notable absence of joint ASR and diarization resources for the Bengali language, making it difficult to train models that handle complex, real-world audio scenarios (noise, reverberation, overlapping speech).

The authors participated in the DL Sprint 4.0 competition, tasked with processing a massive 22-hour hidden test set of long-form Bengali audio under strict time constraints.

2. Methodology

The team, "Team Villagers," adopted a dual-pipeline approach, optimizing ASR and Speaker Diarization separately due to their distinct challenges.

A. Data Contribution: Lipi-Ghor-882

To bridge the data gap, the authors curated Lipi-Ghor-882, a comprehensive dataset:

Scale: ~882 hours of audio (856 hours annotated).
Source: 1,019 YouTube videos from 596 unique channels across 150+ domains.
Processing: Transcriptions were extracted via yt-dlp, and speaker boundaries were established using the Pyannote API, merging temporal annotations with speech segments.

B. Automatic Speech Recognition (ASR) Pipeline

The team evaluated several pre-trained models (Moonshine, Hishab-Titu-Bn-Conformer, Wav2Vec2, Whisper-Medium) and found a trade-off between speed and accuracy.

Optimization: They focused on Whisper-Medium (fine-tuned by Tugstugi) for its accuracy potential. By converting the model to CTranslate2 (CT2) format and running it on dual T4 GPUs with faster-whisper and Silero VAD for chunking, they reduced inference time from 4 hours to 26 minutes (RTF $\approx$ 0.019).
The "Hard to Hear" Strategy (Key Innovation):
- Standard fine-tuning and Parameter-Efficient Fine-Tuning (PEFT) yielded negligible gains.
- Counter-intuitive Approach: The team trained the model on a small, perfectly aligned subset where 20% of the audio was artificially corrupted with random noise and reverberation.
- Result: This "extreme augmentation" forced the model to rely on deep phonetic features rather than acoustic memorization, significantly boosting accuracy.
Failed Hypotheses: Contextual biasing caused catastrophic degradation on out-of-distribution data; ensembling (ROVER) performed worse than single models; and using Demucs for noise removal increased inference time too much to be viable.

C. Speaker Diarization Pipeline

The team tested global State-of-the-Art (SOTA) models including ECAPA-TDNN, Pyannote 3.1, and Diarizen.

Model Failure: Surprisingly, Diarizen (a leading open-source SOTA) performed poorly. Retraining models on muffled audio or integrating Bengali-specific WavLMs yielded no improvement in Diarization Error Rate (DER).
The "Easy to Learn" Strategy (Key Innovation):
- The team realized that model retraining was a dead end for this specific dataset.
- Success was achieved through heuristic post-processing of the baseline model outputs (specifically using Pyannote Community-1).
- Algorithm 1 (Strict Gap Post-Processing): A custom algorithm was applied to:
  1. Enforce strict inter-speaker gaps ( $\theta_{gap} = 0.17s$ ).
  2. Merge micro-segments of the same speaker if the gap was small ( $\theta_{merge} = 3.79s$ ).
  3. Filter out short segments ( $\theta_{seg} = 0.75s$ ) and speakers with low total duration ( $\theta_{spk} = 9.0s$ ).
  4. Resolve overlaps by shifting start times to ensure no temporal overlap between different speakers.

3. Key Results

The optimized pipelines achieved the following benchmarks on the 22-hour test set:

ASR Performance:
- The final Fine-Tuned Whisper-Medium (Faster Whisper + Noise Augmentation) achieved a Word Error Rate (WER) of 30.84% (Public) and 31.07% (Private).
- This was a significant improvement over the baseline Whisper-Medium (44.44% WER) and other models like Wav2Vec2 (51.38% WER).
- Inference Speed: Achieved an RTF of ~0.019 (26 minutes for 22 hours of audio).
Speaker Diarization Performance:
- The Pyannote Community-1 + Post-Processing pipeline achieved a Diarization Error Rate (DER) of 24.52% (Public) and 26.64% (Private).
- This outperformed the SOTA Diarizen model (27.89% Private DER) and Pyannote 3.1 (28.14% Private DER).
- Inference Speed: Achieved an RTF of ~0.019.

4. Significance and Contributions

Paradigm Shift in Low-Resource ASR: The paper demonstrates that for Bengali, targeted fine-tuning with synthetic acoustic degradation (making the data "hard to hear") is superior to raw data scaling or complex ensembling. It forces models to learn robust phonetic features.
Diarization via Post-Processing: It challenges the assumption that better SOTA architectures are always the solution. For complex, long-form Bengali audio, rigid algorithmic post-processing of baseline outputs is more effective than extensive model retraining.
Dataset Release: The release of Lipi-Ghor-882 provides the first large-scale, multi-domain, multi-speaker Bengali dataset, serving as a foundational resource for future research.
Practical Benchmark: The work establishes a highly optimized, low-latency pipeline (RTF ~0.019) that makes long-form Bengali speech processing feasible for real-time applications, setting a new empirical benchmark for the field.

5. Limitations

The authors noted constraints including limited GPU resources (5 free hours/month on Lightening AI), which restricted hyperparameter tuning, and issues with data quality (missing transcriptions, varying audio quality) that required significant preprocessing. Additionally, pre-trained models like Titu fastConformer were not fully optimized for Bengali features, requiring careful fine-tuning to avoid overfitting.