WhisperAlign: Word-Boundary-Aware ASR and WhisperX-Anchored Pyannote Diarization for Long-Form Bengali Speech

This paper presents WhisperAlign, a solution for the DL Sprint 4.0 that combines word-boundary-aware ASR using whisper-timestamped chunking and domain-fine-tuned Pyannote diarization anchored by WhisperX to achieve high-accuracy transcription and speaker separation for long-form Bengali speech.

The Gist

Imagine you are trying to transcribe a very long, chaotic conversation between several people speaking in Bengali. It's like trying to write down a 60-minute movie script where the audio is muddy, people talk over each other, and the camera keeps cutting in and out.

This paper describes how a team of researchers built a smart system to solve two big problems:

1. Transcription (ASR): Turning the speech into text accurately.
2. Diarization: Figuring out who said what and when.

Here is the breakdown of their solution using simple analogies.

---

Part 1: The Transcription Problem (The "Word-Boundary" Puzzle)

The Challenge:
Standard AI models (like the famous "Whisper") are great at short clips, but they get confused with hour-long recordings. If you just chop a long audio file into random 30-second chunks, you might cut a sentence in half right in the middle of a word. The AI gets confused, thinks it's hearing something else, and starts "hallucinating" (making up words that weren't said).

The Solution: The "Smart Scissors"
Instead of using a ruler to chop the audio into equal pieces, the team invented a pair of "smart scissors."

• How it works: They used the AI itself to listen to the audio and find the exact moment each word starts and ends.
• The Analogy: Imagine you are cutting a long loaf of bread. A normal person cuts it every 3 inches, often slicing right through a crusty piece of bread. The "Smart Scissors" wait until they see a gap between the loaves (the word boundaries) before cutting.
• The Result: They created perfect, bite-sized chunks of audio that always start and end with a complete word. They then taught the AI specifically on these perfect chunks. This stopped the AI from making up words and drastically improved accuracy.

Part 2: The Speaker Problem (The "Who's Talking?" Puzzle)

The Challenge:
In a crowded room, people talk over each other. Standard systems often get lost, thinking two people are speaking at once when the rules say only one person can be "on the mic" at a time. Also, the system that identifies the speaker often uses a different "ear" (Voice Activity Detector) than the system that writes the text, causing them to disagree on exactly when a sentence started or ended.

The Solution: The "Specialized Detective"

• Training the Detective: They took a general-purpose speaker-detecting AI and gave it a crash course specifically on Bengali conversation styles. It's like taking a detective who knows how to solve crimes in New York and training them specifically on the slang and habits of a specific neighborhood in Dhaka.
• The "Exclusive" Rule: The competition required that at any given second, only one person is credited with speaking. The team used a special feature that forces the AI to pick the most likely speaker for that exact moment, rather than letting it guess two people are talking.
• The "Double-Check" (VAD Intersection): This is the secret sauce. The text system and the speaker system usually have different ideas about when silence ends and speech begins. The team forced them to agree by only accepting speaker labels that matched the text system's "silence detector."
  • Analogy: Imagine two security guards checking a guest list. Guard A (Text) says, "The party starts at 8:00." Guard B (Speaker) says, "The party starts at 8:05." The team made a rule: "We only let people in if both guards agree." This eliminated false alarms where the system thought someone was talking during silence.

Part 3: The Results (The Scoreboard)

Before their system, the best attempts had a high error rate (about 67% of words were wrong or the wrong speaker was identified).

• Transcription: By using their "Smart Scissors" and specialized training, they dropped the error rate to 25%. That's a massive improvement, like going from a student who fails every test to one who gets a B+.
• Speaker Identification: They reduced the error in identifying speakers by nearly half compared to standard tools.

The Big Picture Takeaway

The team didn't invent a brand new type of brain; they invented a better workflow.

1. They stopped chopping audio randomly and started cutting it at natural word breaks.
2. They taught the AI specifically on Bengali conversation quirks.
3. They forced the different parts of the system to agree with each other before making a final decision.

In short: They turned a messy, hour-long recording into a clean, organized script where the words are correct and the speakers are clearly identified, proving that even for a language with fewer digital resources (like Bengali), smart engineering can beat brute force.

Technical Summary

Here is a detailed technical summary of the paper "WhisperAlign: Word-Boundary-Aware ASR and WhisperX-Anchored Pyannote Diarization for Long-Form Bengali Speech."

1. Problem Statement

The paper addresses the dual challenges of Long-Form Bengali Speech Recognition (ASR) and Speaker Diarization within the context of the DL Sprint 4.0 competition. The core difficulties include:

• Low-Resource Language: Bengali lacks robust linguistic tools, phoneme dictionaries, and pre-trained models compared to English.
• Long-Form Complexity: Processing hour-long, multi-speaker audio introduces issues with context preservation, overlapping speech, and Voice Activity Detection (VAD).
• Model Limitations: Standard models (like Whisper) suffer from "hallucinations" when decoding segments longer than 30 seconds or when processing silence. Traditional fixed-length audio splitting often truncates words mid-utterance, destroying semantic context.
• Diarization Mismatch: Existing diarization models often fail to capture Bengali conversational dynamics (prosody and turn-taking) and struggle to align with ASR timestamps due to differing VAD models.

2. Methodology

The authors propose a dual-stream pipeline: one for ASR and one for Speaker Diarization, both heavily optimized for the Bengali language.

A. ASR Pipeline: Word-Boundary-Aware Chunking & Fine-Tuning

Instead of splitting audio by fixed time intervals, the authors developed a self-contained, word-boundary-aware chunking strategy:

1. Silero VAD Pre-processing: Isolates speech regions to remove silence and background noise, ensuring chunks only contain active speech.
2. Timestamp Extraction: Uses whisper-timestamped (leveraging cross-attention heads) to generate per-word start/end timestamps on 28-second windows.
3. Sequence Matching & Alignment: Aligns Whisper's word sequence with ground-truth text using difflib.SequenceMatcher.
   • Correctly transcribed words keep Whisper's timestamps.
   • Mis-transcribed words retain ground-truth spelling but borrow Whisper's timestamps.
   • Missing words are interpolated; hallucinated words are discarded.
4. Smart Chunking: The aligned word list is partitioned into segments strictly respecting word boundaries, ensuring no word is cut off. Segments are filtered to be between 20–28 seconds (optimal for Whisper's context window).
5. Fine-Tuning: A Bengali-specific Whisper-medium model (tugstugi_bengaliai-asr-whisper-medium) is fine-tuned on these clean, boundary-respecting chunks using teacher forcing.
6. Inference: Uses greedy decoding with condition_on_previous_text=False to prevent error propagation, followed by post-processing filters for n-gram repetition and English boilerplate removal.

B. Diarization Pipeline: Pyannote Adaptation & VAD Intersection

The diarization system focuses on domain adaptation and temporal alignment:

1. Parameter-Efficient Fine-Tuning: Instead of retraining the massive pipeline, only the temporal segmentation model (~1.4M parameters) of Pyannote is fine-tuned on Bengali audio. This teaches the model Bengali-specific prosody and turn-taking syntax.
2. Exclusive Overlap Handling: Utilizes Pyannote's native exclusive_speaker_diarization feature. This probabilistically assigns overlapping speech to the earliest dominant speaker, satisfying the competition's requirement for mutually exclusive tracks without manually slicing data (which destroys acoustic information).
3. WhisperX VAD Intersection: To solve the "temporal drift" caused by different VAD models in ASR and Diarization, the authors perform a logical AND intersection. Pyannote's clustering outputs are masked strictly against the Silero VAD boundaries used by WhisperX. This ensures perfect alignment between text and speaker labels and eliminates ambient hallucinations.

3. Key Contributions

• Self-Contained Chunking Strategy: A novel method to create utterance-level training data from long-form recordings without external alignment tools (like Montreal Forced Aligner) or pronunciation lexicons, which are unavailable for Bengali.
• Domain-Specific Segmentation Fine-Tuning: Demonstrated that fine-tuning only the lightweight segmentation module of Pyannote on Bengali data significantly improves diarization accuracy compared to generic models.
• VAD Intersection Mechanism: Introduced a robust method to align ASR and Diarization outputs by intersecting their respective VAD masks, effectively eliminating boundary hallucinations and temporal drift.
• Exclusive Speaker Handling: Successfully adapted the exclusive_speaker_diarization feature (previously limited to premium models) to open-weight models (community-1), enabling high-precision non-overlapping speaker tracks.

4. Results

The system achieved state-of-the-art performance on the competition benchmarks:

ASR Performance (Word Error Rate - WER):

• Baseline: The raw tugstugi model had a Public WER of 0.675.
• Final Result: The full pipeline (VAD + Fine-tuning + Post-processing) achieved a Public WER of 0.252 and a Private WER of 0.278.
• Improvement: A reduction of over 62% in WER from the baseline. The most significant gain came from the chunk-and-align fine-tuning strategy.
• Comparison: Outperformed the Hishab TITU-BN baseline by ~23 absolute points and the raw tugstugi baseline by ~7 points.

Diarization Performance (Diarization Error Rate - DER):

• Baseline: The standard Pyannote 3.1 base model had a Public DER of 0.422.
• Final Result: The fine-tuned community-1 model with VAD intersection achieved a Public DER of 0.194 and a Private DER of 0.260.
• Improvement: A reduction of over 54% in DER.
• Benchmark: The proposed system reduced DER by 12–16% compared to the state-of-the-art Bengali-Loop baseline (40.08%).

5. Significance

This paper provides a robust, computationally efficient blueprint for scaling speech technologies in low-resource languages.

• Practicality: It proves that high-accuracy long-form transcription and diarization can be achieved without expensive external alignment tools or massive retraining, relying instead on intelligent data preprocessing and targeted fine-tuning.
• Generalizability: The "word-boundary-aware chunking" and "VAD intersection" techniques are transferable to other low-resource languages where phoneme dictionaries are missing.
• Efficiency: The system processes hour-long files in minutes (approx. 6–7 mins per file) using standard Kaggle GPU environments, making it accessible for real-world deployment.
• Solving the Overlap Problem: By leveraging native probabilistic overlap resolution rather than brittle manual heuristics, the system maintains acoustic integrity while meeting strict non-overlap constraints.