💬 NLP

Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs

This paper proposes a multilingual speech correction pipeline that combines token-level disfluency detection with instruction fine-tuning and a contrastive learning objective to effectively remove fillers and repetitions from ASR transcripts while preserving grammatical structure and semantic coherence, demonstrating superior performance over existing baselines in Hindi, Bengali, and Marathi.

Original authors: Deepak Kumar, Baban Gain, Asif Ekbal

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Deepak Kumar, Baban Gain, Asif Ekbal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Stuttering" Robot

Imagine you are talking to a robot assistant. You say, "Um, I think, uh, the meeting is at, like, 3 PM."

The robot's hearing system (called ASR) writes down exactly what it hears: "Um, I think, uh, the meeting is at, like, 3 PM."

While this is accurate to the sound, it's a mess for the robot's brain (the Large Language Model or LLM) to understand. It's like trying to read a book where the author kept stopping to clear their throat, repeat words, or start sentences over. This "noise" confuses the robot, making it give bad answers, translate things poorly, or sound unnatural when it speaks back to you.

Most old methods tried to fix this by acting like a pair of scissors: they would find the "um" and "uh" and just cut them out. But often, this left the sentence looking broken, like a sentence missing a piece of a puzzle.

The Solution: A Two-Step "Editor" Team

The authors propose a new, smarter team to fix these transcripts. Think of it as a two-person editing crew working on a messy draft.

Step 1: The Highlighter (The Detector)

First, they use a specialized tool (called MuRIL) that acts like a highlighter pen. It scans the messy sentence and highlights exactly which words are "garbage" (fillers, repetitions) and which are "gold" (the real meaning).

Analogy: Imagine a teacher grading a student's essay. Instead of just erasing the mistakes, the teacher circles them in red ink so the student knows exactly what to fix.

Step 2: The Rewrite Artist (The LLM)

Next, they give this highlighted draft to a powerful AI writer (an LLM). The AI is told: "Here is the messy sentence, and here are the red circles. Rewrite this into a perfect, smooth sentence, but keep the original meaning."

The Secret Sauce: The "Anti-Repeat" Rule

Here is the paper's biggest innovation. Usually, when you teach an AI to rewrite, it learns by looking at the "good" answer and trying to match it. But sometimes, the AI gets lazy and accidentally copies the bad words (the "ums" and "uhs") anyway.

To stop this, the authors added a special rule called Contrastive Learning.

Analogy: Imagine you are teaching a child to bake a cake.
- Standard Training: You show them a perfect cake and say, "Make one like this."
- The Paper's Method: You show them the perfect cake, but you also put a big red "X" over the burnt cookies they made last time. You say, "Make a cake like this, but do not put burnt cookies in it."

This "Anti-Repeat" rule actively punishes the AI if it tries to put the "um" or "uh" back into the sentence. It forces the AI to be extra careful to leave the garbage behind.

What They Tested

The team tested this on three Indian languages: Hindi, Bengali, and Marathi. These languages are tricky because they have complex grammar and lots of different ways people stutter or restart sentences.

They compared their new method against:

Old Scissors: Just cutting out words.
Smart Guessers: AI models that try to guess the fix without help.
Big Famous Models: Like GPT-4 and Gemini, which are very expensive and powerful.

The Results

The paper found that their "Two-Step Team with the Anti-Repeat Rule" was the winner.

Better than the Scissors: It fixed the sentences without breaking the grammar.
Better than the Guessers: It understood the context much better.
Beating the Giants: Surprisingly, their small, specialized model did just as well as, or sometimes better than, the massive, expensive models like GPT-4, especially on real-world, messy audio recordings.

Why It Matters

The authors showed that if you don't clean up the "stuttering" in speech, the robot's brain gets confused.

Question Answering: The robot gives dumber answers.
Translation: The translation gets worse.
Speaking Back: When the robot speaks back, it sounds robotic and awkward.

By using this new method, the robot can take a messy, stuttering human voice, turn it into a clean, smooth sentence, and then understand it perfectly.

In a Nutshell

The paper introduces a smart way to clean up speech transcripts. Instead of just deleting mistakes, it uses a "highlighter" to find them and a "rewrite artist" to fix them, with a special rule that makes sure the artist never accidentally puts the mistakes back in. This makes voice assistants and chatbots work much better, especially for languages like Hindi, Bengali, and Marathi.

Technical Summary: Mind the Pause

Problem Statement
Automatic Speech Recognition (ASR) systems frequently preserve disfluencies from spontaneous speech, such as fillers (e.g., "uh," "um"), repetitions, false starts, and self-repairs. While these artifacts are natural in human conversation, their presence in ASR transcripts degrades readability and significantly hampers downstream Natural Language Processing (NLP) applications, including chatbots, voice assistants, and machine translation.

Existing approaches primarily treat disfluency handling as a detection-then-deletion problem. While effective at identifying disfluent tokens, these methods often disrupt grammatical structure and semantic coherence, resulting in ungrammatical or broken sentences. Furthermore, while Large Language Models (LLMs) have been explored for disfluency detection or data augmentation, they have not been fully integrated into a comprehensive correction pipeline that preserves meaning while removing artifacts. Empirical analysis in the paper reveals that even state-of-the-art ASR systems (e.g., Whisper v3 Large, AI4Bharat Indic Conformer) fail to consistently normalize these disfluencies, leaving approximately 30% of transcribed sentences containing at least one disfluency. This persistence causes measurable degradation in downstream tasks, reducing question-answering accuracy by 0.5–1.6 points and machine translation quality by 2–4.7 BLEU points.

Methodology
The authors propose a multilingual disfluency correction pipeline that unifies token-level detection with instruction-tuned LLM generation, enhanced by a novel contrastive learning objective. The framework operates in three main stages:

Disfluency Detection (MuRIL): A multilingual encoder-based sequence tagger, specifically MuRIL (Multilingual Representations for Indian Languages), is fine-tuned to classify tokens as either fluent (0) or disfluent (1). This model provides token-level signals indicating which parts of the ASR transcript are noisy.
Instruction Fine-Tuning: An LLM (tested with Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct) is fine-tuned using an instruction-following paradigm. The input to the LLM consists of:
- A natural language instruction defining the task.
- The original disfluent ASR transcript.
- The token-level labels predicted by MuRIL.
- A list of identified disfluent tokens.
  The target output is the corresponding fluent reference sentence. This setup guides the LLM to rewrite the text while explicitly conditioning on the detected disfluency signals.
Contrastive Learning Objective: To address the limitation where models trained solely with cross-entropy loss occasionally regenerate disfluent tokens despite receiving labels, the authors introduce a contrastive loss component. This objective explicitly penalizes the model if the generated output contains tokens identified as disfluent. The total loss function combines standard cross-entropy ( $L_{CE}$ ) with a contrastive penalty ( $L_{contrastive}$ ):
$L_{total} = L_{CE} + \lambda \cdot L_{contrastive}$
The contrastive term calculates the probability of generating known disfluent tokens and applies a penalty, effectively pushing the model's generation away from disfluent patterns while pulling it toward fluent targets.

Key Contributions

Contrastive Loss for Disfluency-Aware Tuning: The paper introduces a multi-loss objective that combines cross-entropy with a contrastive penalty. This explicitly discourages the regeneration of known disfluent tokens, a capability often missing in standard instruction tuning.
First LLM-Based Correction Pipeline: Unlike prior work focusing on detection-only or encoder-decoder architectures (e.g., mBART), this work presents the first complete pipeline integrating token-level detection signals with instruction-tuned LLMs for multilingual correction.
Joint Detection and Correction in Multilingual Settings: The framework trains MuRIL for sequence tagging and feeds both the disfluent transcript and predicted labels into an LLM. This avoids the pitfalls of manual token deletion, ensuring the generation of well-formed sentences that preserve semantic integrity.

Results
The system was evaluated on three Indian languages: Hindi, Bengali, and Marathi, using both manually edited synthetic data and real ASR transcripts.

Performance Gains: The proposed contrastive approach consistently outperformed strong baselines, including multilingual instruction fine-tuning and mBART.
- Compared to non-contrastive training, the method achieved a +1.97 BLEU improvement (averaged across models and languages).
- Compared to multilingual instruction fine-tuning, it achieved a +6.16 BLEU improvement.
- Compared to mBART, it achieved an +8.54 BLEU improvement.
Model Comparison: The Qwen2.5-3B-Instruct backbone benefited more significantly from the contrastive objective than Llama-3.2-3B-Instruct, showing larger gains in BLEU and chrF2 scores.
Downstream Impact: Corrected outputs recovered the majority of the performance gap between disfluent and fluent inputs in downstream tasks (Question Answering, Machine Translation, and Text-to-Speech), confirming practical utility.
Comparison with Proprietary Models: The 3B-parameter contrastive model matched or outperformed GPT-4o (under zero-shot and few-shot prompting) on 4 out of 6 evaluation conditions and substantially outperformed Gemini 2.5 Pro across all three languages.

Significance and Claims
The paper claims that detection-only strategies are insufficient for high-quality disfluency correction. By combining token-level cues with instruction tuning and contrastive learning, the authors provide a scalable, language-agnostic solution for multilingual speech-driven NLP systems. The work demonstrates that task-specific contrastive training is necessary to handle the morphological richness and syntactic complexity of Indic disfluencies, a challenge that scaling model size or advanced prompting alone cannot resolve. The authors emphasize that their approach effectively bridges the gap between detection and generation, producing fluent, meaning-preserving text that enhances the reliability of downstream applications.