Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding

Whisper-CD is a training-free, inference-time contrastive decoding framework that mitigates hallucinations and repetition in long-form speech recognition by contrasting clean audio logits against a unified objective derived from multiple acoustically motivated negative perturbations, thereby significantly reducing word error rates and improving generation throughput without requiring model retraining.

The Gist

Imagine you have a very smart, well-read librarian named Whisper. This librarian is amazing at listening to long recordings of people talking and writing down exactly what they said. However, when the recording is very long, has background noise, or has long pauses, the librarian sometimes gets confused.

Instead of just saying "silence," the librarian might start making things up (hallucinations), repeating the same sentence over and over like a broken record, or skipping entire parts of the story. This gets worse if the librarian tries to use their previous notes to help them guess what comes next; if they made a mistake earlier, they might double down on it, creating a chain reaction of errors.

The paper introduces a new method called Whisper-CD to fix this. Think of it as a "Second Opinion System" that works in real-time without needing to retrain the librarian.

Here is how it works, using simple analogies:

1. The Problem: The "Confident Mistake"

When the librarian is unsure (like during a long silence), they might confidently write down, "Thank you for watching," even though no one said that. Because they are so confident, the system keeps writing it, and then repeats it forever.

2. The Solution: The "Three Troublemakers"

Whisper-CD doesn't just ask the librarian to listen once. It asks the librarian to listen to four different versions of the same audio clip at the exact same time:

• Version A (The Original): The clean audio. This is the main answer.
• Version B (The Static): The audio is covered in heavy static noise (like turning a radio dial between stations). This makes it hard to hear what is being said, so the librarian has to guess based on their own habits.
• Version C (The Silence): The audio is completely muted. The librarian is just staring at a blank screen but still trying to guess what words come next.
• Version D (The Time-Traveler): The audio is shifted slightly in time. The librarian hears the end of a sentence before the beginning, creating a confusing mismatch.

3. The Magic Trick: "Contrastive Decoding"

Now, here is the clever part. The system compares the librarian's answer from Version A against the answers from Versions B, C, and D.

• If the librarian says a word in the clean audio (Version A) that they also say when the audio is just static or silence (Versions B & C), the system realizes: "Wait a minute! The librarian is just guessing based on their own habits, not listening to the actual sound!"
• The system then says, "No, don't write that word down. It's likely a hallucination."
• It effectively subtracts the "guessing" tendency from the final answer.

It's like a detective who asks a witness, "What did you see?" If the witness gives the same answer whether they are looking through a foggy window, a dark room, or a distorted mirror, the detective knows the witness is just making things up. But if the answer changes based on the conditions, the detective knows they are actually seeing something real.

4. Why It's a Big Deal

• No Retraining: You don't need to teach the librarian new skills or change their brain. You just change how you ask them the question. This means it can be used on any existing Whisper system immediately.
• Speed: Usually, to get better accuracy, you have to make the system "think harder" (like checking many different possibilities), which makes it slow. Whisper-CD is surprisingly fast. It's actually 48% faster than the standard "check many possibilities" method because it's very good at knowing exactly what not to say, so it doesn't waste time guessing wrong paths.
• Fixes the "Broken Record": It stops the repetition loops and the "Thank you for watching" nonsense that happens in long recordings.

Summary

Whisper-CD is like giving a smart but occasionally daydreaming librarian a set of "reality checks." By comparing what they say when the audio is clear against what they say when the audio is messed up, the system can spot and cancel out the librarian's daydreams, resulting in a perfect transcript without needing to hire a new librarian or retrain the old one.

Technical Summary

Here is a detailed technical summary of the paper "Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding."

1. Problem Statement

Large-scale encoder-decoder models like Whisper have revolutionized Automatic Speech Recognition (ASR), yet they struggle significantly with long-form transcription. When processing extended recordings, these models frequently exhibit three recurring error patterns:

1. Hallucinations: Generating fluent but factually unsupported text, particularly during non-speech intervals (silence) or noisy segments (e.g., "Thank you for watching").
2. Repetition Loops: The model gets stuck repeating phrases across segment boundaries.
3. Content Omissions: Skipping parts of the spoken content.

These errors are exacerbated in long-form processing because Whisper often uses a "divide-and-conquer" approach where the transcription of a previous segment is passed as context to the next. If the previous segment contains an error, it biases the current decoding, causing errors to accumulate and amplify across the entire recording. Existing solutions often require architectural changes, model retraining, or auxiliary modules (like VAD), making them difficult to deploy on already-trained systems.

2. Methodology: Whisper-CD

The authors propose Whisper-CD, a training-free framework that operates purely at inference time. It utilizes Contrastive Decoding (CD) to suppress hallucinations without modifying model parameters.

Core Mechanism

Instead of relying on a single generation path, Whisper-CD contrasts the logits (prediction scores) of the clean audio (positive) against logits derived from acoustically perturbed inputs (negative). The goal is to down-weight tokens that the model predicts with high confidence even when the acoustic evidence is degraded or removed.

The final contrastive logit ($\ell^{CD}_t$) is calculated as:
$$\ell^{CD}_t = (1 + \alpha\tau) \ell^{pos}_t - \alpha\tau \log\left( \frac{1}{K} \sum_{k=1}^{K} \exp(\ell^{neg}_{k,t} / \tau) \right)$$
Where:

• $\ell^{pos}_t$: Logits from the original clean audio.
• $\ell^{neg}_{k,t}$: Logits from $K$ different perturbed inputs.
• $\alpha$: Contrastive strength coefficient.
• $\tau$: Temperature parameter for aggregating negatives (set to 1.0 for arithmetic mean).

Three Acoustic Perturbation Strategies

To capture diverse failure modes, the framework generates three distinct "negative" signals:

1. Gaussian Noise Injection: Adds noise to the waveform (target SNR = 10dB). This weakens fine-grained phonetic evidence, revealing tokens the model prefers even under acoustic uncertainty.
2. Silence Signal: Replaces the input spectrogram with all zeros. This removes all spectral structure, forcing the decoder to rely solely on its unconditional textual prior. This specifically targets "bag of hallucinations" (stock phrases generated during silence).
3. Audio Temporal Shift: Shifts the waveform leftward (discarding the first $\Delta s$ samples and zero-padding the end). This creates a misalignment between the acoustic content and the expected temporal position, disrupting segment-boundary alignment and targeting repetition loops.

Inference Efficiency

To maintain efficiency, the encoder processes the clean input and all three perturbed inputs in a single batched forward pass. During autoregressive decoding, the paths are packed along the batch dimension, allowing a single decoder forward pass per step while reusing the same prefix tokens. This avoids the computational overhead of sequential processing.

3. Key Contributions

• First Application of CD to ASR: This is the first work to apply contrastive decoding specifically to reduce ASR hallucinations, moving beyond its previous use in vision-language and NLP tasks.
• Multi-Negative Objective: Unlike prior CD methods that use a single negative signal, Whisper-CD aggregates three acoustically motivated perturbations to address multiple failure modes (silence hallucinations, repetition, and alignment errors) simultaneously.
• Training-Free & Drop-in Replacement: The method requires no retraining, fine-tuning, or architectural changes. It can be applied to any existing Whisper deployment as a drop-in replacement.
• Context Preservation: Unlike approaches that disable context passing to avoid error accumulation, Whisper-CD preserves the ability to use previous context while actively suppressing the propagation of errors.

4. Experimental Results

The method was evaluated on five English long-form benchmarks (CORAAL, Earnings22, VoxPopuli, TED-LIUM, REV-16) using Whisper Large-v3 and Large-v3-Turbo.

• Accuracy Improvements:
  • Whisper-CD reduced Word Error Rate (WER) by up to 24.3 percentage points on the challenging CORAAL dataset.
  • It significantly suppressed repetition loops, which often caused baseline WERs to exceed 100% (due to inflated output lengths).
  • On CORAAL, the baseline Large-v3-Turbo had a WER of 38.75%, which dropped to 14.43% with Whisper-CD.
• Comparison to Beam Search:
  • While Beam Search (beam size 5) improved accuracy on some datasets, it degraded performance on others (e.g., TED-LIUM) and was significantly slower.
  • Whisper-CD achieved lower WER than Beam Search on all datasets while being 48% faster in token generation throughput.
• Ablation Studies:
  • Using a single perturbation strategy (e.g., only silence) was less effective than the multi-negative combination, confirming that different perturbations target distinct error types.
  • The contrastive strength ($\alpha$) is dataset-dependent; higher $\alpha$ values helped noisy datasets but could over-correct cleaner datasets.

5. Significance

Whisper-CD addresses a critical bottleneck in deploying ASR for long-form content (podcasts, meetings, lectures). By providing a computationally efficient, training-free solution that drastically reduces hallucinations and repetition loops, it enables the reliable use of large pre-trained models in real-world scenarios without the cost of retraining or complex architectural modifications. The ability to outperform beam search in both accuracy and speed makes it a highly practical advancement for production ASR systems.