💬 NLP

Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition

This paper benchmarks nine speech recognition models across five demographic axes and various acoustic conditions, revealing that audio encoder design and compression levels—not LLM scale—are the primary determinants of fairness and robustness, with specific findings that LLM decoders do not inherently amplify racial bias but can exhibit severe, demographic-selective hallucinations under silence injection or masking.

Original authors: Srishti Ginjala, Eric Fosler-Lussier, Christopher W. Myers, Srinivasan Parthasarathy

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Srishti Ginjala, Eric Fosler-Lussier, Christopher W. Myers, Srinivasan Parthasarathy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of transcribers (people who listen to audio and type out what is said). For years, these transcribers were like strict stenographers who just wrote down exactly what they heard, even if the speaker had a heavy accent or the room was noisy.

Recently, we started hiring super-smart AI assistants to help these transcribers. These assistants have read millions of books and know exactly how English should sound. They can guess what a speaker meant to say, even if the audio is fuzzy.

But here's the big question: Does having a super-smart assistant make things fairer for everyone, or does it accidentally make things harder for people with accents or different backgrounds?

This paper is like a massive, rigorous "stress test" to find out. The researchers tested nine different AI systems (from old-school ones to the newest, most powerful ones) on about 43,000 different voices. They asked: Who gets the best transcription? Who gets the worst? And what happens when the audio is terrible?

Here is the breakdown of their findings, using some simple analogies:

1. The "Smart Assistant" Paradox

The Fear: People worried that because these new AI assistants were trained mostly on "standard" English books, they would ignore or misunderstand people with accents (like Indian, African, or regional US accents), making the errors worse.

The Surprise: The new AI assistants did not make racial bias worse. In fact, the most fair system (Granite-8B) used the smartest AI assistant.

The Catch: While the AI didn't make racial bias worse, it did create a new problem. Because the AI is so good at guessing "standard" English, it sometimes gets too confident. If a speaker has a slight accent, the AI might "hallucinate"—meaning it confidently types out words that weren't said at all, just because it thinks that's what should have been said.

2. The "Whisper" Glitch

One specific model, called Whisper, had a very strange bug.

The Analogy: Imagine a translator who is so eager to help that when they hear a difficult accent, they start repeating themselves or making up entire sentences.
The Finding: When Whisper tried to transcribe Indian-accented speech, it started "hallucinating" wildly. It would insert random words at a rate of nearly 10% (which is huge in this field). It was like a student who, when they don't know the answer, just starts writing a whole new essay instead of admitting they don't know.
The Twist: This didn't happen with other accents. It was a specific "crash" for Indian accents.

3. The "Compression" Factor (The Real Hero)

The paper found that the size of the AI (how many "brain cells" it has) wasn't the most important thing. The most important thing was how the audio was squeezed before it reached the AI.

The Analogy: Think of audio like a water hose.
- Low Compression: The hose is wide open. The AI gets a clear, full stream of sound.
- High Compression: The hose is pinched tight. The AI only gets a few drops of water and has to guess the rest.
The Result: Models that used "wide hoses" (low compression) were much fairer and more accurate for people with accents. Models that used "pinched hoses" (high compression) struggled, even if they had a super-smart AI brain. The quality of the audio pipeline matters more than the size of the AI.

4. The "Noise" Test (When Things Go Wrong)

The researchers didn't just test in a quiet room. They tested with loud noise, echoes, and even silence (where they cut out chunks of the audio).

The "Silence" Trap: When they cut out parts of the audio (silence injection), the Whisper model went crazy. It started "hallucinating" specific words for specific groups of people. For example, it would make up words for African accents but not Canadian ones. This made the unfairness 4.6 times worse.
The "Masking" Effect: When they blocked out audio (masking), the old models and the Whisper model started looping, repeating the same phrase over and over (like a broken record). The newer AI models (LLM-based) were much better at stopping this loop, unless the audio was heavily compressed.

5. The "Low-Accuracy Parity" Trap

Here is a weird finding: When the audio was extremely bad (very noisy or silent), the differences between groups actually disappeared.

The Analogy: Imagine a race where the track is on fire. Everyone trips and falls. The gap between the fastest runner and the slowest runner disappears, but nobody wins.
The Lesson: Just because the "gap" between groups looks small in bad conditions, it doesn't mean the system is fair. It just means the system is failing everyone equally.

The Bottom Line

If you are building a speech-to-text system (like for a phone or a medical app):

Don't just buy the biggest AI. A massive brain doesn't fix a broken microphone.
Focus on the audio pipeline. How you process the sound before it hits the AI is the most critical factor for fairness.
Beware of "Silence." If your system has to guess what was said during silence, it might start making up facts, especially for people with accents.
The New Guard: The best systems right now are the ones that use a smart AI but keep the audio signal clear and uncompressed.

In short: The new AI decoders are powerful, but they aren't a magic wand. If you feed them bad audio or compress it too much, they will still be unfair, and sometimes they will lie to you with confidence.

1. Problem Statement

As Automatic Speech Recognition (ASR) systems evolve from traditional Connectionist Temporal Classification (CTC) models to architectures utilizing Large Language Models (LLMs) as decoders, a critical question arises: Do the text-derived priors of pretrained LLMs make recognition fairer or more biased across demographic groups?

While previous studies have documented demographic biases in ASR (e.g., higher error rates for Black speakers or non-native accents), it remains unclear whether integrating explicit, text-only pretrained LLMs into the decoder backbone exacerbates these disparities. The authors hypothesize that LLMs, trained predominantly on standard written English, might amplify existing gaps when decoding non-standard or accented speech, particularly under real-world acoustic degradation where the model must rely more heavily on its language prior.

2. Methodology

The study employs a systematic benchmarking framework evaluating nine ASR models across three architectural generations on approximately 43,000 utterances.

Models Evaluated (Generations)

Generation 1 (No LM): Wav2Vec2-large (CTC encoder, no language model).
Generation 2 (Implicit LM): Three Whisper checkpoints (Small, Medium, Large-v3). These learn an implicit language model from paired audio-transcription training data.
Generation 3 (Explicit LM): Five models routing audio embeddings into pretrained LLM backbones:
- Qwen3-ASR (0.6B, 1.7B): Low-compression direct projection.
- Canary-Qwen-2.5B: Medium compression via FastConformer.
- Granite-Speech (2B, 8B): High compression via a Q-former bottleneck.

Datasets & Demographics

Common Voice 24: Crowd-sourced read speech with self-reported accents (6 groups), gender, and age.
Fair-Speech: Controlled-prompt dataset (same sentences for all speakers) eliminating vocabulary confounds. Includes ethnicity (7 groups), gender, age, first language (30+), and socioeconomic status.
LibriSpeech: Used as a clean reference baseline.

Evaluation Metrics

Accuracy: Word Error Rate (WER).
Fairness: Max/Min Ratio (MMR) of WER across demographic groups ( $MMR = \max(WER_g) / \min(WER_g)$ ). Lower is better; 1.0 is perfect parity.
Error Decomposition: Substitutions, Deletions, and Insertions (specifically analyzing hallucinations: repetition loops, syntactic completions, and content fabrications).

Perturbation Experiments

To simulate real-world conditions, the authors stress-tested all models under 12 acoustic degradation conditions across 4 types:

Additive Noise: SNR 20/10/0 dB.
Reverberation: RT60 0.3/0.6/1.0 s.
Silence Injection: 25/50/75% of utterance duration removed.
Chunk Masking: 10/20/30% of audio zeroed out.
Total: 216 inference runs (9 models × 12 conditions × 2 datasets).

3. Key Contributions

First Systematic Study: Isolates the effect of language model integration (None vs. Implicit vs. Explicit) on ASR fairness across five demographic axes and 12 degradation conditions.
Racial Bias Findings: Demonstrates that LLM decoders do not inherently amplify racial bias; in some cases, they reduce it.
Pathological Hallucination: Identifies a specific failure mode in Whisper-large-v3 where it exhibits non-monotonic performance degradation on Indian-accented speech due to massive insertion rates (hallucinations).
Compression vs. Scale: Shows that audio compression (how raw waveform is transformed before the decoder) is a stronger predictor of accent fairness than LLM parameter scale.
Degradation Dynamics: Reveals that severe degradation generally compresses fairness gaps (all groups fail equally), but silence injection uniquely amplifies bias in specific models by triggering demographic-selective hallucinations.
Architecture-Dependent Hallucinations: Proves that hallucination types (repetition loops vs. content fabrication) are determined by the decoder architecture and audio compression level, not just the presence of an LLM.

4. Key Results

A. Clean Audio Performance

Ethnicity Fairness: Black/African-American speakers consistently face the highest WER across all models. However, Gen 3 models (LLM decoders) do not worsen this gap compared to Gen 2. Granite-8B (Gen 3) achieved the best ethnicity fairness (MMR = 2.28), outperforming all Whisper models.
Accent Fairness & Hallucination:
- Whisper-large-v3 showed a pathological spike in insertion rates for Indian accents (9.62%), worse than its smaller variants. This was driven by autoregressive repetition loops and semantic fabrications.
- Qwen3-1.7B (Gen 3, low compression) was the most equitable model for accents (MMR = 1.82).
- Compression Matters: Among Gen 3 models, low-compression architectures (Qwen3) outperformed high-compression ones (Granite) on accent fairness, suggesting fine-grained phonetic cues are lost in heavy compression.

B. Performance Under Acoustic Degradation

The "Low-Accuracy Parity" Effect: Under severe masking (30% chunk masking), fairness gaps compressed (MMR decreased) for all models. As all groups suffered catastrophic error rates, relative disparities shrank. This indicates that "fairness" in degraded conditions can be a byproduct of universal failure rather than equitable robustness.
Silence Injection Anomaly: Unlike masking, silence injection amplified bias.
- Whisper-small saw its accent fairness gap amplify by 4.64× (MMR jumped from 3.23 to 15.0) under 25% silence, driven by African-accent hallucinations.
- Whisper-large-v3 shifted its bias under 75% silence, worsening England-accent performance while Indian-accent performance improved (as the hallucination pathway was disrupted).
Repetition Loops:
- Whisper-small produced 86% repetition loops under masking (catastrophic autoregressive cycling).
- Explicit-LLM decoders (Qwen3) produced 38× fewer insertions with near-zero repetition.
- Critical Finding: High audio compression (Granite-Speech with Q-former) reintroduced repetition pathology even in explicit LLM decoders, proving that the audio encoder design is the primary lever for robustness.

C. Scaling Trajectories

Whisper: Scaling up (Small → Large-v3) improved ethnicity fairness but worsened accent fairness due to the Indian-accent hallucination pathology.
Qwen3: Scaling up (0.6B → 1.7B) improved both accuracy and fairness.
Granite: Scaling up (2B → 8B) showed dataset-dependent effects, improving ethnicity fairness but worsening accent fairness on Common Voice, likely due to the Q-former bottleneck limiting capacity for diverse acoustic features.

5. Significance and Implications

Decoder Architecture > LLM Scale: The study concludes that audio encoder design (specifically the level of compression before the decoder) is more critical for equitable recognition than the size of the LLM decoder. High compression leads to loss of phonetic detail and increased hallucination risks.
Deployment Recommendations:
- Whisper-large-v3 should not be deployed for accented speech without hallucination mitigation, as it exhibits non-monotonic scaling and accent-selective failure.
- Low-compression LLM decoders (like Qwen3-1.7B) offer the best balance of accuracy, fairness, and robustness.
- Practitioners must report both absolute WER and relative disparity metrics; a low MMR can mask "low-accuracy parity" where all groups perform poorly.
Robustness is Context-Dependent: "Robustness" is not monolithic. A model robust to noise may be fragile to silence, and fairness under clean conditions does not guarantee fairness under degradation.

In summary, the paper challenges the assumption that larger LLMs automatically solve ASR bias. Instead, it highlights that architectural choices regarding audio compression and decoder type are the primary determinants of whether a speech recognition system is equitable and robust.