💬 NLP

Evaluation of Automatic Speech Recognition Using Generative Large Language Models

This paper demonstrates that decoder-based Large Language Models significantly outperform traditional Word Error Rate and existing semantic metrics in evaluating Automatic Speech Recognition by achieving 92–94% agreement with human annotators on hypothesis selection and offering a promising path toward interpretable, meaning-aware assessment.

Original authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a student's essay. For decades, the only way you've graded these essays was by counting how many words were spelled wrong or swapped out. This is like the old standard for checking speech-to-text software (called Word Error Rate or WER).

But here's the problem: If a student writes "The cat sat on the mat" and another writes "The feline rested on the rug," the old grading system would give them a failing grade because almost every word is different. Yet, a human teacher knows both sentences mean the exact same thing! The old system is too rigid; it counts typos but misses the meaning.

This paper asks a simple question: Can we use a super-smart AI (a Large Language Model or LLM) to grade these speech transcripts the way a human would?

Here is the breakdown of their experiment, using some everyday analogies:

1. The "Taste Test" (Choosing the Best Option)

The Setup: Imagine you have two versions of a transcript from a messy recording.

Version A: "It's to him self." (Grammatically wrong, but clear).
Version B: "It's uh to him-self." (Has a stutter, but grammatically correct).

The Old Way (WER): The computer counts the errors. It might say Version A is better because it has fewer "mistakes," even though Version B sounds more natural to a human ear.

The New Way (The AI Judge): The researchers asked a super-smart AI (like GPT-4 or Qwen) to read both and pick the one that makes the most sense to a human.

The Result: The AI was a fantastic judge. It agreed with human experts 92–94% of the time.
The Analogy: Think of the old system as a robot counting how many letters are missing. The new AI is like a seasoned editor who understands that a little stutter ("uh") is fine, but a broken sentence is not. The AI was much better at spotting what actually matters to a human listener.

2. The "Fingerprint" (Measuring Meaning with Math)

The Setup: Sometimes you don't just want to pick a winner; you want to measure how close two sentences are in meaning.

The Old Way: You take a "fingerprint" of the sentence using older AI models (Encoders). These are like high-quality cameras that take a picture of the meaning.
The New Way: The researchers tried taking fingerprints using the new "Decoder" LLMs (the chatbots). They wondered: Do these chatbots understand meaning as well as the specialized cameras?

The Result: Surprisingly, the chatbots were just as good at capturing the "fingerprint" of meaning as the specialized cameras.

The Twist: They found that just grabbing the very last word of the sentence (like looking at the last frame of a movie) wasn't enough to understand the whole story. Instead, they had to look at the average of the whole sentence.
The Analogy: Imagine trying to guess the plot of a movie by only looking at the final scene. You might get it wrong. But if you look at the average of all the scenes, you get the whole story. The AI works best when it considers the whole "movie," not just the ending.

3. The "Report Card" (Categorizing Errors)

The Setup: Instead of just giving a score, the researchers asked the AI to give a qualitative grade, like a teacher's comment. They asked the AI to sort errors into four buckets:

Identical: Perfect.
Useful: A few typos, but you get the point.
Bad: The meaning is messed up, but you can guess what was meant.
Incomprehensible: Total gibberish.

The Result: The AI was excellent at this. It didn't just give a number; it explained why a transcript was bad.

The Analogy: The old system gives you a score of "7/10." The new AI gives you a report card that says, "You lost points because you missed the main idea, but the spelling was okay." This is much more helpful for fixing the software.

The Big Takeaway

For years, we've been measuring speech recognition with a ruler that only measures length (Word Error Rate). This paper shows that we should be using a smart, understanding AI instead.

It's more human: It cares about meaning, not just spelling.
It's interpretable: It can tell you why something is wrong, not just that it is wrong.
It's accessible: Open-source AI models (free ones) performed just as well as the expensive, paid ones.

In short, we are moving from a world where computers count our mistakes to a world where computers understand our intent. This is a huge step forward for making voice technology that actually works for real people.

1. Problem Statement

Automatic Speech Recognition (ASR) systems are traditionally evaluated using Word Error Rate (WER) and Character Error Rate (CER). While these metrics are standard, the paper identifies critical limitations:

Lack of Semantic Sensitivity: WER treats all errors equally, failing to distinguish between minor typos and errors that drastically alter meaning or comprehension.
Poor Correlation with Human Perception: WER often correlates poorly with how humans perceive transcription quality, particularly regarding disfluencies, grammatical coherence, and overall semantic intent.
Interpretability Issues: Existing semantic metrics (e.g., cosine similarity between embeddings) produce continuous scores (e.g., 0.62) that are difficult to interpret compared to error percentages.
Underutilization of Decoder LLMs: While encoder-based models (like BERT) have been explored for semantic evaluation, decoder-based Large Language Models (LLMs) (e.g., GPT, Llama, Gemma) remain largely unexplored for this specific task despite their superior contextual understanding.

2. Methodology

The study utilizes the HATS dataset, which contains human annotations of speech recognition errors, to benchmark three distinct evaluation approaches using various LLMs (including GPT-4, Qwen, and Gemma families).

A. Hypothesis Selection ("LLM as Judge")

Task: The LLM is presented with a reference transcription and two incorrect hypotheses (A and B). It must select the hypothesis that best matches the reference in terms of lexical fidelity and semantic coherence.
Prompting Strategy: A one-shot Chain-of-Thought (CoT) prompt is used. The model is asked to justify its reasoning before outputting a final decision (A or B).
Goal: To evaluate if LLMs can replicate human judgment in choosing the "better" transcription.

B. Semantic Distance via Decoder Embeddings

Task: The paper investigates whether vector representations (embeddings) extracted from decoder-based LLMs can serve as effective semantic metrics (SemDist).
Process:
1. Transcriptions are tokenized and passed through the LLM to generate token embeddings.
2. Pooling Strategies: Since SemDist requires a fixed-dimension vector, the token sequence is aggregated using various strategies:
  - Last token ( $t_n$ )
  - Second-to-last token ( $t_{n-1}$ )
  - Mean pooling (average of all tokens)
  - Weighted mean (weights increase linearly with position)
  - Mean of the last 4 tokens.
3. Metric: Cosine similarity between the reference and hypothesis embeddings is calculated to derive a semantic distance score.
Goal: To determine if decoder LLMs can generate embeddings comparable to specialized encoder models (like CamemBERT) for semantic similarity tasks.

C. Qualitative Error Classification

Task: The LLM independently classifies a (Reference, Hypothesis) pair into one of four categories without comparing it to another hypothesis:
1. Identical: Exact match or minor formatting differences.
2. Useful: Meaning preserved despite minor errors (spelling, punctuation).
3. Bad: Meaning partially altered (keyword errors, omissions).
4. Incomprehensible: Meaning completely lost.
Goal: To assess the LLM's ability to provide interpretable, granular feedback on error severity.

3. Key Results

A. Hypothesis Selection Performance

Superiority over Traditional Metrics: The best LLMs (GPT-4.1 and Qwen3.5-35B) achieved 92–94% agreement with human annotators on the strict subset (100% human agreement).
Comparison: This significantly outperforms WER (63%), CER (77%), and even sophisticated semantic metrics like SemDist with CamemBERT-large (80%).
Model Size vs. Performance: While larger models generally performed better, size was not the sole determinant. For instance, Qwen3-8B outperformed Gemma3-27B, and Qwen3.5-35B matched the proprietary GPT-4.1. Open-access models proved competitive with state-of-the-art proprietary ones.

B. Embedding Quality and Pooling

Pooling Strategy: Contrary to intuition, the Last Token embedding (often used for classification in LLMs) performed poorly for semantic similarity (e.g., Qwen3.5-27B scored 59%). Mean pooling and Weighted Mean strategies yielded significantly better results (up to 89%).
- Reasoning: Decoder LLMs are trained for next-token prediction; the last token is optimized to predict the next word, not to represent the entire sentence's semantics.
Specialized vs. Generic Models: Models explicitly fine-tuned for embeddings (e.g., Qwen3-Embedding-8B) outperformed generic LLMs of similar size, achieving 89% agreement with human perception using mean pooling.
Decoder vs. Encoder: Decoder-based LLMs with proper pooling strategies achieved performance comparable to encoder-based models (like BERT/CamemBERT).

C. Qualitative Classification

LLMs successfully mapped qualitative error classes to expected semantic similarity scores.
The "Identical" class corresponded to high similarity scores, while "Incomprehensible" corresponded to low scores.
This approach offers a human-interpretable alternative to opaque continuous scores, allowing users to understand why a transcription is rated poorly (e.g., "bad" vs. "incomprehensible").

4. Key Contributions

Validation of "LLM as Judge": Demonstrated that generative LLMs can effectively replace or augment human annotators in selecting the best ASR hypothesis, achieving near-perfect agreement with humans and outperforming all existing automatic metrics.
Decoder Embedding Analysis: Provided a comprehensive analysis of how to extract semantic vectors from decoder LLMs, revealing that mean pooling is superior to last-token pooling for semantic similarity tasks and that specialized embedding models yield the best results.
Interpretable Evaluation Framework: Proposed a qualitative classification system (Identical/Useful/Bad/Incomprehensible) that bridges the gap between numerical metrics and human perception, offering actionable insights into error types.

5. Significance and Impact

Paradigm Shift: The paper suggests a shift from lexical error counting (WER) to semantic and perceptual evaluation driven by LLMs.
Cost Reduction: High agreement with human annotators implies that LLMs can significantly reduce the cost and time required for ASR evaluation and dataset annotation.
System Optimization: By aligning evaluation metrics with human perception, developers can better optimize ASR systems for real-world applications (e.g., accessibility, subtitles) where meaning preservation is more critical than exact word matching.
Open-Source Viability: The finding that open-source models (Qwen) rival proprietary giants (GPT-4) democratizes access to high-quality ASR evaluation tools.

In conclusion, the paper establishes that decoder-based LLMs, when applied with appropriate prompting and embedding strategies, offer a robust, interpretable, and highly accurate framework for the next generation of ASR evaluation.