💬 NLP

Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study

This paper presents a comparative study demonstrating that while properly configured encoder-based metrics (like BERTScore and SemDist) achieve strong correlation with human judgments for ASR evaluation, generative LLMs offer complementary strengths in pairwise hypothesis selection and improving the interpretability of error analysis.

Original authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

When a computer listens to a human voice and writes down what it hears, the process is called automatic speech recognition. For decades, the standard way to judge how well these systems work has been a simple count of mistakes. If the computer writes "cat" when the speaker said "bat," that counts as an error. If it adds an extra word or misses one, those are errors too. This method, known as word error rate, treats every mistake as equally bad. It does not care if the error changes the meaning of a sentence or if it is just a minor spelling variation that a human reader would easily overlook. As a result, a system could get a perfect score on this old test while still producing text that sounds robotic or confusing to a person.

To fix this, researchers have turned to newer methods that look at the meaning behind the words rather than just the letters themselves. These methods use large language models, which are massive computer programs trained on huge amounts of text to understand how language works. Some of these models are designed to read and understand text, while others are built to generate new text, like writing a story or answering a question. The big question for the field has been whether these powerful new tools can do a better job of judging speech recognition than the old counting methods, and if so, which type of model is the best judge.

A team of researchers at the Idiap Research Institute in Switzerland, along with colleagues from universities in France, set out to answer this question. They wanted to see how different kinds of large language models compare when asked to evaluate the quality of transcribed speech. They did not just look at the models in a general sense; they tested them in two very different ways. First, they treated the models as tools to measure the distance between the spoken words and the written text, looking for a mathematical score that matched human opinion. Second, they asked the models to act as a human-like judge, reading two different versions of a transcription and deciding which one was better, or simply labeling a single transcription as good, bad, or confusing.

The researchers tested their ideas using a dataset called HATS, which contains pairwise human preferences over French ASR hypotheses produced by various computer systems. Crucially, this dataset also includes the opinions of human listeners who were asked to rate the quality of the transcriptions. This allowed the researchers to see if the computer models were making the same judgments that people would make. They began by testing the models as measurement tools. They computed BERTScore and SemDist using embeddings extracted from the models to generate a score based on how similar the computer's version was to the human's version. They found that the best results did not come from simply using the biggest or most powerful model. Instead, success depended on how the researchers chose to process the information inside the model.

For the models that read and understand text, the researchers discovered that the best results came from looking at specific layers of the model's internal structure, rather than just the final output. They also found that models specifically trained to create sentence-level summaries performed better than general-purpose models. For the models that generate text, the results were similar: the best performance required careful selection of which part of the model's internal knowledge to use. In these tests, the models that were designed to generate text performed just as well as, and sometimes slightly better than, the models designed to read text. However, the reading-focused models achieved this with far fewer computer resources, making them a very efficient choice for this specific task.

The researchers then moved to the second part of their study, where they asked the models to act as judges. In the first scenario, they gave the model a reference sentence and two different computer-generated versions of that sentence. The model had to pick the one that a human would prefer. In this task, the most advanced generative models, including some of the largest available, performed exceptionally well. They agreed with human choices about 94 percent of the time in the best cases, outperforming both the old counting method and the new scoring methods. This suggests that when a model is allowed to read the whole sentence and think about the context, it can spot subtle differences in quality that simple math misses.

In the final experiment, the researchers asked the models to look at a single transcription and assign it a label: identical, useful, bad, or incomprehensible. This approach offered a new kind of insight. Instead of a single number, the researchers got a description of the error. For example, the model could identify that a sentence was understandable despite a few spelling mistakes, or that it was completely confusing. While the models were not perfect at this task, they showed a clear ability to organize errors by severity. The results suggested that these models could provide a more detailed and human-friendly report on why a speech system failed, rather than just saying it failed.

The study concludes that large language models offer a flexible and powerful new way to evaluate speech recognition. The reading-focused models provide fast and accurate scores that match human perception, while the text-generating models excel at comparing different versions of a transcript and explaining the nature of errors. The researchers found that the size of the model mattered less than how it was used; a smaller, well-configured model could often do the job of a much larger one. Ultimately, the work shows that by combining these different approaches, we can move beyond simple error counts to a deeper understanding of how well a computer truly understands the human voice.

Technical Summary: Generative vs. Encoder Large Language Models for ASR Evaluation

Problem Statement

Automatic Speech Recognition (ASR) systems are traditionally evaluated using Word Error Rate (WER). While WER is reproducible and simple, it relies on strict surface-form matching, failing to account for semantic similarity or human perception of transcription quality. Minor lexical variations (e.g., casing, synonyms) are penalized equally to meaning-altering errors, often obscuring genuine improvements in perceived quality. Although embedding-based semantic metrics (e.g., BERTScore, SemDist) have shown better correlation with human judgments, their performance is heavily dependent on the underlying model architecture (encoder vs. decoder), the specific layer used for representation, and the pooling strategy. Furthermore, the potential of decoder-only Large Language Models (LLMs) to serve as direct evaluators—beyond just providing embeddings—remains underexplored in structured ASR evaluation settings.

Methodology

The authors conduct a comparative study on the HATS dataset, which provides detailed human pairwise preferences and quality judgments for French ASR hypotheses. The evaluation is structured into three complementary dimensions:

Embedding-Based Semantic Metrics:
- Models: A diverse set of encoder models (e.g., CamemBERT, Sentence-CamemBERT) and decoder models (e.g., Qwen3, Gemma, including embedding-specialized variants) were tested.
- Metrics: The study analyzes BERTScore (token-level embedding comparison) and SemDist (sentence-level embedding distance).
- Variables: The authors systematically vary the representation layer (from early to final layers) and pooling strategies (last token, mean, mean without last, weighted mean, weighted mean without last) to determine optimal configurations for correlation with human judgments.
Generative LLMs as Judges (Pairwise Selection):
- Task: Decoder LLMs are prompted to act as judges, selecting the better hypothesis between two candidates given a reference transcription.
- Setup: A one-shot prompting strategy is used, requiring the model to justify its choice before outputting a selection (A or B).
- Comparison: The agreement between LLM decisions and human preferences is measured across different model sizes and families.
Generative LLMs for Qualitative Classification:
- Task: Models are asked to assign a qualitative label to a single (reference, hypothesis) pair without a direct competitor.
- Categories: Labels include identical, useful (meaning preserved despite minor errors), bad (meaning partially altered), and incomprehensible.
- Evaluation: Since the HATS dataset lacks these specific qualitative labels, the LLM predictions are correlated against SemDist scores (computed via Sentence-CamemBERT-Large) to validate the semantic ordering of the predicted categories.

Key Contributions and Results

1. Embedding-Based Metrics (Encoder vs. Decoder)

Layer and Pooling Sensitivity: The optimal representation layer is highly model-dependent; there is no universal "best" layer (e.g., final vs. intermediate). Similarly, pooling strategies significantly impact performance. For standard autoregressive models, mean and weighted pooling generally outperform last-token pooling. However, embedding-specialized models (e.g., Qwen3-Embedding) show that the last token can become a competitive representation after fine-tuning.
Architecture Comparison: Both encoder and decoder models can achieve strong correlation with human judgments when properly configured.
- Encoders: Sentence-CamemBERT-large achieves high performance with relatively few parameters, highlighting efficiency.
- Decoders: Embedding-specialized decoder models (e.g., Qwen3-Embedding-8B) reach similar or slightly higher scores than encoders but often at a significantly higher parameter cost.
Conclusion: Encoder-based metrics remain highly competitive and efficient, while decoder-based embeddings offer comparable performance when specialized for sentence representation.

2. Generative LLMs for Pairwise Hypothesis Selection

Performance: Generative LLMs significantly outperform both traditional lexical metrics (WER, CER) and the best embedding-based semantic metrics in pairwise selection tasks.
Top Performers: GPT-4.1 (94% agreement) and open-weight Qwen3.5-35B (92% agreement) on the 100% consensus subset demonstrate that open models can approach proprietary systems.
Capability Divergence: A notable finding is the lack of alignment between a model's ability to produce good embeddings (e.g., Qwen3-1.7B performing well in SemDist) and its ability to perform comparative reasoning (where Qwen3.5-35B excels). This suggests that intrinsic evaluation and comparative reasoning rely on different model capabilities.
Implication: Generative LLMs capture aspects of quality such as grammatical coherence and semantic consistency that fixed similarity functions may miss.

3. Qualitative Error Classification

Interpretability: LLMs can assign interpretable labels to transcriptions, offering a distribution of error severities rather than a single scalar score.
Correlation: While the correlation between predicted qualitative classes and continuous SemDist scores is moderate (Spearman $\rho \approx -0.6$ ), the models successfully order categories by severity (Identical > Useful > Bad > Incomprehensible).
Limitations: Substantial overlap exists between adjacent categories (particularly "useful" and "bad"), indicating that while LLMs provide meaningful qualitative descriptions, they are not yet a perfect replacement for continuous metrics.

Significance and Claims

The paper claims that LLMs provide a flexible framework for ASR evaluation that addresses the limitations of WER through three distinct approaches:

Efficiency: Encoder-based representations yield strong, efficient semantic metrics (BERTScore, SemDist) that rival large decoder models.
Accuracy in Comparison: Generative models serve as powerful "judges" for relative evaluation, outperforming traditional metrics by leveraging contextual understanding to select the best hypothesis.
Interpretability: Generative models enable the construction of qualitative error distributions, offering a more detailed and human-aligned view of system behavior than single-score metrics.

The authors conclude that while encoder models are efficient for semantic scoring, generative models excel in comparative judgment and qualitative analysis. Together, these approaches open the way toward more human-aligned and interpretable evaluation methodologies for speech recognition systems. The paper does not claim to replace WER entirely but suggests a shift toward metrics that better reflect human perception and provide actionable feedback.