💬 NLP

HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics

This paper introduces HATS, a novel French dataset containing human-perceived transcription errors from 143 annotators, to investigate and evaluate the correlation between human preferences and various automatic speech recognition metrics.

Original authors: Thibault Bañeras Roux, Jane Wottawa, Mickael Rouvier, Teva Merlin, Richard Dufour

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thibault Bañeras Roux, Jane Wottawa, Mickael Rouvier, Teva Merlin, Richard Dufour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge in a cooking competition. Two chefs (let's call them Chef A and Chef B) have made a dish based on a secret recipe. Your job is to taste both and decide which one is closer to the original recipe.

In the world of computers, this is exactly what Automatic Speech Recognition (ASR) does. It listens to a human speaking and tries to write down the words, just like a chef trying to recreate a dish.

For a long time, the only way to judge these "computer chefs" was by counting mistakes. If the computer wrote "cat" instead of "bat," that was one error. If it wrote "c-a-t" instead of "b-a-t," that was also an error. This is called the Word Error Rate (WER). It's like a judge who only counts how many ingredients are missing, without tasting the flavor.

The Problem: The "Counting" Judge is Too Strict

The authors of this paper argue that the "counting" judge isn't very good at understanding what humans actually care about. Sometimes, a computer makes a mistake that changes a word but keeps the meaning clear (like saying "I ate an apple" instead of "I ate a pear"). A human listener would understand the sentence perfectly fine, but the "counting" judge marks it as a failure.

The researchers wanted to know: Which computer metric actually matches what a human thinks is a "good" transcription?

The Solution: The HATS Experiment

To find out, the team created a new dataset called HATS (Human Assessed Transcription Side-by-Side). Think of this as a massive, organized taste-testing event.

The Setup: They gathered 143 human volunteers.
The Task: They played a recording of someone speaking. Then, they showed the volunteers two different computer-generated transcripts of that same recording. Both transcripts had errors.
The Choice: The volunteers had to pick which of the two written versions was "better" or easier to understand. They couldn't say "they are equal." They had to pick a winner.

The researchers specifically chose difficult cases where the two computer versions were very similar in quality, forcing the humans to really think about the nuances of language.

The Results: What Did the Humans Prefer?

The researchers then compared the humans' choices against various computer scoring systems to see which one agreed with the humans the most.

Here is what they found, using simple analogies:

The Old Way (WER/CER): The traditional "Word Error Rate" was like a judge who only counts how many letters are wrong. It performed poorly. It often picked the "worse" transcript compared to what humans preferred.
The New Way (Semantic Metrics): The researchers tested smarter metrics that look at the meaning of the words, not just the spelling. They used tools like BERTScore and SemDist.
- Imagine BERTScore as a judge who understands that "big" and "huge" mean the same thing, even if the words are different.
- Imagine SemDist as a judge who looks at the whole sentence structure to see if the "vibe" or meaning is correct.

The Winner: The metric that agreed most with the human volunteers was SemDist (specifically using a model called Sentence-BERT). It was the best at predicting which transcript a human would find easier to understand.

A Surprising Twist: The "Sound" of Text

One interesting finding was about Phoneme Error Rate (PhonER). This metric counts errors based on how words sound (their sounds), not how they are spelled.

Even though the humans were only reading the text (not listening to the audio), the "sound-based" metric actually predicted human choices better than the "word-counting" metric.
The Analogy: It's as if the humans, while reading the text, were subconsciously "hearing" the words in their heads. They preferred the text that sounded right, even if they were just looking at the letters.

The Takeaway

The paper concludes that if you want to know if a speech-to-text system is actually good for humans, you shouldn't just count the spelling mistakes. You need a metric that understands the meaning and flow of the sentence.

They released their "taste-test" data (HATS) for free so other scientists can use it to build better, more human-friendly speech recognition systems.

Important Note: The authors warn that this study was done entirely in French. Just because these rules work for French doesn't mean they automatically work for English or other languages. Also, because they picked very tricky examples for the test, the data might not represent every single mistake a computer makes in the real world.

1. Problem Statement

Automatic Speech Recognition (ASR) systems are traditionally evaluated using Word Error Rate (WER) and Character Error Rate (CER). These metrics count the number of incorrect words or characters relative to a reference transcript. However, the authors argue that these metrics are insufficient because:

They are system-oriented rather than human-oriented.
They lack semantic understanding (e.g., substituting "cat" with "dog" is penalized the same as "cat" with "car," even though the former might be more comprehensible).
They do not account for the fact that humans can often extract meaning from a sentence despite transcription errors, provided the errors are not fundamental.

There is a need for evaluation metrics that correlate more closely with human perception of transcription quality, particularly for transcripts intended for human consumption (e.g., captions, messages).

2. Methodology

The HATS Dataset

The core contribution is the creation of HATS (Human-Assessed Transcription Side-by-Side), a new open dataset for the French language.

Source: Transcripts were generated from the REPERE test corpus (approx. 10 hours of radio/TV broadcast audio) using 10 different ASR systems (8 end-to-end systems based on Speechbrain/wav2vec2/XLS-R, and 2 DNN-HMM systems based on Kaldi).
Annotation Protocol:
- Task: A "side-by-side" experiment where 143 human participants were presented with a written reference transcript and two automatic hypotheses (from different ASR systems).
- Constraint: Participants had to choose the "best" hypothesis. They were not allowed to select "equal."
- Stimuli Selection: To ensure the task was challenging and informative, triplets were not selected randomly. They were filtered based on three criteria:
  1. Hypotheses must differ from each other and the reference.
  2. Hypotheses from all systems were contrasted against each other.
  3. Metric-based selection: Pairs were chosen to create specific scenarios:
    - Same/Different scores: Metrics agree or disagree on the quality.
    - Metric Conflict: One metric favors Hypothesis A while another favors Hypothesis B.
    - Ambiguity: Cases where WER/CER are equal but semantic metrics differ.
Data Volume: 1,000 reference-hypothesis triplets, resulting in 7,150 total annotations (each triplet evaluated by at least 7 participants).
Inter-rater Reliability: Fleiss' Kappa was 0.46, with 82% of triplets showing at least 71.4% agreement among humans, indicating the task is difficult but humans can consistently identify the better option.

Evaluation Metrics Tested

The study evaluated how well various automatic metrics align with human choices. The metrics included:

Lexical: WER, CER, Phoneme Error Rate (PhonER).
Semantic/Embedding-based:
- Embedding Error Rate (EmbER): WER weighted by cosine distance of word embeddings (fastText).
- SemDist: Cosine similarity between sentence-level embeddings (tested with CamemBERT, FlauBERT, SentenceBERT).
- BERTScore: Token-level similarity using contextual embeddings (tested with multilingual BERT, CamemBERT-base, CamemBERT-large).

Analysis Approach

The authors calculated the agreement rate between the metric's prediction (which hypothesis is better) and the human consensus.

Consensus Definition: Agreement was calculated as $\frac{\max(A, B)}{A + B}$ , where $A$ and $B$ are the counts of humans choosing Hypothesis A or B.
Filtering: Results were analyzed under three conditions:
1. 100% Agreement: Only triplets where all humans agreed.
2. 70% Agreement: Triplets with strong consensus.
3. 0% Agreement (Full): No filtering applied.

3. Key Results

The performance of metrics was measured by the percentage of times they matched human preference.

Metric Type	Best Performing Metric	Performance (70% Agreement)	Key Findings
Lexical	CER	64%	CER outperformed WER. This contradicts some previous studies and may be due to French orthography (silent letters) or the use of written text references. WER performed near random chance (53%) in this dataset.
Phonemic	PhonER	69%	Performed better than WER and CER, suggesting humans subconsciously consider phonetic similarity even when reading text.
Semantic	SemDist (Sentence CamemBERT-large)	78%	The top-performing metric. It consistently outperformed others, likely because Sentence-BERT embeddings are specifically trained to maximize semantic similarity.
Semantic	BERTScore (CamemBERT-large)	68%	Performed well but slightly lower than SemDist when using the same embedding models.

Notable Observations:

Embedding Sensitivity: The choice of embedding model significantly impacts metric performance. For instance, SemDist with FlauBERT-base (59%) performed worse than CER (60%), while SemDist with Sentence CamemBERT-large (78%) was the best.
Metric Disagreement: In cases where WER and CER gave equal scores, semantic metrics often provided the differentiator that aligned with human choice.
Human Perception: Humans seem to prioritize semantic coherence and phonetic flow over strict lexical accuracy, which lexical metrics fail to capture.

4. Key Contributions

HATS Dataset: The release of a large-scale, manually annotated dataset of French ASR errors with human preference labels. It is the first to use a side-by-side protocol with 10 diverse ASR systems and rigorous stimulus selection criteria.
Empirical Evaluation: A comprehensive benchmark of lexical vs. semantic metrics against human perception in French.
Insight on French ASR: The finding that CER is a better proxy for human perception than WER in French contexts, challenging the universal dominance of WER.
Semantic Metric Validation: Demonstration that SemDist (using Sentence-BERT embeddings) is currently the most reliable automatic metric for predicting human preference in ASR evaluation.

5. Significance and Future Work

Significance: The study provides a roadmap for moving ASR evaluation beyond simple error counting toward human-centric quality assessment. It validates that embedding-based metrics are necessary for evaluating transcripts intended for human consumption.
Limitations:
- The dataset is specific to French; results may not generalize to other languages.
- The stimuli were selected based on strict criteria, potentially making the dataset less representative of "average" errors.
- The reference was text, not audio.
Future Directions:
- Replicating the experiment with an audio reference to test if CER remains superior to WER when visual text cues are removed.
- Developing new metrics that balance human correlation with computational efficiency and interpretability (a current strength of WER/CER).
- Expanding the dataset to other languages to test cross-lingual generalizability.

In conclusion, the paper argues that while WER and CER remain useful for their low cost and interpretability, SemDist with Sentence-BERT is the superior metric for aligning ASR evaluation with human perception, and the HATS dataset is a critical resource for developing and validating future evaluation methods.