Qualitative Evaluation of Language Model Rescoring in Automatic Speech Recognition

1. Problem Statement

Automatic Speech Recognition (ASR) systems are predominantly evaluated using the Word Error Rate (WER). While WER is simple and widely adopted, the authors argue it suffers from significant limitations:

• Lack of Linguistic Depth: It treats all errors (substitutions, insertions, deletions) as equivalent, ignoring grammatical categories (Part-of-Speech) and semantic meaning.
• Task Agnosticism: It does not account for the fact that some words are more critical to a specific downstream task than others.
• Binary Nature: A substitution is counted as a full error regardless of whether the wrong word is semantically or syntactically close to the correct one (e.g., "Patricia" vs. "playing" vs. "Patrick").

The paper addresses the need for a more granular, qualitative evaluation of ASR systems, specifically to understand how Language Model (LM) rescoring impacts transcription quality across different linguistic dimensions (morpho-syntactic and semantic).

2. Methodology

The authors propose a framework that supplements WER with six additional automatic metrics derived from Natural Language Processing (NLP) tasks. These metrics are applied to a French ASR system to evaluate the impact of a posteriori rescoring.

A. Proposed Metrics

The study introduces and utilizes the following metrics:

1. Character Error Rate (CER): Measures errors at the character level to distinguish between minor morphological errors (e.g., tense, gender) and major lexical errors.
2. Part-of-Speech Error Rate (POSER): Calculates error rates based on grammatical tags rather than words. Two variants are used:
   • dPOSER: Uses detailed POS tags (e.g., "feminine plural noun").
   • uPOSER: Uses generic Universal Dependency tags (e.g., "noun").
3. Lemma Error Rate (LER): Calculates errors based on the dictionary form (lemma) of words, ignoring inflectional variations.
4. Embedding Error Rate (EmbER): A novel metric that weights WER based on semantic distance. Instead of a binary 0/1 error, the penalty is scaled by the cosine similarity between the reference word's embedding and the hypothesis word's embedding.
5. BERTScore: A metric using contextual embeddings (WordPiece) to calculate precision, recall, and F-measure based on semantic proximity between reference and hypothesis tokens.
6. Sentence Semantic Distance (SemDist): Uses SentenceBERT to generate sentence-level embeddings and calculates the average cosine similarity between the reference and hypothesis sentences.

B. Experimental Protocol

• Dataset: French broadcast data (Radio/TV) from ESTER 1 & 2, EPAC, ETAPE, REPERE, and internal LIA data (~940 hours training, ~10 hours test on REPERE).
• ASR System: A state-of-the-art Kaldi-based system using a TDNNF acoustic model with data augmentation.
• Rescoring Setup:
  • Base System: Uses a trigram LM.
  • Rescoring System: Applies a posteriori rescoring using a Quadrigram LM and a Recurrent Neural Network Language Model (RNNLM) consisting of TDNN and LSTM layers.
• Tools: Flair (for POS tagging), SpaCy (for lemmatization), FastText (for word embeddings), and SentenceBERT (for sentence embeddings).

3. Key Contributions

1. Multi-Dimensional Evaluation Framework: The paper moves beyond WER by introducing a suite of metrics that separately evaluate morpho-syntactic (POS, Lemmas) and semantic (Embeddings, Sentence similarity) aspects of transcription errors.
2. Introduction of EmbER: A hybrid metric that bridges the gap between traditional error counting and semantic embedding analysis, weighting errors by semantic distance.
3. Qualitative Analysis of Rescoring: The study provides a deep dive into how and where LM rescoring improves ASR, revealing that improvements are not uniform across all linguistic levels.

4. Results and Analysis

A. Metric Correlations

• Low Correlation with SemDist: The Sentence Semantic Distance metric showed the lowest correlation with other metrics (including WER), suggesting that minimizing WER does not guarantee better sentence-level semantic alignment. This supports the view that intrinsic metrics (like WER) may not fully predict extrinsic task performance.
• EmbER as a Bridge: EmbER correlated well with both WER and embedding-based metrics (BERTScore, SemDist), validating its role as a hybrid measure.
• POS and Lemmas: High correlation was observed between Lemma Error Rate (LER) and POSER, indicating that grammatical errors often coincide with lemma errors.

B. Impact of Rescoring

The study compared the Base System (Trigram only) against the Rescoring System (Trigram + Quadrigram + RNNLM).

• Overall Improvement: Rescoring reduced error rates across all metrics.
  • WER Reduction: 14.3%
  • CER Reduction: 10.2%
  • POSER Reduction: ~11-14%
  • Semantic Metrics (SemDist, BERTScore): Showed the lowest relative gains (~8-9%).
• Interpretation: LM rescoring is highly effective at correcting syntactic and lexical errors (grammar, spelling, word choice) but is less effective at correcting words that are semantically distant from the reference. The semantic structure of the sentence is harder to fix via standard LM rescoring.
• POS-Specific Gains:
  • High Benefit: Interjections (INTJ) and Conjunctions (CCONJ) saw the largest reductions in semantic distance.
  • Low Benefit: Determinants (DET) and Numbers (NUM) saw the least improvement.
• Spontaneity Issue: The rescoring process deteriorated performance on spontaneous speech segments containing disfluencies (elisions, reductions). Utterances with more errors post-rescoring had 1.23x more spontaneity markers, suggesting a mismatch between the training data (likely cleaner) and the testing conditions (spontaneous speech).

5. Significance

• Beyond WER: The paper demonstrates that relying solely on WER provides an incomplete picture of ASR performance. A system might improve WER significantly while failing to improve the semantic coherence required for downstream NLP tasks.
• Targeted Optimization: By identifying that rescoring helps syntax more than semantics, developers can better tune their systems. For example, if the goal is semantic understanding, standard LM rescoring may need to be supplemented with semantic-specific models.
• Robustness Warning: The finding that rescoring harms spontaneous speech highlights a critical area for future research: making ASR systems more robust to disfluencies and non-standard speech patterns.
• Future Directions: The authors suggest combining these metrics (e.g., analyzing semantics specifically for proper nouns) and evaluating the correlation between these metrics and human perception of error quality.

In conclusion, this paper provides a robust methodology for qualitative ASR evaluation, proving that while LM rescoring improves overall accuracy, its benefits are unevenly distributed across grammatical and semantic dimensions, necessitating a multi-metric approach for comprehensive system assessment.