OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics

Technical Summary of OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics

Problem Statement
While deep learning and transformer-based architectures have enabled robust multilingual Automatic Speech Recognition (ASR) models, the evaluation of their transcription accuracy remains hindered by limitations in standard metrics. The predominant metric, Word Error Rate (WER), relies on the Levenshtein distance but suffers from several critical issues:

1. Lack of Linguistic Nuance: As a purely string-based metric, WER treats all errors equally, ignoring semantic impact and human-perceived quality.
2. Cross-Lingual Bias: Improvements to WER, such as text normalization, are predominantly available for English. This creates a disparity in evaluating low-resource languages and hinders fair cross-lingual comparisons.
3. Destructive Pre-processing: Standard normalization often involves removing punctuation and converting text to lowercase. While this reduces WER scores, it destroys information regarding casing and punctuation errors, which are crucial for assessing the readability and utility of ASR output.
4. Compound Word Errors: Standard tokenization often fails to account for compound words (e.g., "game-changer" vs. "game changer"), leading to inflated error rates that do not reflect actual semantic misunderstandings.

Methodology
The authors present OpenWER, an open-source Python library designed to improve WER robustness and enable granular, token-based accuracy metrics. The system employs a modular architecture with the following core components:

• Token-Based Alignment: Unlike traditional string-list comparisons, OpenWER uses tokens as alignment units. This preserves original text, metadata (such as timestamps and confidence scores), and allows for the differentiation between words and punctuation during the alignment process.
• Non-Destructive Normalization: The system applies language-specific normalization (e.g., handling contractions and spelling variations) without destroying original token values. Metadata is preserved throughout the process, allowing for the tracking of modifications.
• Extended Levenshtein Algorithm:
  • Variable Costs: The algorithm assigns lower costs to punctuation edits and higher penalties for substitutions between different token types to improve semantic relevance.
  • Compound Word Detection: The algorithm is extended to detect compound words by merging adjacent tokens during cost matrix computation. This addresses errors caused by missing or inserted hyphens and spaces without requiring a dictionary.
  • Punctuation Preservation: The alignment handles punctuation tokens explicitly, allowing them to be preserved in the edit path rather than removed.
• Flexible Input and Tokenization: OpenWER accepts both raw text and structured ASR word lists. It supports basic tokenization as well as NLP-based tokenizers (e.g., spaCy) to incorporate Part-of-Speech (POS) tags or Named-Entity Recognition (NER) labels as metadata.

Key Contributions

1. OpenWER Library: An open-source implementation that improves WER robustness across high- and low-resource languages through language-specific normalization and compound word detection.
2. Token-Based Metrics: The ability to calculate complementary metrics for punctuation and casing accuracy, as well as custom metrics derived from NLP metadata (e.g., noun accuracy or proper name accuracy).
3. Granular Evaluation: The framework supports the integration of ASR model metadata (such as confidence scores) to enable novel evaluations, such as correlating confidence with word correctness.

Results
The authors evaluated OpenWER using the Common Voice 17 dataset, transcribing 691,648 samples across 52 languages using Whisper (large-v3) and SeamlessM4T (v2-large) models. Key findings include:

• WER Reduction: OpenWER achieved absolute WER reductions of up to 25% compared to commonly used libraries (specifically JiWER). The average reduction was 7.7% (14.5% relative) compared to JiWER and 3.8% (9.2% relative) compared to Whisper.
• Compound Word Impact: Enabling compound word detection reduced WER by up to 20% for certain languages, though the impact varied significantly by language.
• Statistical Equivalence: In scenarios without normalization or with standard normalization, OpenWER produced results statistically equivalent to JiWER, confirming that the modified algorithm does not alter standard WER calculations unless specific enhancements are applied.
• Complementary Metrics: The evaluation of 10 ASR models on an English dataset demonstrated that models with similar WERs could have vastly different punctuation and casing accuracy (e.g., IBM showed a 98.3% punctuation error rate vs. 22.0% WER).
• Performance: While OpenWER is implemented in Python and is slower than the C-based JiWER (1.9 tokens/ms vs. 4618.9 tokens/ms), Just-in-Time (JIT) compilation improved OpenWER's speed to 34.8 tokens/ms.

Significance
The paper claims that OpenWER contributes to fairness in ASR research by increasing the reliability of WER across diverse languages. By moving away from destructive pre-processing and enabling the preservation of punctuation, casing, and metadata, OpenWER allows for more comprehensive and granular accuracy evaluations. The authors posit that while WER remains the standard due to the lack of widely adopted alternatives, OpenWER addresses its limitations by reducing noise from non-semantic string differences and providing a foundation for future metrics that better reflect transcription quality. The open-source, modular nature of the library is intended to encourage community-driven improvements and the development of language-specific normalizers for low-resource languages.