What Counts as an Error? Dual-Reference Benchmarking for Atypical ASR

Technical Summary: Dual-Reference Benchmarking for Atypical ASR

Problem Statement
Current Automatic Speech Recognition (ASR) systems frequently underperform on atypical speech, particularly stuttered speech. A critical, often overlooked issue in evaluating these systems is the conflation of two distinct, valid transcription references: verbatim (capturing the actual produced speech, including repetitions and prolongations) and intended (the canonical form with disfluencies removed). Most existing evaluations rely solely on intended transcripts, implicitly rewarding models that delete disfluencies while penalizing those that preserve them. This single-reference approach obscures model capabilities, creates misleading rankings, and fails to address the specific needs of diverse stakeholders, such as speech-language pathologists who require verbatim fidelity for clinical assessment versus users of voice assistants who prioritize semantic intent.

Methodology
The authors benchmarked 11 ASR models representing three primary architectural families: Autoregressive (Seq2Seq), Transducer, and CTC (Connectionist Temporal Classification). The evaluation utilized the FluencyBank Timestamped dataset, which contains both verbatim and intended transcriptions for stuttered speech. To further analyze performance relative to specific stuttering phenomena, the authors aligned these audio segments with CASA (Clinical Stutter Events) annotations, categorizing events into types such as Syllable Repetition (SR), Incomplete Syllable Repetition (ISR), Multisyllable Unit Repetition (MUR), Prolongation (P), and Blocks (B).

The study employed a dual-reference evaluation framework:

1. Intended Speech WER (isWER): Measured against intended transcripts to assess semantic accuracy.
2. Verbatim WER (vWER): Measured against verbatim transcripts to assess acoustic faithfulness.
3. Semantic Metrics: SeMaScore and BERTScore were used alongside Word Error Rate (WER) to evaluate intended speech recognition nuances.

Crucially, the authors evaluated all models "as-is" using default open-source configurations without fine-tuning to ensure reproducibility and isolate the impact of architectural design on transcription style.

Key Results
The study revealed significant instability in model rankings depending on the chosen reference, challenging the notion of a single "best" ASR model for atypical speech:

• Architectural Specialization: A clear trade-off exists between model families. Autoregressive sequence-to-sequence models (e.g., NVIDIA Canary-1B-v2, Whisper Large-v3) consistently performed better on intended transcription, likely due to their ability to use linguistic context to normalize uncertain acoustic regions. Conversely, CTC-based models (e.g., NVIDIA CTC) excelled at verbatim transcription, as their reliance on local acoustic evidence allows them to preserve disfluencies more faithfully.
• Ranking Volatility: The top-ranked model for intended speech (NVIDIA Canary-1B-v2, isWER 13.85) ranked only 5th for verbatim speech, while the top verbatim model (NVIDIA CTC, vWER 17.20) ranked 5th for intended speech.
• Event-Specific Errors: Performance varied significantly by stutter event type. Prolongations (P) were the easiest for models to handle in both contexts. However, Incomplete Syllable Repetitions (ISR) and Multisyllable Unit Repetitions (MUR) proved most difficult. For intended tasks, models struggled to remove disfluent fragments; for verbatim tasks, they struggled to accurately preserve the repeated segments.
• Metric Limitations: The authors found that standard WER is overly strict for intended speech (penalizing formatting variants like "20" vs. "twenty"), while semantic metrics like SeMaScore and BERTScore have blind spots, sometimes assigning high similarity scores to meaning-breaking errors (e.g., "bully" vs. "bullet").

Significance and Claims
The paper argues that the definition of "accuracy" in atypical ASR is inherently dependent on the downstream use-case. By conflating verbatim and intended references, current research practices risk "hard-coding" normative decisions about how stuttered speech should be represented, potentially masking systemic biases.

The authors claim that:

1. Dual-Reference Benchmarking is Essential: A single ground truth is insufficient for atypical speech. Valid model selection requires evaluating performance against both verbatim and intended references to understand a system's specific strengths and failure modes.
2. Architecture Dictates Output Style: The choice of modeling paradigm (Autoregressive vs. CTC) systematically biases the output toward either semantic normalization or acoustic preservation.
3. Transparency in Reporting: Future research must explicitly state which transcription reference is being optimized for. For clinical and diagnostic applications, verbatim references and vWER should be reported; for dictation and command/control, intended references and isWER (supplemented by semantic metrics) are appropriate.

The work concludes that building inclusive speech technology requires acknowledging that atypical speech legitimately maps to multiple textual outputs, and evaluation metrics must reflect this duality rather than forcing a single, potentially misleading, standard.