Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs

Technical Summary: Preference-ASR

Problem Statement
Current Automatic Speech Recognition (ASR) benchmarks fail to evaluate the capabilities of modern Speech-Augmented Large Language Models (SpeechLLMs). While SpeechLLMs can follow natural-language instructions to dictate transcription style and format, standard evaluation relies on fixed human-annotated references and "blanket" normalization pipelines (e.g., lowercasing, converting numbers to words, removing symbols). These pipelines erase the very formatting distinctions users care about. Furthermore, existing test sets (e.g., Librispeech, SPGISpeech, Earnings-22) exhibit inconsistent annotation conventions regarding number formatting, disfluency handling, and casing. Consequently, model rankings often reflect how well a system matches a specific dataset's quirks rather than its genuine ability to follow user preferences.

Methodology
The authors introduce Preference-ASR, a test set designed to evaluate ASR systems on their ability to follow natural-language preference instructions. The methodology consists of three core components:

1. Dataset Construction:

   • Source: The dataset is built from seven open-source corpora (AMI, Common Voice, Earnings-22, GigaSpeech, LibriSpeech, SPGISpeech, VoxPopuli), initially pooling 3,545 samples.
   • Pipeline: A two-stage, LLM-assisted pipeline with human verification was employed.
     • Stage 1: An LLM (Qwen3-30B) classifies each sample into one of four preference categories: Normalization, Entities, Disfluencies, or Case.
     • Stage 2: The LLM generates specific instructions and corresponding "preference texts" (references) for each sample.
   • Deduplication: A priority-based ordering (Normalization > Entities > Disfluencies > Case) ensures each sample belongs to a single category.
   • Final Output: The released dataset contains 3,210 unique triples (audio, instruction, reference) and 335 baseline samples without preference instructions.

2. Preference Categories:

   • Normalization: Tests handling of numbers (digits vs. words), symbols (currency/math notation vs. verbal equivalents), and website links.
   • Entities: Tests recognition of specific named entities (companies, drugs, etc.) and measures "confabulation" by including false-positive entity names in instructions that are absent from the audio.
   • Disfluencies: Tests the ability to retain or remove filler words (um, uh), repetitions, false starts, and grammatical errors based on instructions.
   • Case: Tests adherence to lowercase-only or proper punctuation/capitalization rules.

3. Preference-Aware Normalizer:
   To evaluate compliance fairly, the authors propose a custom normalizer that selectively skips normalization steps matching the active instruction. For example, if an instruction requests written-form numbers (ITN), the normalizer does not convert the model's digit output to words, preventing the penalization of correct behavior.

Experimental Setup
The study benchmarks four models:

• Parakeet-TDT-0.6B-v3: A traditional non-LLM baseline (no instruction input).
• Canary-Qwen-2.5B: A hybrid model with a Qwen 2.5 LLM backend, trained on standard ASR pairs without preference tuning.
• Phi-4-Multimodal: A 5.6B model integrating speech, vision, and text.
• Qwen3-Omni-30B: A native multi-modal model trained with contextual biasing.

Models were evaluated under two settings: Default (standard prompt) and Instructed (preference instruction appended). Performance was measured using both Standard WER (full normalization) and Preference-Aware WER (selective normalization).

Key Results

• Ranking Shifts: Model rankings change significantly depending on the preference type, a phenomenon obscured by standard evaluation. For instance, Qwen3-Omni outperforms others in normalization and disfluency compliance when instructed, but its performance degrades under standard evaluation which masks these differences.
• Instruction Sensitivity:
  • Qwen3-Omni shows strong compliance for formatting (normalization, case, disfluencies) but exhibits entity hallucination, inserting entity names from prompts even when absent from the audio (WER spikes from 5.12% to 12.85% in entities).
  • Canary-Qwen shows minimal sensitivity to instructions across all categories, suggesting that an LLM backbone alone, without preference-specific alignment training, is insufficient for instruction-following ASR.
  • Phi-4 demonstrates strong compliance for disfluencies when instructed but fails significantly on case instructions, indicating category-dependent instruction following.
• Prompt Sensitivity: Even minor rephrasing of generic prompts (without preference directives) caused measurable WER shifts in SpeechLLMs (e.g., Phi-4), highlighting their sensitivity to prompt wording.

Significance and Claims
The paper claims that Preference-ASR is the first test set to condition ground truth on explicit human-specified formatting instructions, thereby exposing quality differences that traditional benchmarks miss. The authors argue that as SpeechLLMs evolve, evaluation must move beyond fixed references to directly test instruction-following capabilities. The results highlight a critical trade-off: while models can follow formatting instructions, they struggle to balance textual prompt context against acoustic evidence, leading to hallucinations in entity recognition. The authors conclude that current benchmarks are insufficient for the era of SpeechLLMs and that future work must address multi-speaker preferences and multilingual settings.

Limitations
The authors acknowledge that the dataset is English-only, does not yet cover multi-speaker preferences, and required manual verification of LLM-generated texts, particularly for normalization tasks where acoustic context cannot be inferred solely from text.