Technical Summary: ESCUCHA – A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions

Problem Statement

Despite the rapid advancement of Large Audio Language Models (LALMs), robust evaluation frameworks for speech understanding under realistic, heterogeneous acoustic conditions remain insufficient. Existing benchmarks are predominantly English-centric, rely on normative (read or curated) speech, and often suffer from train-test contamination due to their derivation from existing datasets. Crucially, there is a significant gap in evaluating how models handle non-normative speech, such as that affected by neuromotor disorders (e.g., ALS, post-stroke dysarthria), which presents challenges in articulation, prosody, and intelligibility. Furthermore, current multilingual benchmarks either exclude Spanish or evaluate it only through narrow, text-derived reading comprehension tasks that do not require genuine audio understanding.

Methodology

The authors introduce ESCUCHA, the first Spanish speech understanding benchmark designed to evaluate LALMs across diverse acoustic conditions and reasoning abilities. The benchmark construction followed a four-stage pipeline:

1. Target & Category Refinement: Defining question types and a multi-label taxonomy based on perceptual and reasoning skills.
2. Audio Selection & Question Authoring: Sourcing 162.9 hours of "in-the-wild" audio from public recordings (primarily YouTube), favoring spontaneous speech. This included specific retrieval strategies for pathological speech (ALS and stroke) based on self-reported diagnoses and metadata.
3. LLM-Assisted Review: Using an LLM (Gemma-4-31B-IT) to flag items for manual verification regarding category assignment, audio necessity, distractor quality, and grammatical correctness.
4. Evaluation: Benchmarking models against human annotators.

Benchmark Composition:

• Scale: 1,000 expert-curated questions paired with audio.
• Duration: Ranges from 6.7 seconds to 85.2 minutes (median 86.2s), with a total of 162.9 hours.
• Taxonomy: Each question is annotated with a multi-label taxonomy comprising 9 perceptual categories (e.g., lexical recognition, speaker identification, paralinguistic emotion) and 10 reasoning categories (e.g., causal reasoning, quantitative reasoning, temporal ordering).
• Task Types:
  • MCQA (900 items): Multiple-choice questions with 2–4 options.
  • Audio Instruction Following (AIF) (100 items): Open-ended tasks where the instruction is delivered via spoken audio, requiring the model to recover and follow the constraint.
  • Comparative (254 items): Questions requiring the model to relate two separate audio clips.
  • Spoken Questions (100 items): Questions where the prompt itself is embedded in the audio.
• Non-Normative Subset: 158 questions (15.8%) feature pathological speech from speakers with ALS (114) or post-stroke conditions (44), annotated for intelligibility levels.

Experimental Setup

The study evaluated a diverse range of systems to distinguish between genuine audio understanding and capabilities recoverable via transcription or text priors:

• End-to-End LALMs: Including Audio-Flamingo-3, Qwen2.5-Omni, Qwen3-Omni-30B-A3B, Voxtral-Mini, Gemma-4-12B-IT, and Gemini-2.5-Flash.
• Cascaded Systems: A pipeline using Whisper-Large-v3 for transcription followed by Qwen3-4B-Instruct-2507.
• Text-Only LLMs: Models (Gemma-4 variants, Qwen3-4B) provided with questions and options but no audio input.
• Baselines: Random guessing, positional baselines (always selecting a specific option), and trained human annotators.

Key Results

• Human-Model Gap: Trained human annotators achieved 90.10% overall accuracy, significantly outperforming the best model, Qwen3-Omni-30B-A3B, which scored 74.40%. Most other audio models clustered between 49% and 53%.
• Pathological Speech Performance: A consistent performance drop was observed for most models when transitioning from normative to pathological speech. For instance, Gemini-2.5-Flash and Gemma-4-12B-IT dropped by 19 and 15 percentage points, respectively. Conversely, Qwen3-Omni-30B-A3B and Voxtral-Mini maintained or slightly improved their scores on pathological items.
• Transcription Sufficiency: The cascaded system (Whisper + LLM) achieved 60.00% overall, outperforming all audio models except Qwen3-Omni-30B-A3B. This suggests that a significant portion of the benchmark relies on lexical understanding recoverable from transcripts.
• AIF Performance: Qwen3-Omni-30B-A3B achieved 88.00% on Audio Instruction Following, surpassing the human score of 79.00% on this specific subset.
• Text-Only Baselines: Text-only models scored near random for Gemma variants (due to refusal to answer without audio) but achieved 41.00% with Qwen3-4B, indicating that question structure and language priors allow for partial solvability without audio.

Significance and Claims

The paper claims ESCUCHA as a critical step forward for three primary reasons:

1. First In-the-Wild Spanish Benchmark: It is the first benchmark to evaluate LALMs on Spanish audio and questions sourced directly from real-world conditions, avoiding the contamination risks of curated datasets.
2. Non-Normative Speech Evaluation: It is the first benchmark to specifically evaluate speech understanding and reasoning over pathological speech (ALS and stroke), highlighting a persistent weakness in current LALMs trained primarily on clean, read audio.
3. Comprehensive Reasoning Assessment: By combining MCQA with comparative, spoken, and instruction-following tasks, it provides a broad evaluation of LALM behavior across diverse conditions, revealing substantial performance gaps relative to human capabilities.

The authors conclude that while current models show promise, particularly in instruction following, they still struggle with the complexity of real-world, non-normative Spanish speech, and that genuine acoustic reasoning remains a distinct challenge beyond what transcription alone can solve.