RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark

Technical Summary of RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark

Problem Statement
Current evaluations of Large Audio-Language Models (LALMs) are predominantly task- or domain-centric, focusing on end-performance metrics (e.g., transcription accuracy, classification scores) while neglecting the underlying auditory cognitive behaviors. This approach creates a disconnect between how human auditory cognition is understood—encompassing perception, reasoning, memory, and processing efficiency—and how it is evaluated in artificial systems. Existing benchmarks lack a cognitively grounded theory, failing to capture real-world auditory scenarios such as maintaining working memory across spoken instructions or distinguishing subtle acoustic cues. Consequently, there is no principled framework to diagnose specific capability gaps, interpret model behavior, or align LALMs with human auditory objectives.

Methodology
To address these gaps, the authors introduce RAIL, a human-centric benchmark grounded in the Cattell–Horn–Carroll (CHC) theory of cognitive abilities. The methodology involves a four-stage curation pipeline:

1. Cognitive Framework Selection: The authors selected the CHC framework over alternatives (e.g., Bloom’s Taxonomy, Multiple Intelligences) due to its data-driven, measurable, and hierarchical structure.
2. Task Formulation: Tasks were designed to translate CHC broad abilities into auditory evaluations, adhering to two principles: auditory dependence (tasks rely solely on audio cues) and capability independence (each ability is assessed without prerequisite dependencies on others). The benchmark covers five core capabilities and 32 fine-grained subcapabilities:
   • Auditory Processing (Ga): Fine-grained perception (e.g., phonetic coding, rhythm judgment, sound localization).
   • Fluid Reasoning (Gf): Logic and inference from audio (e.g., induction, sequential reasoning, quantitative reasoning).
   • Memory (Gsm/Glr): Retention and manipulation of auditory information (e.g., working memory, associative memory, memory for sound patterns).
   • Processing Efficiency (Gs/Gt): Speed and accuracy of simple cognitive tasks (e.g., reaction time, mental comparison speed).
   • Acquired Knowledge (Gc/Gkn): Accessing stored knowledge via auditory cues (e.g., mechanical knowledge, general information).
3. Dataset Curation: The dataset comprises 5,306 speech samples (30.6 hours total) spanning 32 tasks. Data was sourced from existing corpora or synthesized, with QA pairs generated via rule-based pipelines or LLM-assisted generation, followed by human verification.
4. Quality Control: Cognitive experts reviewed instances to ensure alignment with intended cognitive abilities, answerability from audio, and unambiguous answers.

Evaluation Protocol
The authors evaluated 26 state-of-the-art LALMs (167M–33.5B parameters, including open-source and closed-source models) against a human baseline (24 participants).

• Metrics: For most tasks, strict rule-based accuracy (ACC) and a relaxed LLM-as-Judge evaluation (semantic equivalence) were used.
• Efficiency Metric: To measure processing efficiency, the authors introduced B-AUC (Budgeted Area Under the Curve), which captures the trade-off between accuracy and reasoning token length (a proxy for computational effort). This metric prioritizes early accuracy over delayed correctness.

Key Results

1. Capability Imbalance: LALMs exhibit strong performance in Knowledge (mean LLM-Judge score: 56.21) and Memory (55.05), largely inherited from text-based pretraining. In contrast, Auditory Processing (43.83) and Processing Efficiency remain weak points.
2. Human vs. Model Comparison: While six LALMs exceed human performance in overall aggregate scores, humans outperform all models in Auditory Processing and Processing Efficiency. Models struggle with non-verbal acoustic cues and generate unnecessarily long reasoning traces for simple tasks, indicating inefficient computation.
3. Subcapability Analysis:
   • Auditory Perception: Models perform significantly better on language-supported tasks (e.g., speech discrimination) than on perceptual-only tasks (e.g., phonetic coding, absolute pitch), revealing a reliance on linguistic shortcuts rather than true acoustic perception.
   • Reasoning: General Sequential Reasoning is the weakest reasoning capability, particularly for open-source models. This suggests a mismatch between Chain-of-Thought (CoT) post-training, which promotes plausible text, and the stateful, stepwise updates required for auditory reasoning.
   • Memory: Models excel in speech-based short-term memory but fail in Memory for Sound Patterns (non-speech audio), where no model exceeds 60% accuracy. Free-recall memory shows a sharp divide, with some models achieving near-perfect token-level recall while others fail completely.
   • Efficiency: Model size does not correlate with processing efficiency. Efficiency is driven more by generation behavior and training objectives than parameter count.
   • Knowledge: Mechanical Knowledge (identifying machine functions from sound) is an outlier, with performance near chance levels even for frontier models, unlike other knowledge tasks which benefit from semantic priors.

Significance and Contributions
The paper claims to establish a new framework for human-aligned evaluation of auditory intelligence. By grounding evaluation in the CHC theory, RAIL moves beyond task-centric metrics to provide a structured, interpretable assessment of model capabilities. The findings highlight that prevailing training paradigms (text-based pretraining, task supervision, CoT) do not sufficiently support generalizable auditory understanding. The authors argue that future progress requires a shift in both evaluation and training toward developing generalizable, sensory-grounded auditory cognitive capabilities, rather than optimizing for specific tasks or domains.