Technical Summary: AU-Harness

Problem Statement

The rapid advancement of Large Audio Language Models (LALMs) has outpaced the development of robust evaluation infrastructure. Existing evaluation toolkits (e.g., AudioBench, VoiceBench, Kimi-Eval) suffer from three critical limitations that hinder systematic assessment and fair comparison:

1. Inefficiency: Processing pipelines often under-utilize batching and parallelism, creating throughput bottlenecks that make large-scale studies impractical.
2. Lack of Multi-Turn Support: Most frameworks are restricted to static, single-turn interactions, failing to address the complexities of cross-turn context integration and performance dynamics in extended conversations.
3. Non-Standardization: The absence of a unified, scalable framework leads to ad-hoc prompting and non-standardized settings, resulting in incomparable performance metrics across different studies.

These limitations constrain the iterative development of LALMs and prevent researchers from conducting rigorous, reproducible comparisons across diverse model architectures and audio benchmarks.

Methodology

The authors introduce AU-Harness, a unified, efficient, and highly customizable evaluation framework designed to address these gaps. The system architecture consists of three primary components:

1. Config Module: A hierarchical, YAML-based configuration system that decouples tasks, metrics, models, and judges. It allows for flexible definition of evaluation settings, including prompt overrides, metric aggregation, and task-specific preprocessing (e.g., transcription strategies) without requiring custom Python code.
2. Central Request Controller (CRC): A token-based scheduling architecture that manages a global pool of concurrency slots shared across all models and evaluation engines. The CRC ensures that throughput is governed by user-defined global limits rather than model-specific constraints. It employs adaptive retry mechanisms and a layered request synchronization strategy to minimize idle waiting time and manage token budgets efficiently.
3. Concurrent Engines: These modules execute task-specific evaluations in parallel. They support dataset sharding, where evaluation data is partitioned into disjoint subsets distributed across model endpoints in proportion to their concurrency capacity. This enables near-linear throughput scaling across heterogeneous resources.

Key Technical Features:

• Backend Agnosticism: AU-Harness supports vLLM for high-throughput inference, third-party APIs, and custom FastAPI endpoints, provided they expose a standard /v1/chat/completions interface.
• Multi-Turn Dialogue Support: The framework supports synchronous, turn-based evaluation chains where model outputs are recursively appended to the dialogue context. Crucially, it allows dynamic configuration of history representation (original audio, ground-truth text, or model-generated transcription) to accommodate different model architectures.
• Compositional Configuration: New tasks can be instantiated by modifying minimal fields in a base configuration (e.g., changing input modality or transcription strategy), facilitating rapid experimentation.

Key Contributions

1. Efficient Evaluation Engine: The authors propose an engine leveraging vLLM batching and dataset sharding to scale evaluations to multi-node infrastructures. This achieves a speedup of up to 151% in Samples Per Second (SPS) and a 61% reduction in Real-Time Factor (RTF) compared to existing toolkits like LMMS-Eval, even when those toolkits utilize similar inference backends.
2. Unified Configurable Framework: AU-Harness standardizes prompting protocols and metrics across benchmarks, enabling fair and reproducible comparisons. It is the first framework to provide comprehensive support for multi-task parallel execution and configurable evaluation design.
3. Configurable Multi-Turn Evaluation: To the best of the authors' knowledge, AU-Harness is the first unified framework to make multi-turn dialogue evaluation fully configurable. This enables systematic analysis of dialogue behaviors, such as error accumulation and context retention, over extended interactions.

Results and Analysis

The authors evaluated AU-Harness against existing frameworks (AudioBench, Kimi-Eval, VoiceBench, LMMS-Eval, AHELM) and conducted extensive empirical studies on a diverse set of LALMs (ranging from <5B to >30B parameters, including open-source and proprietary models).

Efficiency Benchmarks:

• On the MELD-Emotion, LibriSpeech, and ClothoAQA datasets, AU-Harness consistently outperformed all baselines in both RTF and SPS.
• In "Reality" parallel runtime scenarios (accounting for realistic overheads), AU-Harness demonstrated a 26% relative reduction in RTF and a 34% relative gain in SPS compared to LMMS-Eval, highlighting the importance of deliberate orchestration beyond default pipeline configurations.

Insights Enabled by the Framework:

• Instruction Modality Gap: Using a unified pipeline, the authors analyzed the "Qwen3-Omni-30B" model and found that instruction-following performance degrades significantly when moving from text to audio, even with high-quality ASR (Whisper-v3). However, mathematical reasoning tasks showed different sensitivity patterns, suggesting that the "cascaded intelligence bottleneck" is task-dependent.
• Multi-Turn Dynamics: Evaluations on the SpokenWOZ dataset revealed that Joint Goal Accuracy (JGA) declines sharply as dialogue turns increase. The framework allowed the authors to isolate the cause: performance collapse is driven primarily by the accumulation of unfilled slot errors rather than a failure in per-turn understanding. Furthermore, an "audio-text perception gap" was identified, where audio inputs performed worse than text-only baselines even when controlling for error accumulation.

Significance and Claims

The paper claims that AU-Harness provides a standardized, efficient, and highly customizable foundation for the LALM community. Its significance lies in:

• Scalability: Enabling large-scale evaluations that were previously considered impractical due to throughput constraints.
• Reproducibility: Eliminating fragmented, toolkit-specific workarounds through a unified configuration system.
• Analytical Depth: Unlocking in-depth analyses (e.g., multi-turn dynamics, modality gaps) that require a unified foundation to conduct systematically.

The authors position AU-Harness not merely as a benchmark suite, but as an extensible infrastructure designed to support the rapid evolution of LALMs and audio benchmarks. They acknowledge limitations, including dependency on mature backends (like vLLM) for maximum efficiency and a current skew toward English-language datasets, but emphasize that the framework is designed to lower barriers to entry for systematic research and practical deployment.