How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation

1. Problem Statement

Large Language Models (LLMs) serve as the cognitive backbones for Large Audio Language Models (LALMs). While LALMs are typically fine-tuned with audio encoders to process sound, the extent to which the underlying text-only LLMs possess inherent auditory knowledge (acquired solely through text pre-training) remains unclear.

• The Gap: Existing research often selects LLM backbones (e.g., Llama, Qwen) based on general language performance or architectural convenience, without evaluating their specific auditory reasoning capabilities.
• The Question: How much auditory knowledge is encoded in text-only LLMs, and does this knowledge correlate with and influence performance in downstream audio tasks (both cascade and end-to-end)?
• The Limitation of Current Benchmarks: Existing audio benchmarks conflate the quality of the audio encoder, the training data, and the LLM's internal knowledge, making it difficult to isolate the specific contribution of the LLM backbone.

2. Methodology

The authors propose a holistic evaluation framework consisting of three complementary settings to isolate the LLM backbone as the sole variable:

A. New Benchmark: AKB-2000 (Auditory Knowledge Benchmark)

• Construction: A curated dataset of 2,000 multiple-choice questions covering 6 categories: Music, Sound, Paralinguistic, Phonetic, Audio Quality, and Technical Knowledge.
• Process: Questions were generated using proprietary LLMs based on a two-level taxonomy and verified by human experts with audio backgrounds.
• Purpose: To directly probe the breadth and depth of factual and common-sense auditory knowledge in text-only LLMs.

B. Three Evaluation Paradigms

1. Direct Probing (Text-Only): Evaluating LLMs on AKB-2000 to measure raw auditory knowledge.
2. Cascade Evaluation (Text-Only):
   • Audio samples from existing benchmarks (MMAU, MMAR) are converted into detailed text descriptions using a strong captioner (Gemini-2.5-Pro).
   • Text-only LLMs answer the original audio questions based only on these text descriptions.
   • Goal: To test if LLMs can apply their encoded knowledge to reason over real-world audio scenarios represented as text.
3. Audio-Grounded Evaluation (End-to-End):
   • Each LLM is fine-tuned into an LALM using the DeSTA self-distillation framework.
   • Setup: A frozen audio encoder (Whisper-large-v3) and a modality connector (Q-Former) are paired with the LLM. The LLM generates training targets from text metadata, then learns to map raw audio to these targets.
   • Goal: To determine if inherent text-based knowledge transfers to performance when processing actual audio waveforms.

C. Models Evaluated

• 12 Open-Weight LLMs: Spanning 4 families (Qwen, Llama, Phi, OLMo) across various scales (4B–14B) and generations.
• 5 Proprietary Models: GPT-5, GPT-4o, Gemini-2.5-Pro, Gemini-2.0-Flash, and Claude-Sonnet-4.5 (used as strong baselines).

3. Key Contributions

1. Systematic Evaluation: The first comprehensive study isolating the impact of the LLM backbone on audio system performance across text-only, cascade, and end-to-end settings.
2. AKB-2000: Introduction of a new, high-quality benchmark specifically designed to test auditory knowledge across diverse domains (6 categories, 48 subcategories).
3. Empirical Correlation: Demonstration that text-only performance is a strong predictor of audio-grounded performance, offering a lightweight proxy for model selection.
4. Identification of Blind Spots: Discovery of a systematic failure in phonological reasoning (e.g., rhyming, stress patterns) across all text-only models.
5. Architecture Insights: Evidence that current end-to-end LALMs may be bottlenecked by audio encoders rather than the LLM's reasoning capabilities, as cascade pipelines often match or surpass state-of-the-art end-to-end systems.

4. Key Results

A. Auditory Knowledge Varies by Family

• Family Differences: Auditory knowledge is not uniform. The Qwen family consistently outperforms Llama and OLMo across all settings.
• Scale vs. Knowledge: In the Qwen family, performance does not strictly scale with parameter count (Qwen3-4B and Qwen3-8B perform similarly). However, in the Phi family, performance drops significantly at smaller scales (Phi-4-mini).
• Generational Shifts: Newer models do not always imply better auditory knowledge (e.g., Llama-3.1-8B underperforms Llama-3-8B in some metrics).

B. Strong Correlation Between Text and Audio

• Predictive Power: There is a strong positive correlation ($r \approx 0.71 - 0.82$) between text-only benchmark scores (AKB-2000, Cascade) and audio-grounded performance.
• Impact of Backbone: When fine-tuned with identical training recipes, the choice of backbone alone can result in a >10% absolute performance difference in the final LALM. A strong text-only LLM (e.g., Qwen2.5-7B) can outperform a weaker LLM (Llama-3.1-8B) even when the latter uses 10x more training data.

C. The Phonological Deficit

• Systematic Failure: All models struggle significantly with Phonetic tasks (e.g., identifying stress patterns, rhyming, homophones), scoring 10–15% lower than other categories.
• Cause: Text-only pre-training learns semantic token correlations but lacks exposure to the acoustic realization of language, making it impossible to ground representations in "how words sound."

D. Cascade vs. End-to-End

• Cascade Superiority: A simple cascade pipeline (Strong Captioner + Strong LLM) often matches or surpasses state-of-the-art end-to-end LALMs (e.g., Qwen3-8B in cascade > Audio Flamingo 3 in end-to-end).
• Bottleneck: This suggests that current end-to-end systems are limited by audio-text alignment and the audio encoder's ability to preserve fine-grained details, rather than the LLM's reasoning capability. The LLM's inherent knowledge is underutilized in current end-to-end architectures.

5. Significance and Implications

• Design Decision: Selecting the LLM backbone is a first-order design decision for LALMs. Researchers should prioritize backbones with high inherent auditory knowledge (like Qwen) over generic language models.
• Efficient Selection: Text-only benchmarks (AKB-2000) and cascade evaluations serve as reliable, low-cost proxies for selecting backbones before expensive multimodal training.
• Future Directions:
  • Phonology: Future models require phonology-aware training (e.g., pronunciation lexicons) to bridge the gap between text and sound.
  • Data Coverage: The weak correlation in Sound/Music domains suggests that current training data is insufficient; targeted data collection is needed to unlock the full potential of strong backbones in these domains.
  • Architecture: The success of cascade pipelines implies a need to improve audio encoders and alignment mechanisms in end-to-end models to better leverage the LLM's reasoning power.

In conclusion, the paper establishes that auditory knowledge is an intrinsic property of the LLM backbone that fundamentally shapes audio understanding systems. Ignoring this knowledge leads to suboptimal LALMs, while leveraging it through careful backbone selection and addressing phonological blind spots is crucial for the next generation of audio AI.