Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs

1. Problem Statement

Recent Audio Large Language Models (AudioLLMs) exhibit a significant performance inversion: while they excel at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception.

• The Gap: Models often achieve ~70% accuracy on reasoning benchmarks but drop to ~40% on fundamental perception tasks (e.g., identifying speaker emotion, age, accent, or non-linguistic sounds like door slams).
• Root Cause: The authors attribute this to ASR-centric training. Existing models rely heavily on Automatic Speech Recognition (ASR) as the primary training signal. To recover canonical text, ASR objectives implicitly normalize away paralinguistic cues (prosody, emotion, timbre) and acoustic context, treating them as noise. This creates a systematic de-emphasis on "how" something is said and "what else" is heard, leading to perceptual blind spots.

2. Methodology: Unified Audio Schema (UAS)

To address this, the authors propose the Unified Audio Schema (UAS), a holistic, structured supervision framework that decomposes audio information into three explicit components within a unified JSON format.

A. Schema Definition

The schema is grounded in the semiotic framework of speech signals (Laver, 1994) and organizes audio into:

1. Transcription: Verbatim speech content (preserving 100% linguistic fidelity equivalent to ASR).
2. Paralinguistics: Structured attributes describing how speech is produced, including:
   • Speaker Traits: Age, Gender, Accent.
   • Emotional State: Emotion (e.g., Neutral, Happy).
   • Prosody: Pace, intonation, rhythm.
   • Timbre: Voice quality (e.g., muffled, resonant).
3. Non-linguistic Events: Acoustic context beyond speech, divided into:
   • Discrete Events: Instantaneous sounds (e.g., door slam, laughter).
   • Continuous Events: Background conditions (e.g., ambient noise, music).

Key Design Features:

• Explicit Disentanglement: Forces the model to distinguish between semantic content and acoustic attributes, preventing feature conflation.
• Syntactic Invariance: Unlike unstructured captions (which have high entropy/variability), the JSON format provides a low-entropy, consistent supervision target.
• Programmatic Accessibility: The rigorous JSON format bridges probabilistic LLM outputs with deterministic software interfaces.

B. Scalable Data Generation Pipeline

Since manual annotation of UAS is expensive, the authors developed an automated 3-stage pipeline to synthesize UAS data from existing ASR corpora:

1. Acoustic Caption Generation: An acoustic captioner model generates rich descriptions of paralinguistics and environmental sounds from raw audio.
2. Structured Schema Synthesis: An LLM merges the ground-truth transcription with the generated captions to produce the structured JSON, ensuring semantic fidelity and acoustic completeness.
3. Quality Validation: Automated rules enforce ontology constraints (closed-set vocabularies), verify transcription integrity, check logical consistency (e.g., gender vs. timbre), and filter hallucinations.
   • Human Verification: A manual audit of 400 samples showed >95% accuracy for most attributes.

C. UAS-QA Dataset

To train the model to apply this knowledge, the authors synthesized a UAS-QA dataset containing:

• Direct QA: Specific queries about UAS fields (e.g., "What is the speaker's emotion?").
• Multiple Choice: Questions with options derived from the UAS vocabulary.
• Yes/No Questions: Binary verification of acoustic attributes.

D. Model Architecture (UAS-Audio)

The authors implemented UAS-Audio, validating the approach on both Continuous (flow-matching based) and Discrete (token-based) architectures. The training pipeline involves:

1. Discrete Token Alignment: Aligning text and discrete audio codes for generation.
2. Audio-LLM Adaptation: Training exclusively on UAS data to prevent the formation of ASR-centric representations.
3. Full Instruction Tuning: A mixture of foundational audio tasks, UAS annotation generation, and UAS-QA.
4. GRPO (Group Relative Policy Optimization): Used to further refine reasoning and generation capabilities.

3. Key Contributions

1. Diagnosis: Identified that the perceptual weakness of current AudioLLMs stems from ASR-centric supervision, which systematically suppresses paralinguistic and non-linguistic information.
2. Innovation: Proposed the Unified Audio Schema (UAS), a structured representation that explicitly decomposes audio into transcription, paralinguistics, and non-linguistic events, coupled with a scalable pipeline to synthesize this data.
3. Validation: Demonstrated that restructuring supervision yields consistent gains across both continuous and discrete architectures, achieving state-of-the-art results without sacrificing reasoning capabilities.

4. Experimental Results

The model was evaluated on MMSU (Perception/Reasoning), MMAR (Deep Reasoning), and MMAU (Multimodal Understanding).

• Perception vs. Reasoning Trade-off:
  • On the MMSU benchmark, UAS-Audio achieved 55.7% accuracy on perception tasks, a +10.9% absolute improvement over the best baseline (Kimi-Audio at 44.8%).
  • Crucially, reasoning accuracy remained robust at 77.4% (only 0.2% below the top baseline), proving that perception enhancement does not degrade reasoning.
• Generalization:
  • Achieved the highest overall score on MMAR (60.1%) and competitive results on MMAU (69.4%), showing balanced performance across speech, sound, and music domains.
  • Applied to discrete architectures (UAS-Audio-D), it nearly doubled the baseline performance (24.4% → 44.2%).
• Generation Quality:
  • On TTS benchmarks (Seed-TTS), UAS-Audio achieved the best average Word Error Rate (WER) of 1.6, outperforming Qwen2.5-Omni (1.9) and Step-Audio2-mini (2.7).
• Robustness:
  • Generating full UAS JSONs did not significantly degrade transcription accuracy (WER difference was negligible, ~0.1), confirming the model can handle complex structured outputs without losing ASR precision.

5. Significance

This work challenges the prevailing paradigm that AudioLLMs must choose between high-fidelity reasoning and fine-grained perception. By shifting from unstructured, ASR-centric supervision to a structured, multi-dimensional schema, the authors demonstrate that:

• Perception and Reasoning are independent factors: Enhancing one does not require sacrificing the other.
• Structured Supervision is Critical: The format of the training signal (JSON vs. unstructured text) significantly impacts the model's ability to learn acoustic nuances.
• Scalability: High-quality, perception-aware supervision can be synthesized at scale, making this approach viable for large-scale model training.

The paper concludes that structural richness in supervision is a key driver for evolving AudioLLMs beyond simple scaling, enabling them to achieve "holistic audio understanding."