Technical Summary: BAT – Learning to Reason about Spatial Sounds with Large Language Models

1. Problem Statement

While Large Language Models (LLMs) have successfully extended to multimodal tasks involving images and monaural audio (e.g., speech recognition, audio captioning), they currently lack the capability to process and reason about spatial audio in complex, 3-D environments. Humans possess binaural hearing, allowing them to not only identify sound events but also infer their direction, distance, and the spatial relationships between multiple concurrent sources. Existing models for Sound Event Localization and Detection (SELD) are often limited to specific tasks (e.g., direction-of-arrival estimation only) or rely on monaural inputs that lack spatial cues. Furthermore, there is a significant scarcity of large-scale, "in-the-wild" spatial audio datasets with ground-truth labels for distance and direction, hindering the development of models capable of spatial reasoning.

2. Methodology

2.1. Data Synthesis: SPATIALSOUNDQA

To address the lack of training data, the authors synthesized a large-scale binaural audio dataset and a corresponding question-answering (QA) benchmark:

• Audio Generation: Using SoundSpaces 2.0 as a simulator and AudioSet clips as sound sources, the authors generated binaural audio in diverse, reverberant 3-D environments (based on Matterport3D meshes). They filtered AudioSet labels to ensure 355 event types were identifiable solely by audio, excluding those requiring visual cues.
• SPATIALSOUNDQA Dataset: A diverse QA dataset containing 872K samples structured as (audio, question, answer). The tasks are categorized into five types:
  • Perception: Sound Event Detection (Type A & C) and Direction/Distance Estimation (Type B & D) for single and dual sound sources.
  • Reasoning (Type E): Complex questions requiring the model to infer spatial relationships between concurrent sound sources (e.g., "Is the dog to the left of the music?").

2.2. Model Architecture: BAT

The proposed model, BAT, integrates a novel spatial audio encoder with a Large Language Model.

• Spatial Audio Encoder (SPATIAL-AST):

  • Input: Binaural audio is processed via Short-Time Fourier Transform (STFT) to generate Mel-Spectrograms and Interaural Phase Differences (IPD). The IPD is transformed using cosine and sine functions to handle phase wraparound.
  • Backbone: The features are fused via a 3x3 convolution and passed through a Patch-Embed CNN. The architecture employs a Transformer encoder (12 layers) with three learnable [CLS] tokens specifically designed to extract category, distance, and direction information.
  • Pre-training: SPATIAL-AST is pre-trained on three tasks: Sound Event Detection, Distance Prediction, and Direction of Arrival (DoA) estimation using a multi-task loss function.

• Integration with LLM:

  • The SPATIAL-AST encoder is fused with LLaMA-2 (7B) via a projection module and LLaMA-adapter v2.
  • The audio tokens from SPATIAL-AST are projected into the LLaMA-2 text embedding space.
  • Training Curriculum: A "perception-to-reasoning" curriculum is employed:
    1. Stage I: Single-source perception (Types A & B).
    2. Stage II: Multi-source perception with implicit separation (Types C & D).
    3. Stage III: Full spatial reasoning (Type E).

3. Key Contributions

1. SPATIALSOUNDQA: The first spatial audio-based question-answering dataset, offering tasks ranging from basic perception to complex spatial reasoning in 3-D environments.
2. SPATIAL-AST: A novel binaural spatial audio encoder that jointly performs sound event detection, spatial localization, and distance estimation, achieving strong performance across all three tasks.
3. BAT: The first spatial audio-based LLM capable of reasoning about multiple sound sources in a 3-D environment, demonstrating that LLMs can effectively interpret complex spatial audio cues.

4. Experimental Results

• SPATIAL-AST Performance:

  • Achieved a Mean Average Precision (mAP) of 50.03% for sound event detection.
  • Achieved a Mean Angular Error (MAE) of 17.94° for DoA estimation.
  • Achieved a Distance Error Rate (DER) of 32.54% (within 0.5m of ground truth).
  • The use of binaural data with IPD and a two-stage training curriculum significantly outperformed monaural baselines and joint-training approaches for spatial tasks.

• BAT Performance:

  • On the reasoning tasks (Type E), BAT achieved a 76.89% Binary Accuracy, significantly outperforming random baselines (50%) and monaural-only variants (~54.53%).
  • The experiments highlighted the necessity of the multi-stage curriculum; removing multi-source separation tasks (Types C & D) from training caused a drastic drop in reasoning performance, indicating that the model requires intermediate steps to learn source separation before reasoning.

5. Significance and Claims

The paper claims that BAT represents a significant step forward in bridging the gap between LLMs and spatial audio. By successfully integrating a specialized spatial encoder with a powerful LLM, the authors demonstrate that LLMs can be trained to navigate and interpret complex spatial audio environments, a capability previously absent in multimodal models.

The authors position this work as foundational for:

• Spatial Audio Advancements: Enabling applications in virtual reality, gaming, and audio engineering where spatial cues are critical.
• Multimodal AI: Moving towards truly multimodal systems that can process auditory spatial information alongside other modalities.
• Embodied AI: Enhancing the ability of robots and autonomous agents to navigate and interact with their environments using spatial sound.

The authors acknowledge limitations, including the current restriction to binaural audio (rather than ambisonics), the maximum of two sound sources, and the "sim-to-real" gap, noting that future work will explore these areas.