Technical Summary: PhaseCoder

Problem Statement

Current multimodal Large Language Models (LLMs) typically process audio as a mono stream, rendering them "audio-blind" to the rich spatial information inherent in real-world sound. While spatial awareness is critical for embodied AI (e.g., robotics, next-generation assistants), existing spatial audio models are constrained by fixed microphone geometries. This necessitates training separate encoders for every new device configuration, preventing the generalization advantages of LLMs from being applied to spatial audio. Conversely, microphone-agnostic representations like Ambisonics require specialized, bulky hardware not found in consumer devices. There is a lack of a universal spatial audio encoder that can operate on raw multichannel audio from arbitrary microphone arrays and integrate seamlessly with LLMs for complex reasoning tasks.

Methodology

The authors propose PhaseCoder, a transformer-only spatial audio encoder designed to be inherently agnostic to microphone geometry. The system operates in two primary stages:

1. PhaseCoder Encoder Architecture

• Input Processing: The model accepts raw multichannel audio and the Cartesian coordinates of each microphone in the array. It computes the Short-Time Fourier Transform (STFT) on 250 ms windows (33 frames) to extract magnitude and phase information.
• Embedding Generation: For each microphone and time frame, a patch embedding is created. These are flattened into a sequence.
• Positional Embeddings: To handle variable microphone counts and geometries, three types of positional embeddings are summed:
  1. Sequential: Standard 1D sinusoidal embedding for the flattened sequence position.
  2. Frame: Sinusoidal embedding based on the time frame index.
  3. Microphone Positional: Derived from the microphone coordinates (converted to spherical coordinates relative to the array centroid) using phase modulation, adapted from GI-DOAEnet.
• Backbone: A Transformer encoder (5 blocks, 6M parameters) processes the sequence. A learnable [CLS] token aggregates the spatial information.
• Output Heads: The [CLS] token is passed through an MLP to produce a "spatial audio token" (soft embedding) for LLM integration. Additionally, three separate classification heads predict discrete bins for azimuth, elevation, and distance.

2. Training Strategy

• Supervised Training: The encoder is trained on purely synthetic data generated using the LibriSpeech dataset convolved with simulated Room Impulse Responses (RIRs). The dataset features random microphone geometries (3–8 mics), varying room dimensions, and noise augmentation.
• Curriculum Learning: A two-stage strategy is employed:
  1. Training on clean, noise-free speech.
  2. Fine-tuning with distractor sounds and varying Signal-to-Noise Ratios (SNR).
• LLM Integration: The authors fine-tune the Gemma 3n (4B parameter) LLM. A trainable projection layer maps PhaseCoder's 256-dimensional spatial tokens to the LLM's 2048-dimensional embedding space. These spatial tokens are prepended to the standard mono audio token sequence, framed by special tokens ([BSA], [ESA]).
• LLM Fine-Tuning Tasks: The LLM is trained on four tasks using a curriculum approach:
  1. Source Localization: Outputting coordinates for detected speakers.
  2. Spatial Reasoning: Answering binary (Yes/No) questions about spatial relationships.
  3. Spatial Transcription: Transcribing speech while providing location data.
  4. Targeted Transcription: Transcribing only the speech from a specific spatial direction (e.g., "the speaker on the left").

Key Contributions

1. PhaseCoder Encoder: A novel, transformer-only encoder that achieves state-of-the-art (SOTA) performance in microphone-invariant localization. It processes raw audio and coordinates without requiring device-specific beamformers or hardware.
2. Spatial Audio Tokens: The demonstration that the encoder produces robust, geometry-independent representations ("spatial audio tokens") suitable for downstream LLM tasks.
3. LLM Integration: The first demonstration of an LLM (Gemma 3n) being fine-tuned to reason over these spatial tokens, enabling complex tasks like targeted transcription and spatial reasoning from arbitrary microphone arrays.

Results

• Localization Benchmarks: On real-world datasets (RSL2019 and LOCATA), PhaseCoder-6M outperforms the SOTA microphone-invariant model GI-DOAEnet on the challenging LOCATA dataset (86.96% Acc@10 vs. 82.48%). While GI-DOAEnet performs slightly better on RSL2019 (attributed to PhaseCoder's 10-degree classification resolution vs. GI-DOAEnet's 1-degree regression), PhaseCoder provides a richer multi-task representation (distance and elevation).
• Microphone Scaling: Performance improves significantly as the number of microphones increases from 3 to 8, with azimuth Mean Absolute Error (MAE) dropping from 80.24° to 3.03°.
• LLM Performance: Fine-tuning Gemma with spatial tokens yields substantial improvements over baselines (including zero-shot Gemma and Gemini 3 Flash) in spatial reasoning (improving from ~48% to ~76% accuracy on Yes/No questions) and targeted transcription. The spatial tokens act as effective "diarization prompts," reducing Word Error Rate (WER) in multi-speaker scenarios.
• Efficiency: PhaseCoder demonstrates superior memory efficiency and inference speed compared to hybrid models like GI-DOAEnet and Spatial-AST, attributed to its pure transformer architecture.

Significance and Claims

The paper claims to bridge the gap between universal spatial audio encoding and LLM understanding. By enabling LLMs to process "Spatial Audio Tokens" from arbitrary microphone arrays, PhaseCoder allows embodied AI agents to perceive and reason about their acoustic environment without hardware-specific constraints. The authors assert that this approach opens new avenues for:

• Accessibility: Enhancing assistive devices for the hearing-impaired by isolating specific speakers in noisy environments (the "cocktail party effect").
• Hardware Independence: Enabling universal spatial understanding across diverse devices, from legacy phones to custom robotics, without retraining encoders.
• Embodied Intelligence: Providing AI assistants with the ability to localize and reason about physical surroundings through sound, a critical step toward natural human-robot interaction.

The authors note limitations, including the assumption of omnidirectional microphones (ignoring device baffling effects) and a current focus on single-dominant speech sources rather than full Sound Event Localization and Detection (SELD). They suggest future work could integrate visual depth estimation to resolve distance ambiguities.