JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments

JAEGER is a novel framework that advances audio-visual large language models into 3D space by integrating RGB-D observations with multi-channel ambisonics and a learned Neural Intensity Vector representation, enabling robust spatial grounding and reasoning through the newly proposed SpatialSceneQA benchmark.

The Gist

Imagine you are trying to find a friend who is shouting your name in a crowded, echoey room.

The Old Way (Current AI):
Most current "smart" computers (Audio-Visual Large Language Models) are like people wearing blindfolds and noise-canceling headphones that only let in a tiny, single-channel whisper. They can see a flat, 2D picture of the room and hear a flat, mono sound. Because they lack depth perception and true 3D hearing, they are terrible at figuring out where the sound is coming from or which object in the room is making it. They might guess, "The voice is coming from the left," but they can't tell you if the speaker is on a table, on the floor, or behind a wall.

The New Way (JAEGER):
This paper introduces JAEGER, a new AI system designed to be a "super-sensor" for the real, 3D world. Think of JAEGER as a detective with 3D X-ray vision and surround-sound hearing.

Here is how it works, broken down into simple concepts:

1. The "3D Glasses" (RGB-D Vision)

Instead of just looking at a flat photo, JAEGER wears special glasses that see depth. It doesn't just see a "chair"; it sees a chair that is two meters away, one meter high, and has a specific volume. This allows it to understand the physical shape of the room, not just the picture on a screen.

2. The "Super-Ears" (FOA Audio)

Instead of listening with one ear (mono), JAEGER listens with a 4-channel surround-sound system (First-Order Ambisonics). Imagine being in the center of a room with microphones facing North, South, East, and West simultaneously. This lets the AI hear not just what is being said, but exactly where the sound waves are hitting from.

3. The "Magic Brain" (Neural Intensity Vector)

This is the paper's coolest invention.

• The Problem: In a noisy room with echoes or two people talking at once, traditional math (like a standard compass) gets confused. It's like trying to find a needle in a haystack while the haystack is shaking.
• The Solution: JAEGER uses a Neural Intensity Vector. Think of this as a "smart compass" that the AI learns to build itself. Instead of using a rigid, pre-made rulebook, it learns to ignore the confusing echoes and focus on the true direction of the voice, even when two people are shouting over each other. It's like a detective who can tune out the background noise to focus on the specific voice they are looking for.

4. The "Training Ground" (SpatialSceneQA)

To teach JAEGER these skills, the researchers didn't just use real-world data (which is hard to get). They built a massive, hyper-realistic video game simulation.

• They created 61,000 different 3D rooms.
• They placed virtual speakers in them.
• They recorded the sound and the 3D view perfectly synchronized.
• They asked the AI millions of questions like, "Where is the male voice coming from?" or "Point to the speaker on the left."

Why Does This Matter?

Before JAEGER, AI was like a person trying to navigate a 3D world using only a 2D map and a single ear. It could describe a picture, but it couldn't interact with the physical world.

JAEGER proves that for robots or AI assistants to truly understand our world—whether it's helping a blind person navigate a room, finding a lost pet by its bark, or a robot vacuum avoiding a crying baby—it must understand 3D space and 3D sound simultaneously.

In a nutshell: JAEGER is the first AI that can truly "look" and "listen" in 3D, allowing it to find objects and people in a room with the same spatial awareness a human has, even in noisy, echoey environments.

Technical Summary

1. Problem Statement

Current Audio-Visual Large Language Models (AV-LLMs) are fundamentally limited by their reliance on 2D perception (RGB video and monaural audio). This design creates a dimensionality mismatch that prevents reliable source localization and spatial reasoning in complex 3D physical environments.

• Limitations of Existing Approaches:
  • Visual: Most models lack explicit depth understanding, relying on 2D projections.
  • Audio: Many rely on binaural cues (tied to specific HRTFs and recording geometry) or monaural inputs, which lack directional information.
  • Integration: Existing multimodal systems often use modular, cascaded pipelines (e.g., traditional signal processing for localization followed by an LLM), which hinders end-to-end learning and prevents fully integrated spatial reasoning.
• Goal: To bridge the gap between 2D-centric AV-LLMs and 3D physical reality by enabling joint spatial grounding and reasoning using explicit 3D geometry and multi-channel spatial audio.

2. Methodology: The JAEGER Framework

JAEGER is an end-to-end framework that extends existing AV-LLMs (initialized from Qwen2.5-Omni) to handle 3D settings. It integrates RGB-D observations and First-Order Ambisonics (FOA) audio.

A. Neural Intensity Vector (Neural IV)

A core innovation is the Neural IV, a learnable spatial audio representation designed to replace traditional STFT-based intensity features.

• Problem with Classical IV: Classical methods rely on fixed signal processing (STFT) which struggles in reverberant environments or with overlapping sound sources.
• Neural IV Architecture:
  • Uses a learnable 1D-CNN backbone (inspired by data2vec) to extract latent features directly from raw FOA waveforms.
  • Computes an interaction between the omnidirectional channel ($W$) and directional channels ($X, Y, Z$) via element-wise multiplication in the latent space.
  • Projects these interaction features through an MLP to produce a robust spatial embedding ($v_{NIV}$).
  • Benefit: Encodes robust directional cues that generalize better across overlapping sources and reverberation compared to classical methods.

B. 3D-Aware Visual Encoding

To enable metric 3D grounding, the visual stream is augmented with depth information:

• Point Cloud Reconstruction: Converts RGB-D inputs into a metric point cloud using camera intrinsics.
• 3D Positional Encoding: Downsampled metric coordinates are mapped to sinusoidal positional encodings and injected into the visual feature embeddings. This allows the model to explicitly ground visual tokens in 3D space.

C. Model Architecture

• Dual-Path Audio Stream: Extracts semantic content from the omnidirectional FOA channel while capturing spatial cues via either Classical IV or Neural IV.
• Unified LLM: All modalities (RGB, Depth, Audio Semantic, Audio Spatial) are aligned via MLP adapters and fed into the LLM backbone.
• Training: Efficiently adapted using LoRA (Low-Rank Adaptation) for end-to-end fine-tuning.

3. Key Contributions

A. The SPATIALSCENEQA Dataset

The authors constructed SpatialSceneQA, a large-scale benchmark containing 61k instruction-tuning samples generated via simulation (Habitat-Sim and SoundSpaces 2.0).

• Data Composition: Synchronized RGB-D frames, 4-channel FOA audio, and precise 3D object-level annotations.
• Task Diversity:
  • Perception: Single-source and overlapping-source Direction-of-Arrival (DoA) estimation; 3D visual grounding (bounding box prediction).
  • Reasoning: Multi-speaker matching (identifying which visual speaker corresponds to a specific audio source, even with overlapping voices).
• Significance: It is the first spatial audio-visual benchmark providing degree-level azimuth/elevation supervision and dense 3D annotations for overlapping source scenarios.

B. The JAEGER Framework

The first end-to-end AV-LLM framework that natively models RGB-D geometry and multi-channel FOA for joint 3D grounding and reasoning, eliminating the need for modular signal processing pipelines.

C. Neural Intensity Vector

A novel, data-driven representation for spatial audio that outperforms classical physics-based intensity vectors in complex acoustic scenarios.

4. Experimental Results

Extensive experiments demonstrate that JAEGER significantly outperforms 2D-centric baselines and specialized models.

• Audio Localization (DoA):
  • Single Source: Achieved a Median Angular Error (MAE) of 2.21° (Neural IV), comparable to specialized baselines (BAT: 2.16°).
  • Overlapping Sources: Significantly outperformed baselines, reducing MAE from 19.09° (BAT) to 13.13° (JAEGER/Neural IV).
• 3D Visual Grounding:
  • Achieved a 3D Intersection over Union (IoU) of 0.32 and a median localization error of 0.16 meters.
  • Ablation studies confirmed that explicit depth encoding is critical, improving 3D IoU from 0.29 to 0.32.
• Joint Audio-Visual Reasoning:
  • Single Speaker: 99.5% accuracy.
  • Overlapping Speakers: 99.2% accuracy.
  • Baseline Failure: Standard 2D AV-LLMs (even after fine-tuning) failed to perform these tasks reliably, often collapsing to near-random performance (~45%) when spatial cues were removed.

5. Significance and Impact

• Paradigm Shift: The paper argues that explicit 3D modeling is not optional but indispensable for AI agents operating in physical environments. 2D-centric models fundamentally lack the capacity for reliable spatial disambiguation.
• Robustness: The Neural IV approach demonstrates that learned representations can outperform traditional signal processing in challenging, real-world-like acoustic conditions (reverberation, overlap).
• Foundation for Embodied AI: By providing a unified framework for 3D perception and reasoning, JAEGER advances the development of embodied agents capable of holistic interaction with physical spaces, moving beyond simple captioning to precise localization and causal reasoning.
• Resource Availability: The authors commit to releasing the code, pre-trained models, and the SpatialSceneQA dataset, fostering further research in 3D audio-visual understanding.