🤖 AI

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

This paper introduces ST-OmniQA, a large-scale spatio-temporal audio-visual benchmark with 40K panoramic videos and Ambisonics audio, and proposes ST-Omni-R1, a model that significantly outperforms baselines by integrating directional audio cues with visual context through progressive curriculum and reinforcement learning to reason about sound sources' identity, location, and motion.

Original authors: Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, Jiaying Wu, Di Zhang, Zihan Ma, Guodong Li, Zhou Yang, Yu Xiang, Yifei Zheng, Minnan Luo

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, Jiaying Wu, Di Zhang, Zihan Ma, Guodong Li, Zhou Yang, Yu Xiang, Yifei Zheng, Minnan Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking down a busy street with your eyes closed. You hear a dog barking, a car honking, and someone laughing. Your brain doesn't just hear "noise"; it instantly figures out what is making the sound, where it is coming from, and where it is going. It knows the dog is running left, the car is approaching from the right, and the laughter is coming from a person you can't see yet. This ability to mix hearing and seeing to track moving things is a superpower humans have, but it's a nightmare for computers.

For a long time, computer programs that understand sound (Audio-Language Models) were like people with their eyes closed: they could tell you a dog was barking, but they couldn't tell you if the dog was moving toward you or running away. On the other hand, programs that understand video (Vision-Language Models) were like people with their ears plugged: they could see a dog, but they didn't know if that specific dog was the one barking or just standing there silently. The big challenge in science right now is building a robot brain that can do both at the same time—listening to the sound, watching the video, and figuring out exactly how the sound source is moving through space and time. This paper tackles that exact problem, trying to teach machines to "listen, see, and track" simultaneously.

The Problem: The "Blind" and the "Deaf" Robots

The authors of this paper noticed a gap in how smart computers currently think. Some models are great at understanding audio, but they treat a whole video clip as one big, blurry sound event. They miss the details of where a sound is coming from or how it moves. Other models are great at looking at videos, but they rely on simple audio that doesn't have enough "spatial" information (like direction or distance) to know where things are.

Think of it like trying to find a friend in a crowded park. If you only have a description of their voice (audio), you might know they are there, but you can't point to them. If you only have a photo (video), you can see them, but you don't know if they are the one talking. To really understand the scene, you need to match the voice to the person and track their movement as they walk behind a tree or around a corner. Existing models struggle with this "binding" problem—they can't easily connect a moving sound to a moving object in a video.

The Solution: A New Playground and a New Brain

To fix this, the researchers created two things: a giant new test called ST-OmniQA and a new AI model called ST-Omni-R1.

1. The Test: ST-OmniQA
Imagine a massive video game where the camera spins 360 degrees (panoramic video) and the sound is recorded with special microphones that know exactly where sounds are coming from (called First-Order Ambisonics or FOA). The researchers built a dataset with 40,000 of these video clips and 400,000 questions about them.

The questions get harder in four levels, like a video game:

Level A (The Basics): "What sound is this?" or "How far away is it?"
Level B (The Crowd): "There are two dogs barking; which one is on the left?"
Level C (The Chase): "Which sound source is moving toward the person standing there?"
Level D (The Mystery): "A sound was heard between 2 and 3 seconds, but the source wasn't visible yet. It entered through the left door between 3 and 5 seconds. Who was it?"

This test forces the AI to not just guess, but to reason about time, space, and how sound and sight fit together.

2. The Brain: ST-Omni-R1
The researchers built a new AI model to take this test. Instead of just listening to a flat sound file, this model uses a special "translator" (the STA-encoder) that turns the 3D sound data into a list of "trajectory tokens." Think of these tokens as a trail of breadcrumbs that tells the AI exactly how a sound moved over time.

The model learns in stages, like a student going through school:

Stage A: It learns to identify simple sounds and where they are.
Stage B: It learns to handle multiple sounds at once.
Stage C: It learns to compare how different sounds move relative to each other.
Stage D: It learns to tie the sound to the specific object in the video, even if the object disappears behind a wall (occlusion) and then reappears.

To make the model even smarter, they used a technique called "Reasoning-Tree Reinforcement Learning." Imagine the AI is solving a puzzle. Instead of just giving it the answer, the system checks every step of its thinking process. If the AI says, "The dog is on the left," the system checks if the AI correctly identified the dog, correctly saw it was on the left, and correctly linked the bark to the dog. If the steps don't match up, the AI gets a "red pen" correction. This helps the model learn to be consistent and logical.

The Results: A Big Leap Forward

When they put ST-Omni-R1 through the ST-OmniQA test, it did incredibly well.

The best existing models (like the smartest general-purpose AI available) only got about 37.28% of the answers right on average.
ST-Omni-R1, after its special training, achieved 77.83% accuracy.

This isn't just a small improvement; it's a massive jump. The model proved it could handle complex scenarios, like tracking a sound source that gets hidden behind an object and then reappears, or distinguishing between two similar sounds moving in different directions.

The researchers also tested their model on three other real-world audio datasets (TAU-NIGENS, L3DAS22, and STARSS23) that it had never seen before. Even without being specifically trained on those, ST-Omni-R1 outperformed the previous best models. This suggests that the way the model learned to understand space and motion is a skill it can use in many different situations, not just the specific test they built.

Why This Matters

This paper shows that we are getting closer to giving computers the human-like ability to "hear and see" the world as a single, moving story. By teaching machines to track the journey of a sound source—knowing where it started, where it went, and what it looked like even when it was hidden—we are building a foundation for robots and virtual assistants that can truly understand our dynamic, noisy, and visual world. The authors suggest that this approach of combining 3D sound with panoramic video and teaching the AI to reason step-by-step is the key to unlocking this next level of intelligence.

Technical Summary: Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

Problem Statement

Understanding dynamic sound sources requires a model to jointly determine what produces a sound, where the source is located, and how it moves over time. While existing audio-language models have advanced in global event semantics, they typically overlook the time-varying geometry of individual sources. Conversely, vision-language models and current omni-modal models often rely on monaural audio, lacking the spatial cues necessary for source localization and trajectory tracking.

Current methods address only fragments of this problem:

SELD models recover activity and time-varying directions but are limited by fixed vocabularies and structured outputs, hindering open-ended reasoning.
Spatial LALMs (e.g., BAT, OWL, SPUR) support reasoning about identity, direction, and distance but primarily characterize static scenes or clip-level attributes.
Audio-only approaches (e.g., ST-AudioLM) model moving trajectories but fail to bind these acoustic trajectories to visible instances or reason about occlusion and landmarks.

This creates three specific gaps: a perception gap in recovering time-varying source geometry, a binding gap in associating acoustic trajectories with visible entities, and a reasoning gap in composing spatial, temporal, and cross-modal evidence.

Methodology

1. Benchmark: ST-OmniQA

To evaluate these missing capabilities, the authors introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark.

Data Composition: 40,000 synchronized 10-second panoramic videos paired with First-Order Ambisonics (FOA) audio. The dataset includes 400,000 question-answer pairs.
Scene Generation: Scenes are rendered in navigable indoor environments (based on Matterport3D) with acoustics simulated via SoundSpaces 2.0. Configurations include single/multiple static and dynamic sources.
Annotation: Each source is annotated with a multimodal state $S_i(t)$ capturing event identity, activity, azimuth/elevation/distance, motion, visibility, and relations to landmarks/occluders.
Capability Levels: Questions are organized into four progressive levels:
- Level A: Single-source acoustic perception (event, DoA, distance).
- Level B: Multi-source spatial perception (selection, disambiguation).
- Level C: Spatio-temporal relational reasoning (time-conditioned relations, trajectory comparison).
- Level D: Scene-grounded audio-visual reasoning (instance binding, occlusion, tracking after visual disappearance).
Quality Control: Level D questions enforce modality-necessity constraints, ensuring the answer is unique only when audio and visual evidence are jointly bound (i.e., neither modality alone suffices).

2. Model: ST-Omni-R1

The proposed model, ST-Omni-R1, integrates FOA-derived representations with panoramic visual context.

Architecture:
- STA-encoder: A specialized encoder for time-varying FOA. It processes 4-channel FOA waveforms (W, X, Y, Z) into log-Mel spectrograms and acoustic intensity vectors. It outputs one semantic token (for event identity) and K temporally ordered trajectory tokens (for motion states).
- Visual Encoder: Uses Qwen2.5-VL-7B-Instruct to process panoramic video.
- Connector: Maps audio tokens into the language model's embedding space for joint decoding with video and text tokens.
Training Strategy:
- Initialization: The STA-encoder is pre-trained with static perception (event, DoA, distance) and dynamic adaptation (dense supervision for activity, direction, and distance over time) using a trajectory loss function.
- Progressive Curriculum Learning (Stage I): The model is trained in four stages (A–D) corresponding to the benchmark levels. This gradually increases reasoning difficulty from single-source perception to multi-source spatio-temporal analysis and scene-grounded reasoning.
- Reasoning-Tree Reinforcement Learning (Stage II): To enforce consistency in reasoning steps, the model undergoes reinforcement learning. A task-specific tree is constructed from annotations, where intermediate nodes represent source states and bindings. A reward function evaluates format compliance, node-level reasoning consistency, and final answer accuracy, optimized via Group-Relative Policy Optimization (GRPO).

Key Contributions

ST-OmniQA Benchmark: A multi-level benchmark with 40K videos and 400K QA pairs designed with modality-necessity constraints and executable reasoning graphs to evaluate source perception, multi-source localization, trajectory relations, and compositional reasoning.
ST-Omni-R1 Model: A novel architecture featuring an STA-encoder for time-varying FOA and a progressive curriculum learning framework combined with reasoning-tree reinforcement learning to support increasingly compositional spatio-temporal reasoning.
Comprehensive Evaluation: Extensive task-level comparisons, stage-wise ablations, and transfer evaluations on three public spatial-audio benchmarks (TAU-NIGENS, L3DAS22, STARSS23) demonstrating performance on ST-OmniQA and the transferability of learned representations.

Results

ST-OmniQA Performance: ST-Omni-R1 achieves an average semantic accuracy of 77.83% across the four capability levels (with reasoning-tree RL), significantly outperforming the best baseline (Gemini-2.5-Pro at 37.28%) and other open-source models (e.g., Qwen2.5-Omni at 27.20%).
- It shows particular strength in Level D (Scene-grounded reasoning), achieving 94.70% accuracy.
Transferability: On three real-world spatial-audio benchmarks, ST-Omni-R1 consistently outperforms the spatial-audio baseline BAT.
- On STARSS23 (which includes video), the model achieves 51.54% overall accuracy compared to BAT's 11.74%, highlighting the benefit of binding acoustic trajectories to visual evidence.
- On audio-only benchmarks (TAU-NIGENS, L3DAS22), it also shows superior performance, indicating that the learned spatial and motion representations generalize beyond the specific ST-OmniQA dataset.
Ablation Studies:
- Curriculum Learning: Progressive training from Level A to D is essential; skipping stages results in significant performance drops, particularly in complex reasoning tasks.
- Modality: Combining audio and visual inputs yields a more balanced capability profile than either modality alone.
- RL: The reasoning-tree reinforcement learning stage further consolidates capabilities, improving accuracy across all levels.

Significance

The paper argues that dynamic sound-source understanding is fundamentally a tracking and binding problem. The significance of this work lies in bridging the gap between acoustic perception and visual grounding for moving sources. By introducing a benchmark that requires joint reasoning over source semantics, location, motion, and visual identity, and by demonstrating a model that successfully learns these spatio-temporal representations, the work advances omni-modal language models toward handling persistent multi-modal entities whose properties evolve over time. The results suggest that explicit modeling of time-varying FOA trajectories and the use of curriculum-based reinforcement learning are critical for achieving robust spatio-temporal audio-visual reasoning.