⚡ electrical engineering

Unlocking Spatial Grounding in Large Audio-Visual Retrieval models

This paper introduces LAIP, a framework that leverages intermediate visual tokens and audio-informed spatial pooling to unlock fine-grained spatial grounding capabilities from large-scale audio-visual retrieval models without requiring pixel-level supervision, achieving state-of-the-art performance on localization benchmarks.

Original authors: Hugo Malard, Michel Olvera, Sanjeel Parekh, Gaël Richard, Slim Essid, Stéphane Lathuilière

Published 2026-07-29

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Hugo Malard, Michel Olvera, Sanjeel Parekh, Gaël Richard, Slim Essid, Stéphane Lathuilière

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world not just by looking at it, but by listening to it too. This is the world of audio-visual learning, a branch of artificial intelligence where computers try to figure out how sound and sight connect. Think of it like a detective who doesn't just see a crime scene but also hears the sirens, the breaking glass, and the shouting, trying to piece together exactly where the noise came from.

For a long time, teaching robots to do this was like trying to build a house without a blueprint. To teach a computer to point at exactly where a sound is coming from in a video, you usually need a human to draw a box around that sound source on every single frame of the video. This is called "dense spatial annotation," and it is incredibly tedious, expensive, and slow. It's like asking a friend to draw a perfect outline of every bird in a flock for an hour straight. Because this is so hard, most researchers have been stuck using weak clues, like just knowing that a sound exists in a video, without knowing exactly where it is.

However, recently, scientists have built massive "retrieval models." These are super-smart AI systems trained on millions of videos and sounds to answer questions like, "Find me the video clip that matches this audio." They are excellent at matching the general vibe of a sound to a whole video, but they are terrible at pointing to the specific spot where the sound is happening. They see the whole forest but miss the specific tree making the noise. The big question has been: Can we take these giant, pre-trained models that are great at finding videos, and somehow unlock their ability to point at the exact source of a sound, without needing to hire an army of people to draw boxes?

The Paper's Big Idea: Unlocking the Hidden Map

The paper you are reading, titled "Unlocking Spatial Grounding in Large Audio-Visual Retrieval models," says: Yes, we can. The authors, a team of researchers from France, Meta, and NVIDIA, discovered that these giant retrieval models actually do know where sounds are, but they just hide that information deep inside their brain before they throw it away.

Imagine a giant library (the retrieval model) that has read millions of books (videos) and listened to millions of audiobooks (sounds). When you ask the librarian, "Do you have a book about a barking dog?" they can instantly find the right shelf. But if you ask, "Where exactly in the picture of the dog is the mouth barking?" the librarian usually just shrugs and says, "I just know it's a dog picture."

The authors realized that the librarian isn't actually ignorant. The librarian has a detailed map of the dog's mouth in their head, but they throw that map away the moment they decide to give you a simple "Yes, I have a dog book" answer. The paper introduces a clever trick called LAIP (Localization via Audio-Informed Pooling) to catch that map before it gets thrown away.

How LAIP Works: The Audio Detective

The core of their method is a new module they call Audio-informed Spatial Pooling (AiSP). Here is how it works in plain English:

The Setup: The AI looks at a video frame by frame. Inside the AI, there are thousands of tiny "tokens" (little pieces of data) representing different parts of the image. Usually, the AI squashes all these tiny pieces into one giant summary token to decide if the video matches the sound. This is like taking a whole pizza and mashing it into a single ball of dough to see if it tastes like pepperoni. You lose the location of the pepperoni slices.
The Intervention: Instead of letting the AI mash everything together immediately, the authors insert a "stop sign" right before the final summary. They take the audio (the sound of the barking) and use it as a query.
The Magic: They ask the audio: "Hey, which of these tiny image pieces are you talking to?" The audio acts like a flashlight, shining on the specific image tokens that correspond to the sound. The AI then gathers only those relevant pieces into a new summary.
The Result: Because the AI was forced to pay attention to the specific parts of the image that match the sound, it accidentally learns to draw a map of where the sound is coming from. It's like if the librarian, instead of just handing you the book, had to point to the exact page and paragraph where the word "bark" appears, just to prove they found the right book.

What They Found

The results are surprisingly strong. The authors tested their method on three different benchmarks (standard tests for AI):

AVATAR: On this test, which looks for sounds in complex, moving videos, their method nearly doubled the performance of the previous best models. They jumped from a score of around 13-14% to 27.63% (CIoU) and 27.77% (AUC).
AVSBench: They achieved the best scores on this test as well, with an F-score of 65.18 for single-source sounds, beating the previous best of 45.33.
ADE-SP: They also crushed the competition here, reaching 33.35 m-IoU and 53.57 mAP.

The paper suggests that this works because the giant retrieval models were already trained on massive amounts of data, so they learned rich connections between sound and sight. The authors didn't have to teach the AI what a dog or a car is from scratch; they just had to teach it how to look at the right part of the picture when it heard the sound.

What They Ruled Out

The authors were careful to show what doesn't work. They tried a few other ideas and found them lacking:

Just using the final layer: If you try to do this at the very end of the AI's processing (after it has already mashed the image into a single blob), it fails. The spatial details are gone. You have to do it in the middle layers, where the image is still detailed.
Simple attention: Just asking the AI to look at the image once isn't enough. They found that using a hierarchical approach (doing it in steps, like zooming in from a wide view to a close-up) is necessary.
Gradient maps: They tried using standard math tricks to see what the AI was looking at (called gradient-based maps), but these were terrible at finding the sound, scoring only 4.14 compared to their 26.22. This proves that their specific method of "pooling" is what makes the difference, not just looking at the AI's internal math.

The Catch (Limitations)

The paper is honest about a limitation. While this method makes the AI great at finding where a sound is, it makes it slightly harder to use that same AI for the original job of finding videos quickly. Because the AI now has to look at the sound to decide which part of the video to focus on, you can't pre-calculate the video's "fingerprint" and store it in a database for instant searching. You have to re-calculate it every time you ask a question. The authors note that the original retrieval model still works fine for searching, but the new "localizer" version is a bit slower for massive databases.

The Takeaway

In short, this paper proves that you don't need to build a new AI from scratch to find sound sources. You can take a giant, pre-trained AI that is already good at matching sounds to videos, and with a clever little add-on (LAIP), you can unlock its hidden ability to point at exactly where the noise is coming from. It's a reminder that sometimes the answer isn't to build a bigger brain, but to teach the existing brain how to pay attention to the right details.

Technical Summary: Unlocking Spatial Grounding in Large Audio-Visual Retrieval Models

Problem Statement

Audio-visual sound source localization (AV-SSL) aims to identify the spatial regions within a video frame that correspond to a synchronized audio signal. This task is inherently challenging due to the sparsity of sound sources in dense, cluttered visual scenes. A primary bottleneck is the scarcity of dense spatial annotations (pixel-level masks) required for supervised training, as creating them for long video clips is prohibitively expensive. Consequently, the field has relied on weakly supervised methods that utilize only temporally aligned audio-video pairs.

While large-scale contrastive pretraining has produced powerful multimodal retrieval models (e.g., PE-AV) that align global embeddings of audio and video, these models are optimized for global matching rather than local spatial correspondence. In standard retrieval architectures, dense spatial information is progressively lost through global pooling mechanisms in the upper layers of the backbone, resulting in frame and video features that lack the granularity required for precise localization. The central question addressed by this work is whether these large-scale retrieval models, which inherently discard spatial detail, can be adapted to recover fine-grained spatial grounding using only weak supervision.

Methodology: LAIP Framework

The authors propose LAIP (Localization via Audio-Informed Pooling), a framework designed to inject temporally aligned audio information into the visual backbone of a pretrained retrieval model prior to temporal aggregation.

Core Architecture

LAIP operates on a large-scale retrieval backbone (specifically PE-AV), which consists of audio, frame, and video encoders. Instead of training a localization model from scratch or attaching a heavy head to frozen features, LAIP modifies the interface between the frame encoder and the video encoder.

Intermediate Feature Extraction: Recognizing that spatial detail is suppressed in the final layers of the frame encoder, LAIP extracts intermediate visual tokens ( $V_t^\ell$ ) from an earlier layer (specifically the 16th layer of a 24-layer encoder).
Audio-Visual Alignment: Since audio and video operate at different temporal resolutions, a lightweight local aggregator (a 1D convolutional network followed by temporal average pooling) aligns the audio stream to the video frame rate, producing a synchronized audio embedding $a_t$ for each frame.
Audio-informed Spatial Pooling (AiSP): The core innovation is the AiSP module, which replaces the standard global aggregation. It employs a hierarchical design consisting of three stacked AiSP layers, each followed by a lightweight convolutional adaptation block.
- Mechanism: In each stage, the audio token acts as a query to attend to spatial visual tokens. The attention weights form a segmentation map, and the visual tokens are pooled within local spatial blocks ( $K \times K$ ) weighted by these attention scores.
- Null Token: To handle cases where the audio does not correspond to any visible object (e.g., off-screen sounds), a learnable null token ( $x_\emptyset$ ) is appended to the keys. This allows the audio query to "abstain" from selecting visual tokens if no relevant source is present.
- Progressive Refinement: The hierarchical stacking progressively reduces the spatial resolution of the visual tokens, collapsing them into a single audio-conditioned visual token ( $\tilde{c}_t$ ) that is compatible with the original video encoder.

Training Objective

The model is trained using a global contrastive objective inspired by SigLIP and FLAIR, adapted for the audio-visual domain:

Contrastive Loss ( $L_{SIG}$ ): The pooled video representation (generated via LAIP) is aligned with the global audio [CLS] token using a sigmoid-based contrastive loss. Negative pairs are formed by mismatching audio and video.
Multi-Resolution Regularization ( $L_{MRes}$ ): To ensure the intermediate attention maps are spatially coherent, a loss term enforces consistency between attention maps at different pooling stages (upsampling lower-resolution maps to match higher-resolution ones).
Null Token Regularization ( $L_\emptyset$ ): A penalty is applied to the attention paid to the null token for negative pairs, encouraging the model to abstain from selecting visual regions when no sound source is present.
Adapters: Small adapters are added to the video encoder's MLP blocks to prevent the model from collapsing into imitating the original [CLS] token, while the rest of the pretrained weights remain frozen.

Key Contributions

Exploiting Retrieval Representations: The authors demonstrate that large-scale audio-visual retrieval models, despite being optimized for global alignment, retain exploitable structured spatial information in intermediate visual tokens. This allows for localization without dense spatial annotations.
The LAIP Framework: Introduction of a lightweight, audio-informed architecture that injects audio queries into the visual backbone before video pooling. This enables strong cross-modal synergy while maintaining compatibility with the pretrained retrieval pipeline.
Global Supervision for Local Tasks: The work shows that a global contrastive objective, when combined with the specific architecture (hierarchical pooling) and regularization (multi-scale consistency and null-token handling), is sufficient to learn rich spatial audio-visual representations.

Experimental Results

The method was evaluated on three benchmarks: AVATAR, AVSBench, and ADE-SP.

AVATAR Benchmark: LAIP achieved state-of-the-art performance across all four scenarios (single-sound, mixed-sound, multi-entity, off-screen). Notably, it nearly doubled the performance of previous models on the AVATAR benchmark (e.g., achieving ~27.6% CIoU compared to ~13.4% for the previous best TAVLO). It outperformed baselines even when those baselines operated at higher frame rates.
AVSBench: On the single-source (S4) and multi-source (MS3) test sets, LAIP achieved the best F-scores by a large margin and the best mask-IoU on the challenging MS3 split.
ADE-SP: The model outperformed all baselines, including those explicitly designed for sound-prompted segmentation (e.g., TACO, DenseAV), achieving 33.35 m-IoU and 53.57 mAP.
Ablation Studies: Experiments confirmed the necessity of the hierarchical pooling design, the multi-resolution regularization, and the null-token mechanism. Replacing the hierarchical AiSP with a single attention layer or using features from the final encoder layer resulted in significant performance drops.

Significance and Claims

The paper claims that accurate sound source localization does not need to be learned from scratch on small, curated datasets. Instead, it can be "unlocked" from existing large-scale retrieval representations. By intervening before the final aggregation stage of a retrieval backbone, the authors provide a unified path for both retrieval and localization tasks.

The work highlights that while retrieval models compress spatial information, the intermediate layers contain sufficient structure to be recovered via an audio-informed pooling mechanism. This approach offers a practical regime for AV-SSL that avoids the cost of dense annotation while leveraging the semantic richness of massive pretraining.

Limitations: The authors note a limitation regarding scalable retrieval. Because the visual features in LAIP are conditioned on the input audio, video representations cannot be precomputed independently of the query. While the original PE-AV model can still be used for standard retrieval, the LAIP-enhanced pipeline requires recomputing the video encoder for every audio-video pair, which is computationally impractical for large-scale retrieval systems. Future work is suggested to recover scalable retrieval while retaining localization benefits.