AV-Unified: A Unified Framework for Audio-visual Scene Understanding

The paper proposes AV-Unified, a novel framework that unifies diverse audio-visual scene understanding tasks into a single architecture by converting heterogeneous inputs and outputs into discrete token sequences and employing multi-scale spatiotemporal perception modules to effectively capture cross-modal associations.

The Gist

Imagine you are walking through a busy city square. You don't just see a street performer playing a violin; you hear the music, you see the bow moving, and your brain instantly connects the two. You know the sound is coming from that specific person, not the traffic behind them. You can even guess how long the song will last or if they are about to stop.

Humans do this effortlessly. We don't separate "seeing" from "hearing" into different mental folders. But for decades, computer scientists have been teaching AI to do these things separately. One program learns to find when an event happens (like a dog barking). Another learns to find where it happens (the dog's location). A third learns to answer questions about it. It's like having three different specialists who never talk to each other, trying to solve one big puzzle.

Enter "AV-Unified."

Think of AV-Unified as a super-intelligent, multi-talented conductor who can lead an entire orchestra at once, rather than hiring separate conductors for the strings, the brass, and the percussion.

Here is how it works, broken down into simple concepts:

1. The Universal Translator (The "Sequence-to-Sequence" Magic)

The biggest problem with current AI is that different tasks speak different "languages."

• Event Localization says: "The dog barked from 2 seconds to 5 seconds."
• Segmentation says: "Here are the exact pixels of the dog."
• Question Answering says: "The dog is on the left."

AV-Unified acts like a universal translator. It takes all these different formats and converts them into a single, standard language: a sequence of tokens (think of them like words in a sentence).

• Instead of saying "2 to 5 seconds," it might say: [Event] [Dog] [Bark] [Start] [End].
• Instead of drawing a mask, it might say: [Pixel] [Dog] [Here].

Because everything is now just a "sentence" of data, the AI can use one single brain to learn all these tasks at the same time, just like a human learns to read, write, and speak simultaneously.

2. The Multi-Scale Time Machine (Temporal Perception)

Imagine watching a video. Some things happen fast (a clap), while others take a long time (a conversation).

• Old AI models often looked at the video in rigid, one-second chunks. This is like trying to understand a movie by looking at one frame every second; you miss the flow.
• AV-Unified has a Multi-Scale Time Machine. It can zoom in to see fast, tiny details (like a drum hit) and zoom out to see the big picture (like a whole song performance). It understands that events have different "durations" and adjusts its focus accordingly, ensuring it doesn't miss the beginning or the end of an event.

3. The Cross-Modal Detective (Spatial Perception)

This is where the magic of "hearing" and "seeing" truly meets.

• The Problem: In a video, you might see a person playing a guitar, but the AI doesn't automatically know which sound belongs to which person if there are multiple instruments.
• The Solution: AV-Unified uses a Cross-Modal Detective. It uses the sound to guide the eyes, and the eyes to guide the ears.
  • If the AI hears a "violin," it immediately scans the visual patches (tiny pieces of the image) to find the violin.
  • If it sees a violin, it listens for the specific sound of a violin.
  • They help each other, like a detective using a clue from one witness to find the location of another. This solves the problem of "Where is that sound coming from?" without needing a human to draw a box around it first.

4. The Task-Specific Prompt (The "Menu" for the Brain)

Since the AI is doing everything at once, how does it know what to focus on right now?

• Imagine you are at a restaurant. You have a menu with everything on it (appetizers, main courses, desserts).
• AV-Unified uses Task Prompts as the "Order."
  • If you want to know when something happened, the prompt is: "Tell me the time." The AI ignores the spatial location and focuses on the timeline.
  • If you want to know where something is, the prompt is: "Show me the location." The AI ignores the timing and focuses on the pixels.
  • If you want to answer a question, the prompt is the question itself.

This allows the single model to switch gears instantly, acting like a Swiss Army knife that changes its tool based on the job you give it.

Why Does This Matter?

Before AV-Unified, if you wanted an AI to do all these things, you had to build, train, and maintain five different models. It was expensive, slow, and the models couldn't share knowledge.

AV-Unified is like upgrading from a toolbox full of single-use tools to a single, smart robot arm that can weld, paint, and screw simultaneously.

By training on a mix of datasets (videos of dogs, music performances, street scenes), the model learns a deeper, more human-like understanding of the world. It realizes that a "barking dog" isn't just a sound and an image; it's a unified event with a specific time, place, and meaning.

In short: AV-Unified teaches AI to stop treating sight and sound as separate subjects and start treating them as a single, flowing story, just like we do.

Technical Summary

1. Problem Statement

Current research in audio-visual scene understanding is largely fragmented, with tasks such as Event Localization (AVE), Video Parsing (AVVP), Sound Source Localization (SSL), Audio-Visual Segmentation (AVS), and Question Answering (AVQA) typically explored in isolation. This siloed approach presents several critical challenges:

• Lack of Holistic Understanding: Humans perceive scenes by integrating multiple modalities and tasks simultaneously. Existing single-task models fail to capture the inter-task relationships and mutual benefits that arise from joint learning.
• Input/Output Heterogeneity: Different tasks require diverse data formats (e.g., temporal boundaries, bounding boxes, pixel-level masks, or text answers), making it difficult to train a single model across them.
• Temporal and Spatial Complexity:
  • Temporal: Events occur at varying scales and durations, which uniform sampling often fails to capture.
  • Spatial: There is a lack of supervised data linking specific sounds to visual regions, making it hard for models to associate audio cues with visual objects, especially for rare or domain-specific categories.

2. Methodology: AV-Unified Framework

The authors propose AV-Unified, a unified sequence-to-sequence framework that standardizes inputs and outputs across diverse tasks using a shared-parameter architecture. The core components are:

A. Unified Task Representation

The framework converts all inputs and outputs into discrete token sequences:

• Inputs: Video frames are divided into patches, and audio is processed into spectrogram features. Text prompts (questions or task descriptions) are tokenized.
• Outputs: All task outputs (time intervals, bounding boxes, masks, or text answers) are converted into token sequences, allowing a single decoder to handle heterogeneous tasks.

B. Multi-scale Spatiotemporal Perception Model (MS-STPM)

This is the core engine for capturing audio-visual associations, consisting of three modules:

1. Temporal Perception Module (TPM):
   • Addresses the varying durations of events.
   • Utilizes a multi-scale window attention mechanism (stacked shifted-window Transformer) where window sizes increase with network depth.
   • Captures both fine-grained and coarse-grained temporal dependencies by aggregating self-attention and cross-attention across audio and visual modalities at different time scales.
2. Spatial Perception Module (SPM):
   • Addresses the lack of supervised spatial audio-visual alignment.
   • Implements a cross-modal guidance mechanism:
     • Audio-guided Visual Attention: Refines visual patch representations using audio cues.
     • Visual-guided Audio Attention: Refines audio embeddings using visual context.
   • This bidirectional interaction helps the model locate sounding objects even without explicit category annotations.
3. Task-prompt Guided Learning Module (TPGL):
   • Different tasks require different spatiotemporal focus (e.g., AVS needs spatial precision; AVE needs temporal precision).
   • Uses task-specific text prompts as queries to perform attention mechanisms over the temporal and spatial features.
   • This allows the model to dynamically select and emphasize the most relevant features for the specific task at hand.

C. Training Strategy

• Joint Learning: The model is trained on a mix of datasets (AVE, LLP, VGG-SS, AVS, MUSIC-AVQA) using a shared parameter network.
• Loss Function: To prevent catastrophic forgetting, the training iterates by randomly sampling a batch from a single task per iteration and updating parameters based on that task's loss.
• Architecture: The model uses pre-trained CLIP-ViT for visual features and VGGish for audio features, with a unified encoder-decoder structure.

3. Key Contributions

1. Unified Paradigm: First framework to unify temporal localization, spatial localization, pixel-level segmentation, and spatiotemporal reasoning into a single sequence-to-sequence model with shared parameters.
2. MS-STPM Design: Proposes a novel module that effectively handles multi-scale temporal events and models spatial audio-visual associations without relying on pre-trained object detectors that may miss rare categories.
3. Prompt-Guided Adaptability: Introduces a task-prompt mechanism that steers the model's attention, enhancing adaptability across heterogeneous tasks without requiring task-specific architectural branches.
4. Comprehensive Evaluation: Demonstrates that a single model can achieve state-of-the-art or competitive performance across five distinct types of audio-visual tasks.

4. Experimental Results

The framework was evaluated on five benchmark datasets: AVE, LLP, VGG-SS, AVS (S4, MS3, AVSS), and MUSIC-AVQA.

• Temporal Tasks (AVE, LLP): AV-Unified achieved state-of-the-art performance, improving AVE accuracy by 1.3% and 0.7% over single-task baselines.
• Spatial Tasks (VGG-SS, AVS):
  • On VGG-SS, it achieved a 41.24% AUC, surpassing previous methods.
  • On AVS (MS3), joint training improved mIoU by 0.9% compared to single-task training.
• Spatiotemporal Reasoning (MUSIC-AVQA): The unified model significantly improved performance on complex reasoning tasks (Counting, Localization, Temporal), achieving an average accuracy of 76.42%, outperforming specialized models like TASS and COCA.
• Ablation Studies: Removing the MS-STPM components caused significant performance drops (e.g., a ~3% drop in AVQA accuracy without the full module), confirming the necessity of multi-scale perception and cross-modal guidance.
• Visualization: Heatmaps showed that task prompts successfully guide the model to focus on relevant regions (e.g., specific instruments), whereas models without prompts exhibited scattered attention.

5. Significance and Limitations

Significance:

• Efficiency: Eliminates the need for training separate models for each task, reducing computational overhead and storage.
• Generalization: Proves that audio-visual tasks are deeply interconnected; learning them jointly enhances the model's ability to capture complex scene dynamics.
• Foundation for Future Work: Establishes a sequence-to-sequence paradigm for multimodal understanding, paving the way for more comprehensive AI agents that perceive the world holistically.

Limitations:

• Suboptimal Performance on Specific Subtasks: In some highly specialized subtasks (e.g., S4 segmentation or specific AVQA categories), the unified model slightly underperforms compared to specialized single-task models. This is attributed to the difficulty of balancing multiple objectives and the lower sampling rate used for video data to accommodate multi-task training.
• Data Constraints: The authors note that future improvements may require larger, more diverse datasets and more advanced training strategies to fully optimize cross-task collaboration.

In conclusion, AV-Unified represents a significant step forward in moving from fragmented, single-task audio-visual analysis to a holistic, unified understanding of dynamic scenes.