💻 computer science

AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering

The paper proposes AV-Master, a novel audio-visual question answering framework that enhances reasoning capabilities in complex scenes by employing a dynamic adaptive focus sampling mechanism for temporal relevance, a preference-aware strategy for modality selection, and a dual-path contrastive loss to learn question-specific cross-modal representations, thereby significantly outperforming existing methods on large-scale benchmarks.

Original authors: Jiayu Zhang, Shuo Ye, Qilang Ye, Xun Lin, Zihan Song, Zitong Yu

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiayu Zhang, Shuo Ye, Qilang Ye, Xun Lin, Zihan Song, Zitong Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a busy, noisy concert. There are dozens of instruments playing, lights flashing, and people moving around. Suddenly, someone asks you a specific question: "Which flute started playing first?" or "How many drums are sounding right now?"

To answer correctly, you can't just glance at the whole scene or listen to the whole song. You have to zoom in on the exact moment the flute started and tune your ears specifically to the sound of flutes, ignoring the drums and the crowd.

This is exactly what the new AI model, AV-Master, is designed to do. It's a computer program that watches videos and listens to audio to answer questions about them. The researchers found that previous AI models were like tourists at that concert: they saw the whole crowd and heard the whole noise, but they often got confused by the details or missed the specific clue they needed.

Here is how AV-Master works, broken down into simple concepts:

1. The Problem: Too Much Noise, Wrong Focus

Existing AI models have two main trouble spots:

The "Blurry Camera" Problem: When a video is long, old models try to look at everything at once. This creates a lot of "redundant" information (like watching a 10-minute video to find a 1-second event). They often miss the tiny, crucial moment because they are overwhelmed by the rest of the footage.
The "Wrong Ear" Problem: Sometimes a question is best answered by looking (e.g., "What color is the car?"), and sometimes by listening (e.g., "Is the engine making a sputtering sound?"). Old models treat audio and video as a messy mix, rather than deciding which sense is the "boss" for that specific question.

2. The Solution: AV-Master's Two Superpowers

The researchers built AV-Master with two distinct "paths" (or thinking styles) that work together, like a detective with two different tools.

Path A: The "Dynamic Spotlight" (Temporal Dynamic Perception)

Imagine you are watching a video, but instead of watching it at normal speed, you have a smart spotlight.

How it works: As the video plays, this spotlight doesn't just scan randomly. It starts wide, then progressively narrows down. It asks itself, "Is this part of the video relevant to the question?"
The Magic: If the question is about a specific sound, the spotlight ignores the silent parts of the video and zooms in tightly on the exact seconds where the sound happens. It filters out the "boring" parts and the "noise," keeping only the most important, fine-grained clues. This solves the "too much information" problem.

Path B: The "Sense Switch" (Global Preference Activation)

Imagine you are a detective who knows that some clues are visual and some are auditory.

How it works: Before the AI even tries to mix the video and audio together, it asks: "For this specific question, should I trust my eyes more, or my ears more?"
The Magic: It creates a "preference map." If the question is "Which instrument is loudest?", the model turns up the volume on its "hearing" channel and turns down the "seeing" channel. If the question is "Who is wearing a red hat?", it does the opposite. This ensures the model uses the right sense at the right time, rather than getting confused by a mix of both.

3. The Teamwork: Putting It All Together

The genius of AV-Master is that these two paths talk to each other.

The Spotlight finds the tiny, precise moment in time (e.g., the exact second a drum hits).
The Sense Switch tells the model which sense to trust for that moment (e.g., "Listen to the drum, ignore the visual of the drummer's face").
They combine their findings to give a final answer. The researchers call this a "Dual-Path" system because it looks at the problem from two angles simultaneously to ensure nothing is missed.

4. The Results: Why It Matters

The researchers tested AV-Master on four huge datasets (collections of thousands of videos and questions).

The Score: It beat every other existing AI model, including the current "champions" of the field.
The Special Skill: It was especially good at "complex reasoning" tasks. For example, counting how many instruments are playing or figuring out which one started first. These are the kinds of questions that trip up other AIs because they require precise timing and careful listening.
Efficiency: Even though it is very smart, it doesn't need to be a giant, heavy computer program. It achieves these results with fewer "parameters" (the internal settings that make the AI smart) than other top models, making it a more efficient solution.

In Summary

Think of AV-Master as the ultimate concert-goer. While other AIs are overwhelmed by the noise and the crowd, AV-Master has a smart spotlight to find the exact moment of interest and a sense-switch to know whether to listen or look. By combining these two skills, it can answer tricky questions about videos and sounds better than anyone else has before.

1. Problem Statement

Audio-Visual Question Answering (AVQA) requires models to reason over complex audio-visual scenes to answer diverse questions. Existing methods face two fundamental limitations:

Temporal Redundancy and Fragmentation: Current sampling methods either use discrete selection (e.g., uniform or top-k), which risks losing subtle temporal cues and causing hallucinations, or continuous dense sampling, which introduces significant redundancy and hinders reasoning efficiency.
Inflexible Modality Dependency: Most models treat audio and visual modalities as supplementary or fuse them before assessing their importance. They fail to dynamically identify which modality (audio, visual, or joint) is dominant for a specific question. Existing modality-weighting strategies often occur after multimodal entanglement, making it difficult to explicitly enhance question-relevant unimodal evidence.

2. Methodology: AV-Master

The authors propose AV-Master, a novel framework featuring a Dual-Path Collaborative Learning architecture. It consists of two parallel pathways that interact to produce the final answer:

A. Temporal Dynamic Perception Path (TDPP)

This path focuses on extracting fine-grained, question-relevant temporal evidence from redundant streams.

Dynamic Adaptive Focus Sampling: Instead of fixed sampling, the model uses predefined templates ( $A_{cls}, V_{cls}$ $A_{c l s}, V_{c l s}$ ) that iteratively update from time step $0$ to $T$ $T$ .
- It employs Self-Attention Blocks (SAB) with learnable biases to highlight key regions with significant variations.
- It uses Cross-Attention Blocks (CAB) to interact between the current frame features and the accumulated template features.
- This process progressively refines coarse features into fine-grained "focus features" ( $\tilde{F}_v, \tilde{F}_a$ ) that capture the most relevant segments.
Audio-Visual Key Fusion: The refined focus features are fused with word-level question features via a multimodal decoder to create a refined semantic anchor ( $F_{fu}$ ).

B. Global Preference Activation Path (GPAP)

This path addresses the question-conditioned modality dependency by decoupling modalities before fusion.

Independent Preference Modeling: It processes visual and audio features independently using Self-Attention and Cross-Attention (guided by the question) to generate Visual Preference Features ( $F^p_v$ ) and Audio Preference Features ( $F^p_a$ ).
Global Activation: These features represent the model's global bias toward a specific modality for the given question, providing an auxiliary perspective that complements the fine-grained temporal cues from the TDPP.

C. Optimization Strategy

The model is trained using a Dual-Path Contrastive Loss alongside standard prediction losses:

Answer Loss ( $L_{qa}$ ): Standard cross-entropy loss for the final answer prediction from the fused path.
Preference Losses ( $L^p_v, L^p_a$ ): Losses calculated from the independent audio and visual decoders to ensure each modality learns its specific predictive capability.
Contrastive Loss ( $L_c$ ): A novel loss that maximizes the similarity between the fused feature ( $F_{fu}$ ) and the preference features ( $F^p_v, F^p_a$ ) of the positive sample, while minimizing similarity with negative samples. This enforces consistency between the temporal dynamic perception and the global preference activation.

3. Key Contributions

Dual-Path Architecture: A unified framework combining Temporal Dynamic Perception (for fine-grained localization) and Global Preference Activation (for modality selection), addressing both temporal redundancy and modality imbalance simultaneously.
Dynamic Adaptive Focus Sampling: A mechanism that combines the benefits of discrete and continuous sampling. It uses iterative template updates with learnable biases to progressively extract critical information while suppressing redundancy.
Pre-Entanglement Modality Modeling: Unlike previous methods that weight modalities after fusion, AV-Master models modality preferences directly from initial unimodal representations, allowing for more explicit and effective question-conditioned guidance.
Dual-Path Contrastive Learning: A training objective that aligns the fine-grained temporal path with the global preference path, enhancing the model's discriminative ability and stability.

4. Experimental Results

The model was evaluated on four large-scale benchmarks: MUSIC-AVQA, MUSIC-AVQA-R, MUSIC-AVQA-v2.0, and AVQA.

State-of-the-Art Performance: AV-Master achieved 78.51% accuracy on MUSIC-AVQA, outperforming the previous SOTA (QA-TIGER, 77.62%). It showed particularly strong gains in complex reasoning tasks like counting (e.g., +2.16% on A-Counting).
Robustness: On the robustness-oriented MUSIC-AVQA-R dataset (with rephrased questions and long-tail distributions), AV-Master achieved 71.19%, a +3.20% improvement over the previous best, demonstrating superior generalization to diverse question formulations.
Efficiency vs. Performance:
- AV-Master uses 34M parameters, significantly fewer than QA-TIGER (63M) and large pre-trained models (e.g., 7B-13B parameters).
- It achieves higher accuracy than pre-training-based methods (like VideoLLaMa, AVicuna) in zero-shot settings and competes favorably with fine-tuned large models despite having a much smaller footprint.
Ablation Studies:
- Removing the TDPP caused the largest performance drop, confirming the importance of fine-grained temporal perception.
- Removing the GPAP significantly reduced performance, validating the necessity of global modality preference modeling.
- The dual-path contrastive loss was shown to be crucial for learning discriminative cross-modal representations.

5. Significance

Architectural Innovation: AV-Master shifts the paradigm from "aggregate-then-reason" to "perceive-then-activate," proving that explicit modeling of temporal focus and modality preference before deep fusion yields superior reasoning capabilities.
Resource Efficiency: It demonstrates that carefully designed inductive biases (dual-path, adaptive sampling) can outperform massive pre-trained models in specific domains (AVQA) without requiring billions of parameters or massive pre-training datasets.
Practical Application: The model's ability to handle redundant data and adapt to varying modality importance makes it highly suitable for real-world scenarios involving complex, dynamic audio-visual environments where precise temporal localization and modality selection are critical.