💻 computer science

CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichment

The paper proposes CAE-AV, a novel framework that improves audio-visual learning by employing cross-modal agreement-guided and caption-aligned saliency modules to dynamically balance spatio-temporal relations and inject semantic guidance, thereby effectively mitigating modality misalignment and achieving state-of-the-art performance on multiple benchmarks.

Original authors: Yunzuo Hu, Wen Li, Jing Zhang

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunzuo Hu, Wen Li, Jing Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a movie. The robot has two eyes (to see the video) and two ears (to hear the audio). Usually, these two senses work perfectly together: you see a dog barking, and you hear a bark.

But in the real world, things get messy. Sometimes the camera cuts away to a different scene while the dog is still barking off-screen. Other times, you see a person playing a violin, but the audio track is actually playing a piano. This is called audio-visual misalignment.

Current AI models get confused by this. They try to force the eyes and ears to match up perfectly, even when they shouldn't, which leads to bad guesses and unstable learning.

The paper introduces a new system called CAE-AV (Caption-aligned and Agreement-guided Enhancement) to fix this. Think of CAE-AV as a smart "traffic controller" for the robot's senses. It uses two special tools to help the robot stay calm and accurate when the video and audio don't match.

The Two Magic Tools

1. The "Agreement Gate" (CASTE)
Imagine you are in a noisy room. Sometimes the person talking is right in front of you (perfect match). Other times, they are in the next room, and you only hear their voice (mismatch).

The CASTE module acts like a smart switch. Before the robot tries to combine what it sees and hears, it asks: "Do these two things agree?"

If they agree: The robot focuses on the spatial details (where things are in the picture).
If they disagree: The robot switches to temporal mode (looking at the sequence of events over time) to figure out what's happening, rather than getting stuck on the wrong picture.

It's like a detective who knows when to look at a crime scene photo and when to listen to the timeline of events, depending on whether the evidence matches. This prevents the robot from getting confused by "off-screen" sounds or background noise.

2. The "Caption Anchor" (CASE)
Sometimes, even with the switch, the robot still gets lost. To help, the system brings in a third party: a smart narrator (an AI that reads the video and audio and writes a short caption, like "A man is playing a guitar").

The CASE module uses this written caption as a "truth anchor." It doesn't just let the robot guess; it says, "The caption says 'guitar,' so focus your attention on the guitar, even if the audio is a bit fuzzy or the camera angle is weird."

This is like having a tour guide who points out the important landmarks. Even if the view is blocked or the audio is loud, the guide's description helps the robot know exactly what to look for.

How They Work Together

The paper claims that by using these two tools, the system can learn much better without needing to retrain its entire brain (which saves a huge amount of computing power).

CASTE handles the timing and location issues (deciding whether to look at the picture or the timeline).
CASE handles the meaning issues (using the written description to keep the focus sharp).

The Results

The researchers tested this "traffic controller" on four different types of tasks:

Finding events: Locating exactly when a sound happens in a video.
Parsing videos: Breaking a video down into its sound and visual parts.
Segmentation: Drawing a precise outline around the object making the sound (like drawing a line around a barking dog).
Answering questions: Answering questions like "What instrument is that?" based on the video and audio.

In all these tests, CAE-AV performed better than the previous best methods. The paper shows that it is particularly good at handling the messy, real-world situations where the audio and video don't line up perfectly, making the AI more robust and reliable.

In short, CAE-AV teaches the AI to be flexible: to know when to trust the picture, when to trust the timeline, and when to listen to the "narrator" to make sense of a confusing scene.

Technical Summary: CAE-AV

Problem Statement

Audio-Visual Learning (AVL) aims to jointly model visible content and acoustic sources. However, a critical challenge in open-world video scenarios is audio-visual misalignment. This misalignment arises from off-screen sound sources, background clutter, camera cuts (e.g., switching to a distant view while audio continues), and post-production soundtracks (e.g., a violin visual with piano audio).

Existing parameter-efficient methods that operate on frozen pre-trained backbones often assume strict frame-wise alignment. When misalignment occurs, these methods tend to:

Amplify irrelevant regions or moments by forcing cross-modal fusion on mismatched frames.
Introduce noise into the reasoning process, leading to unstable training and degraded representation quality.
Fail to distinguish between scenarios requiring strong spatial focus (aligned) versus those requiring temporal aggregation (misaligned).

Methodology

The authors propose CAE-AV (Caption-aligned and Agreement-guided Enhancement), a framework built upon frozen pre-trained backbones (Swin Transformer for visual and HTS-AT for audio). CAE-AV introduces two complementary modules to relieve misalignment without unfreezing the heavy backbone parameters:

1. Cross-modal Agreement-guided Spatio-Temporal Enrichment (CASTE)

CASTE operates within each layer of the backbone to dynamically balance spatial and temporal relations based on frame-level agreement.

Agreement Gate: It computes frame-level prototypes for visual and audio tokens and projects them into a shared semantic space. It calculates a cosine similarity score ( $g$ ) to measure agreement.
Bias Mechanism: A fusion vector aggregates unimodal evidence, cross-modal discrepancies, and co-occurring features. A Multi-Layer Perceptron (MLP) generates preferences for spatial vs. temporal enrichment. Crucially, the agreement score $g$ biases this decision: frames with high agreement favor spatial modeling, while frames with low agreement (misalignment) favor temporal modeling to aggregate evidence across time.
Enrichment Modules:
- Spatial Enrichment: Uses separable attention (inspired by MobileViT) to recalibrate tokens within a frame with linear complexity, avoiding quadratic costs.
- Temporal Enrichment: Applies depthwise 1D convolutions along the time axis to smooth dynamics and capture local neighborhood information.
Selective Injection: A Top-K saliency mask selects the most discriminative tokens for enrichment, preventing uniform noise injection. The outputs are combined via a convex combination weighted by the agreement gate.

2. Caption-Aligned Saliency-guided Enrichment (CASE)

CASE leverages high-level semantic priors to guide alignment after the backbone processing.

Caption Generation: A frozen Multimodal Large Language Model (MLLM) generates text captions for raw visual frames and audio waveforms. These captions are then encoded by a frozen CLIP text encoder.
Semantic Anchoring: The caption features are projected to match the backbone's channel dimensions.
Guided Refinement: The module uses cross-attention to inject caption semantics into the visual and audio features. It computes agreement between the enriched features and the caption features to determine injection strength.
Temporal Consolidation: Depthwise separable temporal convolutions smooth the features, ensuring stable focus on sounding regions even under occlusion or viewpoint changes.

3. Training Objectives

To guide token selection and strengthen alignment, CAE-AV employs lightweight loss functions:

Caption-to-Modality InfoNCE: A contrastive loss aligns visual/audio features with their corresponding captions, excluding within-video negatives to prevent easy negatives.
Visual-Audio Consistency: Minimizes the cosine distance between modality-specific representations within the same frame.
Entropy Regularization: Encourages sharper Top-K selection probabilities to avoid diffuse token selection.

Key Contributions

Novel Framework: The introduction of CAE-AV, which addresses audio-visual misalignment through two complementary pathways: agreement-guided spatio-temporal balancing (CASTE) and caption-aligned semantic injection (CASE).
Dynamic Adaptation: The CASTE module's ability to dynamically switch between spatial and temporal focus based on estimated cross-modal agreement, allowing the model to handle both aligned and misaligned scenarios effectively.
Semantic Guidance: The utilization of MLLM-generated captions as explicit, interpretable semantic anchors to correct cross-modal attention drift, rather than relying solely on latent feature interactions.
Parameter Efficiency: The method achieves state-of-the-art performance while keeping the heavy pre-trained backbones frozen, adding only lightweight adapters and enrichment modules.

Experimental Results

CAE-AV was evaluated on four downstream tasks using frozen Swin-V2-L and HTS-AT backbones:

Audio-Visual Event Localization (AVE): Achieved 82.6% accuracy on the AVE test set, matching the current state-of-the-art (SOTA) frozen backbone method (AVMoE) with the same backbone configuration.
Audio-Visual Video Parsing (AVVP): Outperformed AVMoE by 1.7% in audio type and 0.6% in event-level metrics on the LLP dataset.
Audio-Visual Question Answering (AVQA): Achieved an average accuracy of 76.7% on MUSIC-AVQA, surpassing AVMoE (75.7%) and other SOTA methods.
Audio-Visual Segmentation (AVS): On AVSBench, CAE-AV improved the Jaccard index (MJ) by +0.8 (S4 setting) and +0.6 (MS3 setting) compared to AVMoE.

Ablation Studies confirmed that CASTE significantly improves temporal reasoning and localization in misaligned scenarios, while CASE enhances semantic focus and alignment. The combination of both modules yields the best results across all tasks.

Qualitative Analysis demonstrated that CAE-AV successfully handles complex scenarios such as:

Segmentation of objects when the camera angle changes (e.g., lawn mowing).
Distinguishing between multiple sound sources (e.g., piano vs. violin).
Maintaining segmentation consistency when the sound source is temporarily occluded or off-screen.

Significance

The paper claims that CAE-AV provides a robust solution to the pervasive problem of audio-visual misalignment in open-world videos. By decoupling the heavy backbone from the adaptation process and introducing explicit mechanisms to detect and handle misalignment (via agreement gates and semantic captions), the framework achieves state-of-the-art performance with remarkable parameter efficiency. The authors emphasize that their approach stabilizes training and inference without the computational cost of unfreezing large pre-trained models, making it a practical solution for multi-task audio-visual applications.