Technical Summary: TB-AVA

Problem Statement

Audio-visual understanding relies on aligning heterogeneous modalities (audio and vision). However, current systems predominantly depend on temporal co-occurrence as the primary alignment cue, binding audio and visual features that fire simultaneously. This inductive bias fails in real-world scenarios where audible events and visible entities do not align frame-by-frame (e.g., an off-screen cat making noise while a silent dog is visible). Under weak video-level supervision, models relying solely on temporal overlap are structurally biased to bind incorrect pairs, a limitation that cannot be resolved merely by increasing model size or training data. Existing approaches either tie text to specific tri-modal backbones/tasks or rely on implicit cross-modal fusion without an external reference signal to verify alignment.

Methodology

The authors propose TB-AVA (Text-Bridged Audio-Visual Adapter), a parameter-efficient fine-tuning (PEFT) framework designed to operate on frozen audio and visual encoders. The core innovation is using text as a semantic anchor to mediate interaction between modalities, decoupling alignment from the backbone training.

Architecture Overview

1. Frozen Encoders: The framework utilizes pre-trained, frozen encoders:
   • Visual: SigLIP2 (chosen for its text-aligned visual representations).
   • Audio: BEATs.
   • Text: SigLIP2 text encoder (shared embedding space with the visual encoder).
2. Lightweight Adapters: TB-AVA inserts trainable adapter blocks between the first 12 layers of the frozen encoders.
3. Text Conditioning:
   • Class-level descriptions (e.g., "a sound of dog barking") are used as prompts.
   • Learnable soft prompts (K tokens) are prepended to the fixed class text embeddings. Only these soft prompts are trainable; the text encoder remains frozen.
   • This produces modality-specific text embeddings ($T_v$ for visual, $T_a$ for audio).

Core Mechanism: Gated Semantic Modulation (GSM)

The heart of TB-AVA is the Gated Semantic Modulation (GSM) block, which controls how cross-modal context is injected into the backbone features.

• Two-Step Retrieval:
  1. Text-Aware Context: The visual bottleneck features attend to the text embeddings to produce a text-aware visual representation ($c^v_t$).
  2. Cross-Modal Retrieval: This text-aware representation queries the audio features to retrieve audio context ($c_a$) that is semantically aligned with the text, rather than just temporally co-occurring.
• Gating Logic:
  • GSM employs a lightweight gating network that generates channel-wise coefficients ($w \in (0,1)$) based on the text embeddings.
  • The final updated feature ($z'$) is a residual sum of the original feature and the gated cross-modal contexts:
    $$z' = z + w_a \odot c_{audio} + w_t \odot c_{text}$$
  • Selective Injection: If the text indicates a channel is semantically relevant, the gate amplifies the context. If the text deems the context irrelevant (e.g., acoustic similarity without semantic match), the gate suppresses it (approaches zero).
  • This mechanism makes the alignment process explicit and inspectable, preventing the "drowning" of useful signals in noise from unreliable cross-modal correlations.

Key Contributions

1. Text as a Semantic Anchor: The paper reframes text not as a task-specific auxiliary signal but as a backbone-agnostic, task-agnostic primitive that resolves audio-visual ambiguity. It serves as a third signal to verify alignment independent of temporal co-occurrence.
2. TB-AVA Framework: A parameter-efficient adapter that enables text-mediated interaction between frozen audio and visual streams without modifying the backbones.
3. Gated Semantic Modulation (GSM): A novel module that performs text-conditioned channel-wise gating. It selectively injects cross-modal context only where text confirms semantic reliability, turning text guidance into a controllable architectural component.
4. State-of-the-Art Performance: The framework achieves new state-of-the-art results on multiple benchmarks while maintaining a high degree of parameter efficiency.

Experimental Results

The framework was evaluated on three benchmarks: AVE (Audio-Visual Event Localization), AVVP (Audio-Visual Video Parsing), and AVS (Audio-Visual Segmentation).

• AVE (Event Localization):
  • TB-AVA achieved 85.0% accuracy with 16.7% trainable parameters.
  • This exceeds the previous best PEFT method (MoLT, 83.5%) by 1.5 percentage points.
  • Qualitative analysis shows TB-AVA correctly identifies event boundaries where audio and visual streams diverge (e.g., stopping "frying" prediction when the visual scene changes, even if low-level acoustic similarity persists), whereas co-occurrence-based models fail.
• AVVP (Video Parsing):
  • TB-AVA achieved the highest F1 scores on 6 out of 10 metrics, particularly those requiring modality assignment (Visual, Audio-Visual, and Type categories).
  • Improvements were most significant on metrics sensitive to weak video-level supervision, confirming the text anchor's role in disambiguating modality sources.
• AVS (Segmentation):
  • On the single-source (S4) split, TB-AVA reached 81.2 mIoU, outperforming AVMoE (81.1), MoLT (80.8), and Mettle (80.7).
  • On the multi-source (MS3) split, it reached 53.4 mIoU, comparable to AVMoE but slightly below MoLT, highlighting a known limitation of fixed-vocabulary text inputs in dense-prediction settings with multiple concurrent sources.
• Generalization:
  • TB-AVA demonstrated consistent gains (+3.7 to +8.1 pp) across diverse and heterogeneous backbone pairs (e.g., CLIP+CLIP, Swin+HTS-AT), indicating the improvement stems from the text-bridged mechanism rather than a specific encoder choice.
  • Backbone analysis confirmed that the performance jump from ~77% (backbone-only) to 85% is attributable to the adapter mechanism, not the encoders themselves.

Significance and Claims

The paper claims that temporal co-occurrence is a fundamentally limited inductive bias for audio-visual learning under weak supervision. By introducing text as an explicit, controllable semantic anchor, TB-AVA resolves the ambiguity of binding audible events to visible entities without requiring full fine-tuning of large backbones.

The authors emphasize that GSM transforms text from an implicit conditioning signal into an inspectable, channel-aligned coefficient. The gates spontaneously specialize into language-grounded and co-occurrence-grounded roles without explicit supervision, effectively acting as a semantic regularizer that sharpens feature spaces. The work suggests that text can serve as a structural primitive for cross-modal grounding beyond simple temporal alignment, offering a scalable and efficient path for audio-visual understanding.