Technical Summary: UniMVU – Instruction-Aware Gating for Multimodal Video Understanding

1. Problem Statement

Pre-trained Video Large Language Models (Video LLMs) have demonstrated strong capabilities in visual reasoning using RGB frames and text instructions. However, they face significant challenges when processing videos accompanied by auxiliary streams such as audio, depth maps, or dense temporal evidence.

The core issue identified is modality interference. In current approaches, auxiliary streams are often uniformly fused or appended as extra tokens. This static fusion strategy assumes a fixed importance for each modality, failing to account for the fact that the relevance of a modality is query-dependent. For instance, audio is critical for questions about speech or instruments but acts as noise for color recognition; depth cues are essential for spatial layout but redundant for appearance-centric queries. Uniform fusion allows irrelevant channels to distract the model, leading to suboptimal performance. Existing methods often rely on implicit decoder attention or task-specific interaction modules that do not explicitly address the dual questions of which tokens within a stream to emphasize and which stream to prioritize based on the specific instruction.

2. Methodology: UniMVU

The authors propose UniMVU, a unified multimodal video understanding framework designed to perform instruction-aware fusion across diverse modalities (video, audio, depth/3D, dense temporal) before LLM decoding. The framework employs a two-level dynamic gating mechanism conditioned on the text instruction.

Architecture Overview

UniMVU integrates modality-specific encoders and projectors with a Modality Balanced Fusion block, followed by a frozen or LoRA-adapted LLM. The fusion block consists of three primary components:

1. Unified Token Sequence & Cross-Modal Self-Attention:

   • All inputs (video, audio, depth, etc.) are projected into a common embedding space and concatenated with text instruction tokens into a unified sequence.
   • A Cross-Modal Self-Attention layer enables interactions between instruction tokens and modality tokens. Crucially, instruction tokens remain "untouched" (replaced with original embeddings after the layer) to prevent noisy modality features from perturbing the instruction representation, ensuring a one-way flow of information where modalities are conditioned on the text.

2. Instruction-Driven Inner-Modality Gating (Feature Level):

   • This module refines the token sequence within each modality.
   • It leverages the attention map from the cross-modal layer to compute the relevance of each token relative to the instruction.
   • Attention mass from instruction tokens to content tokens is aggregated and normalized within each modality to produce token-level weights ($w_{m,t}$). This emphasizes salient regions within a stream while suppressing irrelevant ones.

3. Instruction-Driven Modality-Level Gating (Stream Level):

   • This module assigns a global weight to each entire modality stream.
   • A Learnable Control Token ($t_c$) is introduced for each modality to summarize the stream's content.
   • The attention between instruction tokens and these control tokens generates a scalar importance score ($\eta_m$) for each modality.
   • These scores are normalized to produce modality-level gating coefficients ($\beta_m$).

Fusion Mechanism:
The final representation for a modality $m$ is computed as a residual refinement:
$$\hat{H}_m = O_m + \beta_m (w_m \odot O_m)$$
Where $O_m$ is the output of the self-attention layer. This design ensures that the original evidence is preserved (via the residual $O_m$) while the instruction-relevant tokens and streams are amplified. The gating acts as a soft reweighting signal rather than a hard switch, reducing the risk of discarding potentially useful evidence.

Training & Scalability:

• Modality encoders remain frozen; only projectors, the fusion block, and LoRA adapters in the LLM are trained.
• The framework supports Unified Multi-Task Training, where a single checkpoint is trained across diverse datasets (Audio-Visual, 3D, and Video QA) without task-specific connectors. Missing modalities are masked out, and gates are renormalized over available streams.

3. Key Contributions

• Unified Framework: A single model capable of handling mixed-modality inputs (video, audio, depth/3D, dense temporal) and generalizing across diverse modality configurations without hand-crafted fusion rules.
• Two-Level Dynamic Gating: An instruction-driven mechanism that operates at both the token level (inner-modality) and stream level (modality-level) to adaptively balance modality importance based on the query.
• Interpretability: The gating mechanism aligns with human-interpretable modality relevance (e.g., up-weighting audio for acoustic questions), providing a transparent decision surface.
• Scalability: The approach scales to different model sizes (0.5B and 7B) and supports unified training across multiple task families, eliminating the need for separate connectors for Audio-Visual, 3D, or dense-video tasks.

4. Experimental Results

The authors evaluated UniMVU on six benchmarks: AVQA, AVSD, Music-AVQA (Audio-Visual), ScanQA, SQA3D (3D), and MVBench (Video).

• Performance Gains: UniMVU consistently outperforms static-fusion baselines (specifically PAVE and LLaVA-OV variants).
  • On Music-AVQA, UniMVU achieved a +4.6% improvement in overall accuracy (0.5B) and +1.8% (7B) over PAVE.
  • On AVSD, it achieved a +12.2 point gain in CIDEr (0.5B) and +13.5 (7B).
  • On SQA3D, UniMVU improved EM@1 by +6.7 (0.5B) and showed significant gains on "What" and "How" questions.
  • On MVBench, the unified model (UniMVU†) achieved the best average score, with particularly large gains on temporally demanding subtasks like Action Prediction (+21.0) and Action Sequence (+16.5) at the 0.5B scale.
• Unified Training: A single checkpoint trained jointly across all tasks (UniMVU†) maintained strong performance, demonstrating that the gating interface successfully transfers modality-selection behavior across different task families.
• Ablation Studies:
  • Removing either inner-modality or modality-level gating resulted in performance drops, confirming the necessity of both levels.
  • The use of a single unified control token outperformed modality-specific tokens.
  • The residual connection in the gating formula was critical; removing it degraded performance.
  • Parameter-free gating (using attention scores directly) outperformed learned MLP or cross-attention scoring heads, suggesting attention-derived scores provide a strong inductive bias.

5. Significance and Claims

The paper claims that UniMVU provides a simple, unified recipe for instruction-aware multimodal video understanding. Its primary significance lies in addressing modality interference by explicitly routing information based on the instruction before the LLM generates an answer.

• Mitigation of Interference: By dynamically re-weighting streams and tokens, the model prevents irrelevant modalities from distracting the decoder, a limitation of uniform fusion.
• Generalization: The framework proves that a single gating interface can handle heterogeneous inputs (audio, 3D, dense video) without requiring separate, task-specific architectures.
• Efficiency and Transparency: The method is lightweight (adding minimal parameters) and offers interpretability, as the gating weights directly reflect the model's focus on specific evidence types required by the query.

The authors conclude that UniMVU enables Video LLMs to better leverage synchronized auxiliary streams, achieving state-of-the-art results across diverse benchmarks while offering a scalable path for future multimodal integration.