SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM

Technical Summary: Shot-Aware Multimodal Audio-enhanced Retrieval of Temporal Segments (SMART)

Problem Statement
Video Moment Retrieval (VMR) aims to localize a specific temporal segment within an untrimmed video corresponding to a natural language query. While Multimodal Large Language Models (MLLMs) have advanced semantic understanding for this task, existing approaches face two primary limitations:

1. Unimodal Reliance: Most current MLLM-based methods rely heavily on visual cues, neglecting the audio modality. This limits performance on queries where sound (e.g., speech, sirens) provides critical evidence or resolves temporal ambiguity.
2. Inefficiency with Long Sequences: Processing long videos with MLLMs incurs high computational costs due to quadratic self-attention complexity. Existing compression strategies often discard fine-grained temporal details or fail to leverage the inherent structure of video shots, leading to redundancy or loss of semantic richness.

Methodology: The SMART Framework
The authors propose SMART (Shot-aware Multimodal Audio-enhanced Retrieval of Temporal Segments), an MLLM-based framework designed to jointly leverage audio cues and shot-level temporal structure. The architecture consists of three core components:

1. Dual-Branch Multimodal Encoding:

   • Visual Branch: Uses a pretrained EVA-CLIP encoder to extract frame embeddings, refined by a Q-Former to generate frame-level features.
   • Audio Branch: Utilizes BEATs, an acoustic tokenizer, to encode raw audio into discrete, semantically rich embeddings.
   • Integration: The model constructs a unified multimodal prompt. Visual features are paired with explicit time tokens. Audio features are temporally pooled into a compact representation and concatenated as complementary context, rather than being strictly aligned frame-by-frame with visual tokens. Special separator tokens distinguish between visual and audio modalities.

2. Shot-Aware Token Compression (STC):
   To address the computational burden of long sequences, SMART introduces a two-stage compression strategy that preserves temporal fidelity while reducing redundancy:

   • Stage 1: Keyframe Identification: The video is segmented into shots using TransNetV2. Within each shot, frames are classified as "keyframes" or "non-keyframes" based on inter-frame motion magnitude (L2 norm of patch-wise differences). Keyframes, which capture dynamic content, are retained as full frames.
   • Stage 2: Intra-Shot Token Filtering: For non-keyframes, the model analyzes token-level variance within the Q-Former outputs. Tokens representing dynamic content (high variance) are retained, while tokens representing static backgrounds or redundant content (low variance) are discarded. This selective compression reduces the sequence length without eliminating essential temporal cues.

3. Prompting and Output:
   The compressed multimodal sequence is fed into a lightweight, LoRA-tuned LLM. The model is prompted to generate temporal segments as a nested list of start and end timestamps (e.g., [[s1, e1], [s2, e2]]), allowing for the prediction of multiple disjoint moments if necessary.

Key Contributions

• Audio-Enhanced MLLM for VMR: The paper introduces a framework that effectively integrates audio as complementary context within an MLLM pipeline, demonstrating that acoustic cues improve temporal grounding, particularly for sound-dependent queries.
• Shot-Aware Token Compression: A novel compression strategy that distinguishes between keyframes and non-keyframes. It preserves representative keyframes in full while compressing only non-keyframes based on token variance, balancing efficiency with the retention of fine-grained temporal evidence.
• Efficiency-Effectiveness Trade-off: The design reduces visual redundancy and computational load (memory and latency) while maintaining or improving retrieval accuracy compared to baselines that process full sequences.

Experimental Results
SMART was evaluated on two benchmark datasets: Charades-STA (shorter videos) and QVHighlights (longer videos).

• Performance Gains: SMART consistently outperformed strong baselines, including LLaVA-MR and Mr. BLIP.
  • On Charades-STA, it achieved a 1.61% improvement in R1@0.5 and a 2.59% improvement in R1@0.7 over LLaVA-MR, along with a new best mIoU of 61.09.
  • On QVHighlights (Test Set), it achieved an R1@0.5 of 78.15 and R1@0.7 of 63.16, surpassing LLaVA-MR by 1.56% and 1.68% respectively.
• Qualitative Analysis: Case studies demonstrated that SMART successfully retrieves moments missed by visual-only models by leveraging audio cues (e.g., detecting speech absence or specific location mentions in background audio) and maintaining shot-level consistency through token compression.
• Ablation Studies: Experiments confirmed that both audio integration and the two-stage STC strategy contribute positively to performance. The combination of keyframe selection and token filtering yielded the best results, while aggressive token filtering alone led to performance drops.

Significance and Claims
The paper claims that SMART offers a robust solution for long-video moment retrieval by addressing the dual challenges of multimodal completeness and computational efficiency. By integrating audio cues, the model captures evidence that visual-only systems miss, particularly for temporally ambiguous queries. Simultaneously, the Shot-aware Token Compression strategy enables the processing of longer video sequences without the prohibitive memory costs associated with dense token inputs. The authors position SMART as a framework that achieves a favorable balance between retrieval accuracy and computational cost, making it suitable for real-world applications involving lengthy and complex video content.

Limitations Acknowledged
The authors note that the current audio branch relies on compressed global context rather than fine-grained audio-visual alignment, which may limit performance on subtle audio-event timing. Additionally, while the compression strategy is effective, aggressive compression could potentially weaken subtle visual details in highly dynamic scenes. The model's behavior under significant distribution shifts or in open-world settings remains an area for future investigation.