Technical Summary: Denoise-then-Retrieve: Text-Conditioned Video Denoising for Video Moment Retrieval

Problem Statement

Text-driven Video Moment Retrieval (VMR) aims to localize specific temporal segments within untrimmed videos that semantically match a free-form text query. While Transformer-based architectures have become prevalent for their strong feature interaction capabilities, existing methods suffer from a critical limitation: they treat all video clips equally during multimodal encoding.

In real-world untrimmed videos, text-relevant clips (including ground truth and challenging non-target frames) often occupy a small fraction of the total duration, while text-irrelevant "noisy" clips dominate. For instance, analysis of the QVHighlights dataset reveals that ground truth clips typically occupy less than 30% of a video, whereas noisy clips account for over 60%. Current approaches that process all clips indiscriminately introduce semantic noise into multimodal representations, disrupting alignment and hindering optimization. Traditional two-stage methods attempt to mitigate this by generating numerous proposals, but this reduces efficiency and complicates the matching process.

Methodology: The Denoise-then-Retrieve Paradigm

To address the issue of semantic noise, the authors propose a Denoise-then-Retrieve paradigm. Instead of processing all clips, the method explicitly filters out text-irrelevant clips to narrow the retrieval range and purify multimodal representations. This is implemented via the Denoise-then-Retrieve Network (DRNet), which consists of three core components:

1. Text-Conditioned Denoising (TCD) Module

The TCD module dynamically identifies noisy clips and generates a noise mask to purify video representations. It operates through the following steps:

• Cross-Attention: Text embeddings are used as keys and values to query video representations, creating text-aware video features.
• Context Interaction Operator (CIO): Based on Structured State Space Models (specifically Mamba), the CIO propagates context in both forward and backward temporal directions. It integrates text-aware video features with text features and learnable global tokens to capture intra-modal, cross-modal, and global context.
• Dynamic Denoising: Text features are pooled and passed through a fully connected layer to generate dynamic convolution kernels. These kernels perform point-wise convolutions on the visual features to produce text-clip alignment scores.
• Masking: A threshold is applied to these scores to generate a binary noise mask. Clips with low alignment scores are masked out, resulting in "purified" visual representations.

2. Text-Reconstruction Feedback (TRF) Module

To provide auxiliary supervision for the denoising process during training, the TRF module ensures that the purified video representations retain the semantic essence of the input text.

• A learnable query embedding interacts with the purified video features (via CIOs) to regenerate a sentence-level query embedding.
• This regenerated embedding is aligned with the original input text embedding using a semantic consistency loss (cosine similarity).
• Minimizing this loss forces the TCD module to retain only the visual features that are semantically consistent with the text, thereby improving the quality of the denoising.

3. Decoder

The decoder performs multimodal interaction on the purified (noise-masked) video and text features to retrieve the target moment. It utilizes:

• Global Retrieval: Regressing the central temporal coordinate and span using global features.
• Boundary Prediction: Predicting left and right offsets for each clip using convolutional layers.
• Foreground Classification: Classifying clips as foreground (relevant) or background using text-conditioned convolution kernels.
• Contrastive Learning: Incorporating intra-video and inter-video contrastive learning to further refine feature alignment.

Key Contributions

The paper outlines four primary contributions:

1. DRNet Architecture: The proposal of a novel network integrating Text-Conditioned Denoising (TCD) and Text-Reconstruction Feedback (TRF) to extract purified visual representations, achieving state-of-the-art performance.
2. Text-Conditioned Denoising Approach: A method integrating cross-attention and structured state space blocks (Mamba) for multi-level multimodal fusion, utilizing dynamic kernels for accurate noise identification.
3. Text-Reconstruction Feedback Mechanism: An auxiliary supervision strategy that aligns regenerated queries from purified features with input text, enhancing the denoising process.
4. Paradigm Generalization: Demonstration that the "denoise-then-retrieve" paradigm can be seamlessly integrated into existing advanced VMR models (e.g., UniVTG, VMS) to boost performance without architectural modifications.

Experimental Results

Extensive experiments were conducted on two benchmarks: Charades-STA and QVHighlights.

• QVHighlights: DRNet outperformed all state-of-the-art methods (including MomentDiff, MESM, UVCOM, and TR-DETR) across all metrics. Notably, it surpassed the nearest competitor (MESM) by 4.36% on the mAP@0.7 metric.
• Charades-STA: The model achieved top-tier performance using both VGG and SlowFast+CLIP backbones. It outperformed the nearest competitor (MESM) by 4.36% on mAP@0.7 with the VGG backbone.
• Ablation Studies:
  • Removing the TCD module resulted in a significant average performance drop of 11.71%, confirming the critical role of noise filtering.
  • Replacing the Mamba-based CIOs with standard Transformer encoders led to a 5.1% performance drop, attributed to Mamba's superior ability to propagate selective information compared to global self-attention.
  • The TRF module was shown to be essential for providing auxiliary supervision to the denoising process.
• Generalization: When applied to existing models like UniVTG and VMS, the denoising paradigm yielded consistent improvements (e.g., UniVTG saw a 2.72% increase in mAP@Avg), validating the adaptability of the approach.

Significance and Claims

The paper claims that the primary bottleneck in current VMR methods is the inclusion of semantically irrelevant noisy clips, which disrupts multimodal alignment. By explicitly filtering these clips before retrieval, the proposed Denoise-then-Retrieve paradigm significantly enhances the quality of multimodal representations.

The authors assert that their approach not only sets new state-of-the-art benchmarks but also offers a generalizable strategy. The ability to integrate this paradigm into existing models without architectural overhaul suggests a practical pathway for improving a wide range of video understanding tasks. The results demonstrate that focusing on "purified" contexts rather than processing all available data leads to more accurate and efficient video moment retrieval.