MomentMix Augmentation with Length-Aware DETR for Temporally Robust Moment Retrieval

This paper proposes MomentMix, a data augmentation strategy combining ForegroundMix and BackgroundMix, and a Length-Aware Decoder to address feature diversity limitations and prediction biases, thereby significantly improving the localization accuracy of short moments in Video Moment Retrieval tasks.

The Gist

Imagine you have a massive library of video clips, like a giant YouTube channel, and someone asks you a question: "Show me the part where the dog catches the frisbee." Your job is to find that exact moment in the video and highlight it. This is called Moment Retrieval.

While modern AI is getting really good at finding long scenes (like a whole soccer match), it often gets completely lost when the moment is very short (like just the split second the dog jumps). It's like trying to find a specific needle in a haystack, but the AI keeps grabbing the whole haystack instead of just the needle.

This paper introduces a new system called LA-DETR (Length-Aware DETR) that fixes this problem using two clever tricks: MomentMix and a Length-Aware Decoder.

Here is how it works, explained with simple analogies:

1. The Problem: The "Short Moment" Blind Spot

The researchers noticed that current AI models are terrible at finding short clips.

• The Data Issue: Short moments are like a small, boring room where everyone is wearing the same gray shirt. There isn't much variety. Because the AI only sees these "gray shirts" (limited visual features), it gets confused and can't tell them apart from the background.
• The Model Issue: The AI tries to guess where a moment starts and ends by guessing its center and its length. For long scenes, this is easy. But for short scenes, the AI is bad at guessing the exact center point. It's like trying to throw a dart at a tiny target while blindfolded; you usually miss the center.

2. The First Solution: MomentMix (The "Chef's Special")

To fix the "boring room" problem, the authors created a data augmentation technique called MomentMix. Think of this as a chef who wants to make a short, delicious dish but only has a few ingredients. Instead of just serving the same dish, the chef remixes it.

MomentMix works in two stages:

• Stage 1: ForegroundMix (The "Lego Breaker"): Imagine you have a long, boring video of a guy walking. The AI cuts this long video into tiny, short Lego blocks. Then, it shuffles those blocks and rearranges them to create new short scenes. This forces the AI to learn that the "action" (the foreground) can happen in many different combinations, not just one way.
• Stage 2: BackgroundMix (The "Green Screen Swap"): Imagine you have a short clip of a dog running. The AI keeps the dog but swaps the background (the park) with a background from a totally different video (like a kitchen or a beach). This teaches the AI: "Hey, the dog is the important part, not the park. Don't get distracted by the background!"

The Result: The AI is now trained on thousands of "fake" short moments that look very different from each other, making it much better at spotting the real ones.

3. The Second Solution: Length-Aware Decoder (The "Specialized Team")

To fix the "bad dart throwing" problem, they changed how the AI thinks about length.

Previously, the AI used a single "generalist" team to find moments of all sizes. It was like asking one person to find a grain of sand, a basketball, and a car all at once. They would get confused.

The new Length-Aware Decoder hires a specialized team:

• The Short Experts: A group of AI "queries" dedicated only to finding tiny moments (under 10 seconds).
• The Medium Experts: A group for medium-sized moments.
• The Long Experts: A group for long moments.

How it works:
When the AI sees a query, it assigns it to a specific "expert" based on how long the answer is expected to be.

• If the answer is a Short Moment, the "Short Expert" focuses intensely on the center of the action (because for tiny things, the center is the most important part).
• If the answer is a Long Moment, the "Long Expert" focuses on the edges (start and end points).

By separating the tasks, the AI stops trying to be a jack-of-all-trades and becomes a master of one.

4. The Results: Why It Matters

The researchers tested this on famous video datasets (like QVHighlights and TACoS).

• Before: The AI was great at finding long highlights but failed miserably at short ones.
• After: With MomentMix and the specialized team, the AI became a "needle finder." It improved its ability to find short moments by a huge margin (sometimes doubling the accuracy).

Summary Analogy

Imagine you are looking for a specific 5-second clip of a cat meowing in a 2-hour movie.

• Old AI: Tries to scan the whole movie with a wide net. It misses the 5-second clip because it's too small and gets distracted by the background noise.
• New AI (LA-DETR):
  1. MomentMix: It creates a "training camp" where it practices finding cats in different rooms, with different backgrounds, and in different lighting, so it knows exactly what a "cat meow" looks like regardless of the setting.
  2. Length-Aware Decoder: It sends a tiny, hyper-focused detective (the Short Expert) specifically to look for 5-second events, ignoring the rest of the movie.

This combination allows the computer to find those tiny, crucial moments that were previously invisible, making video search much faster and more accurate for everyone.

Technical Summary

1. Problem Statement

Video Moment Retrieval (MR) aims to localize specific temporal segments within a video that correspond to a natural language query. While recent DETR-based models (e.g., QD-DETR, UVCOM) have achieved state-of-the-art performance by eliminating the need for complex post-processing (like Non-Maximum Suppression), they exhibit a significant performance degradation when retrieving short moments (typically <10 seconds).

The authors identify two primary root causes for this degradation through empirical analysis:

1. Data Perspective (Limited Feature Diversity): Short moments exhibit a highly concentrated feature distribution compared to longer moments. Analysis using Independent Component Analysis (ICA) reveals that short moments have a much smaller 95% confidence ellipse and lower Median Absolute Deviation (MAD), indicating they lack diverse visual context and are harder to generalize.
2. Model Perspective (Prediction Bias): Existing models predict moments as $(center, length)$. The authors found that for short moments, the accuracy of predicting the center position is significantly lower (only ~37% of predictions fall within the ground truth) compared to middle or long moments. The models struggle to distinguish the precise boundaries of short events, often merging them or misaligning their centers.

2. Methodology

The proposed framework, LA-DETR, addresses these issues through two novel components: MomentMix (data augmentation) and a Length-Aware Decoder (LAD) (model architecture).

A. MomentMix: Two-Stage Data Augmentation

To combat the lack of feature diversity in short moments, the authors propose a mix-based augmentation strategy specifically designed for proposal-free DETR frameworks. It operates in two stages:

1. ForegroundMix (FGMix):
   • Goal: Increase the diversity of foreground features.
   • Mechanism: Long foreground segments are cut into smaller sub-foregrounds. These sub-segments are then shuffled and recombined with shuffled background segments from the same video. This creates new, diverse short foreground samples by mixing rich features from longer contexts.
2. BackgroundMix (BGMix):
   • Goal: Enhance the diversity of background contexts to prevent overfitting to specific video backgrounds.
   • Mechanism: The foreground of a short moment is preserved, but the background is replaced with randomly cropped temporal segments from other videos. Crucially, these replacement backgrounds are selected from videos with semantically dissimilar queries to ensure the model learns robust boundaries between relevant and irrelevant content without introducing query-related noise.

B. Length-Aware Decoder (LAD)

To address the prediction bias in center and length estimation, the authors introduce a decoder that conditions predictions on moment length.

• Length Classification: Moments are categorized into classes (e.g., Short, Middle, Long) based on inflection points in cumulative mAP graphs.
• Class-Pattern Embeddings: Inspired by Anchor-DETR, the decoder queries are assigned specific "length-class" embeddings. This creates "length-wise expert" queries, where specific queries are dedicated to handling specific duration ranges.
• Length-Wise Bipartite Matching: The standard bipartite matching process is modified. Instead of matching all queries to all ground truths globally, matching is performed within each length class. Queries assigned to the "Short" class are only matched with ground-truth short moments. This forces the model to learn specific patterns for short durations, significantly improving center prediction accuracy.

3. Key Contributions

1. Root Cause Analysis: First to systematically analyze short-moment retrieval failure from both data (feature concentration) and model (center prediction bias) perspectives.
2. MomentMix: A novel two-stage augmentation strategy (ForegroundMix + BackgroundMix) tailored for proposal-free DETR models to enhance short-moment feature diversity.
3. Length-Aware Decoder (LAD): A new decoder architecture that introduces length conditioning and class-wise bipartite matching, effectively creating specialized "experts" for different moment durations.
4. State-of-the-Art Performance: The method achieves significant gains across multiple benchmarks, particularly for short moments.

4. Experimental Results

The method was evaluated on QVHighlights, Charades-STA, and TACoS datasets, using QD-DETR, TR-DETR, and UVCOM as baselines.

• QVHighlights (Short Moments):
  • R1@0.7: Improved by 9.62% (from 3.95 to 10.33 for QD-DETR).
  • mAP Average: Improved by 16.9% (from 39.84 to 46.61 for QD-DETR).
  • The method consistently outperformed all baselines across Short, Middle, and Long categories.
• Charades-STA & TACoS:
  • Achieved the highest R1@0.7 on Charades-STA (+3.58% gain) and TACoS (+5.82% gain).
  • Demonstrated strong generalization across datasets with varying moment length distributions.
• Ablation Studies:
  • Component Analysis: Both MomentMix and LAD contributed independently, but their combination yielded the best results, confirming they address different aspects of the problem.
  • Few-Shot Learning: In scenarios with only 10% of training data, LA-DETR outperformed the baseline trained on 100% of the data, proving the efficacy of MomentMix in generating high-quality synthetic data.
  • Attention Visualization: Attention maps showed that LAD queries for short moments focus on the interior of the moment (inside attention), whereas long-moment queries focus on boundaries, aligning with the theoretical need for precise centering in short events.

5. Significance

This paper makes a critical contribution to the field of Video Moment Retrieval by addressing a specific but prevalent weakness in current DETR-based models: the inability to accurately localize short, high-value content (e.g., sports highlights, news clips).

• Practical Impact: By improving short-moment retrieval, the method enables more efficient video exploration and summarization, which is crucial for real-world applications like search engines and content recommendation systems.
• Methodological Innovation: The introduction of "length-wise expert" queries and a specialized mix-based augmentation for temporal data provides a new paradigm for handling class imbalance and feature scarcity in temporal tasks.
• Generalizability: The approach is modular and can be seamlessly integrated into other DETR-based architectures, offering a scalable solution for improving temporal robustness in video understanding.