💻 computer science

Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning

This paper proposes a novel dense video captioning framework that utilizes role-specific queries with overlap suppression loss and contrastive alignment to mitigate multi-task interference and temporal redundancy, thereby achieving more precise event localization and semantically rich captions.

Original authors: Seung Hyup Baek, Jimin Lee, Hyeongkeun Lee, Jae Won Cho

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seung Hyup Baek, Jimin Lee, Hyeongkeun Lee, Jae Won Cho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a long, unedited home video of someone cooking a complex meal. Your goal is to act as a smart assistant who does two things simultaneously:

Pinpoint exactly when each step happens (e.g., "The chopping starts at 1:05 and ends at 1:20").
Describe what is happening in that specific moment (e.g., "Chopping onions").

This is called Dense Video Captioning. It's hard because the video is long, messy, and full of overlapping actions.

The paper you shared, titled "Stay in your Lane," proposes a new way to teach computers to do this job better. Here is the breakdown using simple analogies.

The Problem: The "Jack-of-All-Trades" Mistake

Previous computer models tried to use a single "brain cell" (called a Query) to do both jobs at once: finding the time and writing the description.

The Analogy: Imagine asking a single employee to be both a Security Guard (watching the clock and boundaries) and a Tour Guide (describing the scenery).
The Result: The employee gets confused. They might look at the clock but forget to describe the scenery, or they might describe the scenery but miss the exact time the event started.
The Glitch: Because the model is confused, it often gets "stuck" on one scene. It might say, "Fry the chicken" for three different time slots, creating redundant, repetitive captions for the same event. It's like a broken record skipping over the same part of a song.

The Solution: "Stay in Your Lane"

The authors realized that to fix this, you need to stop mixing the jobs. You need specialists. They introduced a system called ROS-DVC with three main upgrades:

1. Role-Specific Queries (The Specialized Team)

Instead of one confused employee, they hired two distinct teams:

The Boundary Team (Localization Queries): Their only job is to look at the video and say, "The action starts here and ends there." They are like a Stopwatch. They don't care about the words; they care about the timeline.
The Storyteller Team (Caption Queries): Their only job is to look at the specific moment the Boundary Team pointed to and say, "This is frying chicken." They are like a Poet. They focus deeply on the details.

The Magic: By separating them, the Stopwatch gets really good at timing, and the Poet gets really good at describing. They don't fight each other anymore.

2. The "Stay in Line" Rule (Overlap Suppression Loss)

Even with two teams, the Boundary Team might still get greedy. They might try to claim the same 10 seconds of video twice, just to be safe.

The Analogy: Imagine a group of security guards all trying to guard the same door. They bump into each other, creating a traffic jam.
The Fix: The authors added a strict rule called Overlap Suppression. If two guards try to claim the same door, the system punishes them. It forces them to spread out and guard different doors.
The Result: The model stops repeating itself. It says, "Okay, Guard A takes 1:00–1:20, and Guard B takes 1:25–1:40." No more skipping records.

3. The "Concept Guide" (The Vocabulary Coach)

Sometimes, the Storyteller Team describes things too vaguely (e.g., "Cooking food"). The authors added a lightweight helper called the Concept Guider.

The Analogy: Imagine a coach whispering key words into the Poet's ear, like "Chicken," "Pan," "Oil," "Fry."
The Result: The Poet doesn't just guess; they use these core concepts to build a much richer, more accurate sentence like "Frying chicken in a pan with oil."

How They Make Sure the Teams Talk (Cross-Task Alignment)

You might ask: "If the Stopwatch and the Poet are totally separate, how do they know they are talking about the same event?"

The Analogy: Imagine the Stopwatch points to a door, and the Poet stands there. How do they know they are a pair?
The Fix: The system uses a Contrastive Alignment loss. It's like a teacher checking the work. If the Stopwatch says "Event A" and the Poet says "Event A," they get a gold star. If the Poet accidentally describes "Event B" while the Stopwatch is pointing at "Event A," they get a red card. This forces the two teams to stay in sync without needing to be the same person.

The Final Result

When the researchers tested this on cooking videos (YouCook2) and general activity videos (ActivityNet), the results were impressive:

Less Repetition: The model stopped saying the same thing over and over.
Better Timing: It knew exactly when an action started and stopped.
Richer Descriptions: The sentences were more detailed and accurate.

In a nutshell: The paper teaches computers to stop trying to be everything at once. Instead, it gives them a Stopwatch, a Poet, and a Coach, and forces them to work together in separate lanes so they don't trip over each other. The result is a video summary that is precise, non-repetitive, and easy to understand.

1. Problem Statement

Dense Video Captioning (DVC) is a multimodal task requiring the model to simultaneously localize multiple events in an untrimmed video (temporal boundaries) and generate natural language descriptions for each event.

While recent query-based frameworks (inspired by DETR) have enabled end-to-end training, they suffer from two primary limitations:

Task Interference: Existing methods use a single set of shared queries to perform both localization and captioning. This forces the model to learn conflicting objectives within the same representation space, leading to ambiguous attention weights that fail to focus on precise boundaries or coherent semantic regions.
Temporal Redundancy: Shared queries often predict overlapping temporal segments for the same event, resulting in redundant captions and imprecise localization. Previous attempts at "query decomposition" (using MLPs to split features) have not fully separated the tasks, leading to similar attention distributions for both roles.

2. Methodology: ROS-DVC

The authors propose ROS-DVC, a framework that decouples the localization and captioning tasks while maintaining their semantic alignment. The architecture consists of three core innovations:

A. Role Specific Queries (RSQ)

Instead of a single shared query set, ROS-DVC initializes two independent sets of learnable queries from separate embedding spaces:

Localization Queries ( $q_{loc}$ ): Specialized to capture broad temporal context to predict precise event boundaries.
Caption Queries ( $q_{cap}$ ): Specialized to focus intensively on key frames to generate detailed semantic descriptions.
Mechanism: By initializing them separately, the model avoids the "one-size-fits-all" optimization conflict. However, to ensure these separated queries still refer to the same event, they are injected together into cross-attention layers, allowing them to ground their distinct roles in the same visual context.

B. Cross-Task Contrastive Alignment (CTCA) Loss

Since the queries are decoupled, there is a risk that a localization query and its corresponding caption query might not align semantically.

Solution: The CTCA loss treats the paired localization and caption queries (matched via the Hungarian algorithm) as positive pairs and all other query combinations as negative pairs.
Goal: This contrastive objective pulls the semantic representation of the caption query closer to its corresponding localization query, ensuring that the localized event boundaries and the generated text describe the same semantic content.

C. Overlap Suppression Loss (OSL)

To address the issue of redundant, overlapping predictions:

Mechanism: The loss penalizes pairs of predicted boundaries that overlap significantly.
Adaptive Weighting ( $\alpha$ ): Unlike standard Non-Maximum Suppression (NMS) which is post-processing, OSL is integrated into training. It uses a ground-truth-aware weighting factor. If a predicted boundary has a high Intersection over Union (IoU) with the Ground Truth (GT), the penalty for overlapping with other predictions is reduced (allowing for multiple valid detections if necessary). If the prediction has low GT alignment, the penalty is increased.
Formula: $L_{OSL} = -\alpha \cdot \log(\beta - P_o)$ , where $P_o$ is the pairwise IoU between predictions, and $\alpha$ is modulated by the prediction's alignment with the GT.

D. Concept Guider

To enhance the semantic richness of captions without relying on external memory banks:

A lightweight MLP head predicts a multi-hot vector of event-level concepts (nouns and verbs) derived from the caption query embeddings.
This auxiliary task forces the caption query to internalize high-level semantic concepts, improving the specificity and context-awareness of the generated sentences.

3. Key Contributions

Task Separation: Introduction of Role Specific Queries that explicitly separate localization and captioning into independent embedding spaces, resolving task interference.
Semantic Alignment: Development of CTCA Loss to enforce semantic consistency between the decoupled localization and caption queries.
Redundancy Reduction: Design of Overlap Suppression Loss (OSL) with adaptive weighting to suppress redundant temporal overlaps during training, leading to precise localization.
Semantic Enhancement: Introduction of the Concept Guider to enrich caption embeddings with concept-level reasoning without external memory.
Performance: Achievement of state-of-the-art (or competitive) results on major benchmarks without relying on large pre-trained language models (like GPT-2) for the caption head.

4. Experimental Results

The method was evaluated on YouCook2 and ActivityNet Captions.

Captioning Performance:
- YouCook2: Achieved the best CIDEr (39.18) and SODA-c (7.06) scores among non-pretrained methods, outperforming MCCL (which uses an external memory bank) and E2DVC.
- ActivityNet: Achieved the best CIDEr (35.04) and SODA-c (6.45) scores, surpassing previous SOTA methods.
- Notably, the model achieved these results using a standard LSTM decoder rather than a heavy LLM, proving the efficacy of the query design.
Localization Performance:
- Achieved the best Recall, Precision, and F1 scores on YouCook2 among non-pretrained methods.
- The model demonstrated a unique balance where Recall and Precision were nearly matched, indicating the Event Counter module accurately predicts the number of events (unlike baselines that often under-predict or over-predict).
Ablation Studies:
- Removing any component (RSQ, CTCA, OSL, or Concept Guider) resulted in performance drops across all metrics.
- The Overlap Suppression Loss was critical for improving F1 scores and reducing redundant captions.
- The Concept Guider significantly boosted CIDEr scores by improving semantic expressiveness.

5. Significance

This paper addresses a fundamental flaw in query-based video understanding: the assumption that a single query can effectively learn both where an event happens and what it is. By enforcing a "Stay in your Lane" philosophy, ROS-DVC demonstrates that:

Specialization is key: Separating tasks into role-specific queries allows for more precise learning of temporal boundaries and semantic details.
Alignment is necessary: Decoupling requires explicit contrastive losses to maintain semantic coherence.
Efficiency: High performance can be achieved without heavy external memory banks or large language models, relying instead on better query initialization and loss design.

The work provides a robust, end-to-end solution for Dense Video Captioning that significantly reduces temporal redundancy and improves the semantic quality of generated captions.