💻 computer science

Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning

The proposed STaRC framework improves retrieval-augmented Dense Video Captioning by supervising frame-level saliency through a highlight detection module trained on existing ground truth, utilizing these saliency scores to guide temporal segmentation and caption generation for state-of-the-art performance on YouCook2 and ViTT benchmarks.

Original authors: Seung hee Choi, MinJu Jeon, Hyunwoo Oh, Jihwan Lee, Dong-Jin Kim

Published 2026-03-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seung hee Choi, MinJu Jeon, Hyunwoo Oh, Jihwan Lee, Dong-Jin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a story about a long, unedited home video of someone cooking a complex meal. The video is hours long, but the actual "story" only happens in short bursts: chopping onions, frying garlic, simmering sauce. The rest is just the chef walking around, grabbing ingredients, or staring at the stove.

The Problem with Old Methods
Previous AI models trying to describe these videos were like a clumsy editor with a pair of scissors. They would try to cut the video into scenes based on how "different" the pictures looked.

The Mistake: They might cut right in the middle of chopping an onion because the camera moved slightly, or they might group two unrelated scenes together (like "chopping onions" and "washing dishes") because the lighting looked similar.
The Result: When the AI tried to find a description for a segment, it would ask a library, "What's happening here?" But because the segment was a messy mix of two different actions, the library would return a confusing or wrong answer. The AI would then write a caption like, "The chef washes dishes while chopping onions," which makes no sense.

The Solution: STaRC (Follow the Saliency)
The authors of this paper, STaRC, decided to stop guessing where the scenes start and end. Instead, they taught the AI to look for "Highligts" or "Saliency."

Think of Saliency like a flashlight in a dark room.

In a cooking video, the "flashlight" shines brightest on the hands actually doing the cooking (chopping, stirring, pouring).
The "flashlight" is dim or off when the chef is just walking to the fridge or standing still.

Here is how STaRC uses this flashlight in three simple steps:

1. Learning to See the Flashlight (Supervised Saliency)

The AI is trained using the "answer key" from the video dataset (the ground truth). It learns that the moments marked as "events" in the answer key are the bright spots.

The Analogy: Imagine a student studying for a test. Instead of just memorizing the whole book, the teacher highlights the most important sentences. The student (the AI) learns to recognize that highlighted parts are the ones that matter.
The Magic: The AI learns to predict these highlights automatically, frame-by-frame, without needing anyone to draw new boxes around the video.

2. Cutting the Video with the Flashlight (Saliency-Guided Segmentation)

Once the AI knows where the "bright spots" are, it uses them to cut the video.

Old Way: Cut the video every 5 seconds or when the picture changes slightly.
STaRC Way: Cut the video exactly where the flashlight turns on and off.
The Result: The video segments are now perfect "scenes." One segment is just "chopping onions." Another is just "frying the pan." Because the segments are clean, when the AI asks the library for a description, it gets a perfect match.

3. Whispering the Flashlight to the Writer (Saliency Prompts)

This is the cleverest part. Even after cutting the video, the AI needs to write the sentence.

The Analogy: Imagine a writer trying to describe a scene, but they are distracted by background noise. STaRC gives the writer a note (a prompt) that says: "Hey, focus on the bright part! Ignore the dark parts!"
The AI injects these "flashlight scores" directly into the writing process. It tells the decoder: "This specific frame is the most important part of the story right now, so make sure your sentence describes that."

Why This Matters

The paper shows that when you align the video cuts with the actual "important moments" (the highlights), everything gets better:

Retrieval: The AI finds the right reference sentences because the video clips are clean.
Writing: The AI writes better sentences because it knows exactly which moment to describe.
Accuracy: The start and end times of the events match the real world much better.

In a Nutshell:
Previous methods were like trying to describe a movie by looking at random snapshots. STaRC is like having a director who knows exactly which scenes are the "action" and which are just "boring filler," cutting the film perfectly, and then handing the scriptwriter a highlight reel so they can write the perfect story.

The result? A system that can watch a 30-minute cooking video and tell you exactly what happened, step-by-step, with the precision of a human editor, all without needing extra human help to teach it where to cut.

1. Problem Statement

Dense Video Captioning (DVC) aims to detect multiple events in long, untrimmed videos and generate natural language descriptions for each. While retrieval-augmented methods have improved performance by fetching relevant captions from external datastores, they suffer from a critical bottleneck: temporal misalignment.

The Limitation: Existing methods (e.g., Sali4Vid, HiCM2) rely on heuristic strategies (like uniform sampling or similarity-based clustering) to segment videos. These strategies often fail to align with true event boundaries defined in the ground truth.
The Consequence: Misaligned segments lead to the retrieval of irrelevant captions. Consequently, the decoder receives noisy context, resulting in inaccurate event localization and poor caption generation.
The Gap: Current approaches treat saliency (frame importance) as an implicit or heuristic signal. There is a lack of frameworks that explicitly supervise frame-level saliency using ground truth event annotations to guide both segmentation and generation.

2. Methodology: STaRC Framework

The authors propose STaRC (Saliency Training for Retrieval and Captioning), a unified framework that learns frame-level saliency scores directly from ground truth event annotations and uses them as a shared signal for both retrieval and captioning.

A. Supervised Saliency Learning

Unlike previous methods that derive saliency heuristically, STaRC trains a Highlight Detection Module using binary labels derived directly from existing DVC annotations (no extra annotation cost).

Feature Refinement: Input video frames are processed by a Sliding-Window Self-Attention (SWSA) module. This aggregates local temporal context without learnable parameters, refining features before saliency prediction.
Training Objective: The module predicts a saliency score ( $P_s$ ) for each frame. Frames within annotated event boundaries are labeled as "highlights" (1), and others as "non-highlights" (0). The model is trained using a list-wise softmax objective to maximize the probability of highlight frames.

B. Unified Saliency-Based Design

The learned saliency scores ( $P_s$ ) are utilized in two complementary ways:

1. Saliency-Guided Segmentation and Retrieval (SGSR)

Mechanism: SGSR replaces heuristic clustering with Optimal Transport (OT) based clustering (inspired by ASOT).
Saliency Integration:
- Unbalanced OT: The saliency scores act as a soft constraint on the frame marginal distribution. This forces the transport plan to allocate more mass to salient frames, ensuring segments are centered on important events.
- Cost Bias: Saliency is added as a bias term in the cost matrix, prioritizing salient frames during assignment.
Retrieval: Segments are ranked by a score combining semantic alignment and temporal stability. The top- $k$ segments are used to retrieve captions. Crucially, the segment representation is a saliency-weighted average of frame embeddings, emphasizing informative frames.

2. Saliency Prompt (SaliP)

Mechanism: Instead of implicitly masking features, STaRC explicitly injects the frame-level saliency scores into the decoder as prompts.
Function: The saliency scores are projected via a learnable layer and concatenated with video features, retrieved captions, and text tokens. This acts as a "temporal grounding" signal, explicitly guiding the decoder to focus on semantically important frames during caption generation.

3. Key Contributions

Supervised Saliency Learning: The first framework to explicitly learn frame-level saliency from ground truth event boundaries for DVC, eliminating the need for additional annotation costs.
Unified Design: A novel architecture where the same learned saliency signal drives both segmentation (via SGSR) and caption generation (via SaliP), ensuring consistent temporal grounding.
OT-Based Segmentation: Introduction of an unbalanced Optimal Transport approach that uses saliency priors to form event-aligned segments, outperforming standard clustering.
Explicit Prompting: The use of Saliency Prompts to directly guide the decoder, replacing implicit attention mechanisms with explicit temporal cues.

4. Experimental Results

The method was evaluated on two benchmarks: YouCook2 and ViTT.

State-of-the-Art Performance:
- YouCook2: Achieved a CIDEr score of 80.53 (vs. 75.80 for Sali4Vid) and an F1 localization score of 34.34.
- ViTT: Achieved a CIDEr score of 56.04 and the highest precision (49.72) for localization.
Correlation Analysis: The paper demonstrates a strong positive correlation between segment quality metrics (Recall@0.5, Mean IoU) and captioning performance (CIDEr, METEOR). Better segmentation directly leads to better retrieval and generation.
Ablation Studies:
- Removing SGSR or SaliP individually caused significant performance drops, confirming their complementary nature.
- Saliency-weighted pooling in retrieval outperformed uniform averaging.
- Saliency Prompt corruption (replacing prompts with noise or zeros) severely degraded performance, proving the decoder relies on accurate saliency cues.
- The SWSA module provided consistent gains by refining temporal features.

5. Significance

Paradigm Shift: STaRC shifts the paradigm from heuristic segmentation to supervised, event-aligned segmentation. It proves that leveraging ground truth boundaries to learn saliency is superior to unsupervised or heuristic approaches.
Retrieval Quality: By ensuring video segments align with true events, the retrieval module fetches highly relevant captions, solving the "noise" problem in retrieval-augmented DVC.
Decoder Guidance: The explicit injection of saliency as a prompt offers a new mechanism for temporal grounding in generative models, allowing the model to "follow the saliency" of the video content.
Efficiency: The method achieves these gains without requiring new datasets or expensive pre-training, relying instead on a unified design that maximizes the utility of existing annotations.

In conclusion, STaRC demonstrates that accurate temporal segmentation is the prerequisite for high-quality dense video captioning, and this accuracy can be achieved by explicitly supervising frame-level saliency to guide both retrieval and generation.