💻 computer science

LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning

This paper introduces the Learnable Frame Selector (LFS), a method that leverages caption feedback from frozen video-LLMs to dynamically select temporally diverse and event-relevant frames for improved video captioning, while also proposing the human-consistent ICH-CC benchmark to better evaluate detailed video understanding.

Original authors: Lianying Chao, Linfeng Yin, Peiyu Ren, Yifan Jiang, Qiaoyu Ren, Dingcheng Shan, Jing-cheng Pang, Sijie Wu, Xubin Li, Kai Zhang, Xin Chen

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lianying Chao, Linfeng Yin, Peiyu Ren, Yifan Jiang, Qiaoyu Ren, Dingcheng Shan, Jing-cheng Pang, Sijie Wu, Xubin Li, Kai Zhang, Xin Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a two-hour movie to a friend, but you only have time to show them 16 still photos from the film.

If you just pick 16 photos evenly spaced out (one every 7 minutes), you might miss the most exciting parts. You might catch a photo of the hero sitting still, then skip to them sitting still again, and completely miss the 30-second scene where they actually fight the dragon. This is the problem current AI video describers face: they take "evenly spaced" snapshots, which often miss the important action.

This paper introduces a new tool called LFS (Learnable Frame Selector) that acts like a smart film editor. Instead of picking photos randomly or evenly, LFS learns to pick the best 16 photos that tell the whole story.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Boring Snapshot" Trap

Current AI models usually pick frames (photos) from a video using uniform sampling.

The Analogy: Imagine reading a book by only reading page 1, page 50, page 100, and page 150. You might get the general idea, but you'll miss the plot twists, the funny jokes, and the emotional moments that happen in between.
The Result: The AI misses short but critical actions (like a chef chopping an onion quickly) because those moments are squeezed between long periods of nothing happening.

2. The Solution: LFS (The Smart Editor)

LFS is a small, smart module that sits before the main AI (the "brain" that writes the description). Its job is to decide which 16 frames to show the brain. It does this in three clever ways:

It Knows What's Important (Event Awareness):
LFS looks at the video and asks, "Where is the action happening?" It gives high scores to frames where things are changing (like a car turning or a person speaking) and low scores to boring, static frames.
- Metaphor: It's like a spotlight that automatically shines on the actor when they start speaking, rather than shining on the empty stage.
It Spreads the Shots Out (Temporal Diversity):
Just picking the "most important" frames might result in picking 16 frames all from the same 10-second explosion. LFS uses a stratified strategy. It divides the video into 16 time slots and forces itself to pick exactly one "best" frame from each slot.
- Metaphor: It's like a teacher grading a student's essay. Instead of only reading the most exciting paragraph, the teacher ensures they read a bit from the beginning, the middle, and the end to get the full picture.
It Learns from the "Teacher" (Caption Feedback):
LFS doesn't just guess; it learns by looking at the final result. It uses a powerful, frozen AI (a "teacher") to generate a description. If the teacher writes a bad description because LFS picked the wrong frames, LFS gets a "thumbs down" and adjusts its strategy.
- Metaphor: It's like a sous-chef tasting the soup. If the soup tastes bad, the sous-chef realizes, "Oh, I picked the wrong vegetables," and changes what they grab next time.

3. The New Test: ICH-CC

The authors realized that existing tests for video AI weren't very good at measuring if an AI truly understood a video like a human does. So, they built a new test called ICH-CC.

What is it? A collection of videos about Intangible Cultural Heritage Chinese Cuisine (traditional cooking).
Why is it special? Humans wrote the descriptions and questions. It's designed to see if the AI can describe the subtle steps of cooking (like "adding rice wine" or "stirring") just as a human would.
The Result: When LFS was used, the AI got much better at passing this human-like test, proving it could actually "see" the cooking process better than before.

4. The Results: Better Stories, Better Answers

The paper tested LFS on several different video AI models and benchmarks:

Detailed Descriptions: The AI started writing much richer, more accurate descriptions of what was happening in the video.
Video Question Answering: Because the AI had a better "story" to read, it got better at answering questions about the video (even without seeing the raw video again).
Efficiency: It did all this without needing more computing power; it just picked the right 16 frames instead of the wrong 16.

Summary

Think of LFS as a smart camera operator for an AI. Instead of taking a photo every 10 seconds regardless of what's happening, LFS waits for the action, ensures it captures the beginning, middle, and end, and hands the best 16 photos to the AI writer. The result is a video description that is detailed, accurate, and actually makes sense to a human reader.

Technical Summary: Learnable Frame Selector (LFS) for Event-Aware and Temporally Diverse Video Captioning

Problem Statement
Current video captioning models, particularly those leveraging Large Language Models (LLMs), typically convert video frames into visual tokens to generate descriptions. However, encoding all frames is computationally prohibitive. Consequently, uniform sampling has become the default strategy to select a fixed number of frames. While this ensures temporal coverage, it fails to account for the uneven distribution of informative events within a video. Uniform sampling often ignores short but critical actions or clusters redundant frames, leading to suboptimal detailed captions. Existing query-driven selection methods (e.g., for Video QA) are unsuitable for query-agnostic tasks like detailed captioning, and current Top-K or reinforcement learning-based approaches often struggle to balance event relevance with temporal diversity or are difficult to integrate with frozen video-LLMs. Furthermore, the authors identify a significant gap between existing benchmark evaluations and human cognitive understanding of video content.

Methodology: Learnable Frame Selector (LFS)
The paper proposes LFS, a lightweight, plug-and-play module designed to select a small set of temporally diverse and event-relevant frames. LFS operates without modifying the parameters of the downstream language model. Its architecture and training process consist of three core components:

Temporal Importance Modeling (TSNet):
Given frame embeddings from a frozen vision encoder (Long-CLIP), LFS employs a Temporal Scoring Network (TSNet) to predict a continuous temporal importance field. The TSNet uses temporal convolutions to capture local transitions and a global summary with gated modulation to adaptively rescale features. This allows the model to score frame importance in a query-agnostic manner.
Stratified Top-K Selection:
To ensure temporal diversity and prevent clustering, LFS employs a stratified Top-K strategy during inference. The video timeline is divided into $K$ equal segments, and the frame with the highest importance score is selected from each segment. This guarantees full temporal coverage. During training, a soft probability distribution is used to allow gradient propagation, while the hard stratified selection is applied only at inference.
Caption-Guided Optimization:
Unlike methods that optimize for proxy objectives, LFS is trained using caption feedback from a frozen video-LLM (specifically Qwen3-VL-8B). The selector learns to choose frames that directly maximize the quality of the generated caption. The training objective includes:
- Relative Caption Loss: Aligns frame selection with caption quality by comparing against a uniform baseline.
- $\ell_1$ Regularization: Encourages compact frame weighting on representative frames.
- Entropy Regularization: Promotes exploration during early training stages.

Key Contributions

LFS Framework: The proposal of a learnable frame selector that explicitly balances event awareness and temporal diversity through stratified selection and caption-guided supervision. It is designed to be easily integrated into existing pipelines without fine-tuning the LLM.
ICH-CC Benchmark: The introduction of ICH-CC (Intangible Cultural Heritage Cuisine Captioning), a new benchmark constructed to better reflect human cognitive understanding. It features carefully designed questions and human-authored captions for Chinese and English cuisine videos, addressing the misalignment found in existing benchmarks like VDC and Dream-1K.
Empirical Validation: Extensive experiments demonstrating that LFS consistently improves detailed video captioning and downstream zero-shot video question answering across multiple backbones and benchmarks.

Experimental Results
The authors evaluated LFS on three benchmarks: ICH-CC, VDC, and Dream-1K, using various video-LLM backbones (e.g., Qwen3-VL-8B, AuroraCap-7B, Tarsier2-7B).

ICH-CC: LFS significantly improved performance. For Qwen3-VL-8B, it boosted accuracy by 4.47% on the English subset and 3.14% on the Chinese subset. Qualitative analysis showed LFS successfully captured short, critical steps (e.g., boiling, garnishing) that uniform sampling missed.
VDC: LFS provided consistent gains across categories, with the most notable improvement in the "Detailed" category (e.g., +2.02% accuracy for Qwen3-VL-8B).
Dream-1K: As a short-video robustness test, LFS maintained performance comparable to uniform sampling (F1 score of 0.40), indicating it does not degrade performance on short clips where uniform sampling is already effective.
Zero-Shot Video QA: Enhanced captions generated via LFS led to improved performance on downstream video QA tasks (e.g., +2.1% on MVBench), confirming that better frame selection yields more informative descriptions for reasoning.

Significance and Claims
The paper claims that frame selection is a critical bottleneck for detailed video captioning. By explicitly modeling event relevance and temporal diversity, LFS provides an effective and easy-to-integrate solution that improves captioning quality without requiring the retraining of large language models. The authors emphasize that their approach bridges the gap between computational constraints and the need for fine-grained, temporally coherent descriptions, ultimately supporting downstream tasks like video retrieval and multimodal QA. The introduction of ICH-CC is also highlighted as a necessary step to align benchmark evaluation with human cognition.