Shot-Aware Frame Sampling for Video Understanding

1. Problem Statement

Vision-Language Models (VLMs) have advanced video understanding but face significant constraints when processing long videos due to high computational costs and context window limits. To mitigate this, frame sampling is used to select a subset of frames. However, existing methods suffer from critical limitations:

• Uniform Sampling: Allocates frames by time, leading to oversampling of static shots and undersampling of fast-moving or short-duration events.
• Task-Specific/Query-Dependent Sampling: Methods relying on anomaly scores or query matching require supervision or prior knowledge of the query, limiting generalizability.
• The "Sub-Second" Gap: Many critical events (e.g., anomalies, transient objects) last only a fraction of a second. If these frames are missed during sampling, even powerful VLMs cannot infer the event, leading to reasoning failures.

The core challenge is to design a task-agnostic sampler that, under a strict frame budget, preserves both the global video structure (shots) and sparse, decision-critical within-shot deviations (transient events) without retraining the downstream model.

2. Methodology: InfoShot

The authors propose InfoShot, a shot-aware, model-independent frame sampling framework guided by an information-theoretic objective. The method operates in three main stages:

A. Theoretical Foundation

The authors model a video as a stochastic piecewise-constant process in feature space:

• Shots ($S$): Stable segments where features follow a Gaussian distribution.
• Transient Events ($F$): Sparse deviations within a shot (outliers).
  The objective is to maximize the mutual information $I(C_{\mathcal{K}}; S_{1:T}, F_{1:T})$ between the sampled frames and the latent video structure. This decomposes into:

1. $I(C_{\mathcal{K}}; S_{1:T})$: Preserving shot boundaries.
2. $I(C_{\mathcal{K}}; F_{1:T} | S_{1:T})$: Preserving rare within-shot deviations.

B. Algorithm Pipeline

1. Feature Extraction: Frames are encoded into deep features using a pretrained visual encoder (e.g., ResNet-50).
2. Greedy Shot Segmentation:
   • The video is segmented into semantically consistent shots in the deep feature space.
   • A Windowed Generalized Likelihood Ratio Test (GLRT) is used to detect boundaries by comparing within-window cohesion vs. cross-window similarity.
   • The video is split into $M = \lfloor K/2 \rfloor$ segments, reserving two frames per segment for the next stage.
3. Dual-Keyframe Selection (Per Shot):
   For each detected shot, InfoShot selects two complementary frames:
   • Common Frame ($t_{com}$): Represents the dominant semantics. Selected by maximizing typicality (average similarity to other frames in the shot) while minimizing local volatility.
   • Unique Frame ($t_{uni}$): Captures the largest within-shot deviation. Selected by maximizing a combination of atypicality (low similarity to the shot mean) and local volatility (high dissimilarity to temporal neighbors).

This dual-selection strategy ensures that the sampled set retains the "background" context while explicitly targeting potential anomalies or transient changes.

3. Key Contributions

1. InfoShot Framework: A task-agnostic, plug-and-play sampler that segments videos in deep feature space and selects a "common + unique" pair of frames per shot. It improves semantic coverage without requiring task-specific supervision or retraining.
2. SynFlash Benchmark: A new synthetic Video-QA benchmark designed to evaluate sub-second anomaly detection. It features:
   • Controllable injection patterns: Camouflage (color-matched), High-Saturation (flashing), Curtain (wipe transitions), and Picture-in-Picture (small spatial support).
   • Exact frame-level ground truth for transient anomalies.
3. Comprehensive Evaluation: Demonstrated improvements on real-world anomaly datasets (HIVAU-70k) and general video reasoning benchmarks (Video-MME), showing that preserving transient events does not degrade general reasoning performance.
4. Real-World Deployment: Successfully deployed in TikTok's video moderation pipeline, observing significantly higher recall for evasive, transient harmful content under tight compute constraints.

4. Experimental Results

Experiments were conducted under strict frame budgets (e.g., 0.1–0.5 fps) comparing InfoShot against Uniform sampling, VSUMM (clustering-based), and TransNet (boundary detection).

• SynFlash Performance:
  • InfoShot achieved the highest Frame Recall (84.7% average) compared to baselines (Uniform: 11.8%, VSUMM: 46.5%, TransNet: 46.3%).
  • It significantly outperformed others in Anomaly Retrieval (AR) and Temporal Ordering (TO) tasks, particularly on difficult patterns like High-Saturation and PiP where global similarity measures fail.
  • Feature Distortion: InfoShot showed the lowest feature-space distortion, indicating better retention of information density.
• Real-World Anomaly Detection (HIVAU-70k):
  • Under an extreme 0.1 fps budget, InfoShot achieved the best Event Recall (ER) and Complete Coverage Rate (CR), improving coverage for short-duration violent events.
• General Video Understanding (Video-MME):
  • InfoShot matched or slightly outperformed Uniform sampling on standard Video-QA benchmarks, proving that the strategy for capturing anomalies does not sacrifice general context understanding.
• Ablation Studies:
  • Feature Extractor: Semantic features (ResNet-50, SigLIP) significantly outperformed low-level color histograms, which failed to detect subtle anomalies like camouflage.
  • Budget Robustness: InfoShot maintained superior performance as the frame budget decreased, whereas uniform sampling collapsed in recall.

5. Significance and Impact

• Bottleneck Resolution: The paper identifies frame extraction as a primary failure mode in long-video VLMs and provides a robust solution that operates independently of the downstream model.
• Safety & Moderation: The ability to detect sub-second, evasive harmful content (e.g., flash frames of violence or prohibited items) is critical for large-scale content moderation. InfoShot's deployment in TikTok demonstrates its practical utility in reducing user exposure to unsafe material.
• Efficiency: By intelligently allocating the frame budget based on content structure rather than time, InfoShot allows VLMs to operate effectively under strict latency and compute constraints without sacrificing critical evidence.

In summary, InfoShot bridges the gap between efficient video compression and the need to capture fleeting, critical events, offering a theoretically grounded, high-performance solution for next-generation video understanding systems.