Technical Summary: LookWhen – Fast Video Recognition by Learning When, Where, and What to Compute

1. Problem Statement

Transformers have become the dominant architecture for video recognition, splitting videos into thousands of tokens. However, their computational cost scales superlinearly with the number of tokens, creating a significant bottleneck for processing long videos. While videos contain substantial redundancy (some tokens are redundant, others inferable from surroundings), standard Transformers process all tokens regardless of necessity. Existing adaptive computation methods often merge tokens at intermediate layers or rely on attention maps that may contain artifacts or irrelevant noise, failing to fully exploit sparsity without sacrificing accuracy or incurring high runtime overheads due to non-GPU-friendly operations.

The core challenge is to factorize video recognition into three distinct learning tasks:

1. When: Determining the temporal moments to process.
2. Where: Identifying the spatial locations to process.
3. What: Defining the representation to extract.

This requires isolating input tokens responsible for the output representation quickly (to boost efficiency) while accurately approximating dense video representations using only a sparse subset of tokens.

2. Methodology: The LookWhen Framework

LookWhen introduces a selector–extractor framework specifically designed for video, extending previous image-based work (LookWhere) with critical architectural and pre-training adaptations.

2.1 Architecture

The model consists of two distinct components:

• Shallow Selector: Receives a downscaled video (reduced in time, height, and width). It processes the full input to score all tokens across space-time but remains computationally cheap due to its shallow depth (3 transformer layers) and reduced input resolution. It outputs a selector map to identify the most informative tokens.
• Deep Extractor: Receives only the top-K selected tokens (plus video and frame tokens) from the full-resolution video. It is a deep, expressive model (ViT-B size) that approximates the full-video representation using only these sparse inputs.

2.2 Pre-Training Strategy

A key innovation is the definition of effective supervision targets for both selection and extraction, addressing the lack of suitable "teachers" for video.

A. Selection Targets: Learning "When" and "Where"

• Problem with Attention: Standard attention maps from video teachers (e.g., InternVideo2) often contain artifacts or are noisy, making them poor targets for selection.
• Solution (Top1-Distance): Instead of predicting attention, the selector is trained to predict token uniqueness.
  • An image teacher (DINOv3) with spatially-grounded features is used to compute features for all patches.
  • A "top1-distance" algorithm calculates the cosine similarity between each patch and its nearest neighbor in the feature space.
  • Tokens with the largest distance to their nearest neighbor (i.e., the most unique/isolated) are ranked highest.
  • The selector learns to predict this uniqueness score, effectively selecting the least redundant tokens.

B. Extraction Targets: Learning "What"

• Dual-Teacher Distillation: The extractor is trained to approximate representations from two frozen teachers:
  1. Video Teacher (InternVideo2): Provides a video-level token representing the global context.
  2. Image Teacher (DINOv3): Provides frame-level and patch-level features.
• Time-Normalization: To teach the extractor what changes within a video (crucial for fine-grained tasks), DINOv3's frame-wise class tokens are normalized over time (zero mean, unit variance) before concatenation. This creates a target that emphasizes temporal dynamics rather than static content.
• Loss Functions: The model minimizes a combined loss:
  • $L_{map}$: Binary Cross-Entropy between predicted and target selector maps.
  • $L_{video}$: Mean Squared Error (MSE) between the extractor's video token and the teachers' video tokens.
  • $L_{frame}$ and $L_{patch}$: MSE losses for frame and patch tokens, where DINOv3 features are normalized over space-time.

2.3 Inference

During inference and fine-tuning, the teachers are discarded. The system operates efficiently: the shallow selector scores the downscaled video to pick top-K tokens, and the deep extractor processes only these selected tokens to generate features for downstream tasks.

3. Key Contributions

1. Framework Extension to Video: The authors extend the selector-extractor paradigm to video by adapting the architecture (shallow selector, deep extractor) and pre-training objectives to handle space-time redundancy.
2. Novel Selection Mechanism: They introduce top1-distance, a method to train selectors based on feature-space uniqueness rather than attention maps. This avoids the need for artifact-free attention and effectively identifies redundant tokens.
3. Dual-Teacher Extraction: They propose a pre-training strategy using both a video teacher (for global context) and an image teacher with time-normalized features (for learning temporal changes), enabling the extractor to learn general, fine-grained representations.
4. Efficiency without Hardware Constraints: Unlike token-merging methods that may suffer from Flash Attention incompatibility or high peak memory, LookWhen uses standard deep learning operations, making it highly efficient in practice.

4. Experimental Results

The authors evaluated LookWhen on six datasets: Kinetics-400 (K400), Something-Something-v2 (SSv2), Epic-Kitchens-100 (EK100), Diving48, Jester, and Charades.

• Accuracy vs. FLOPs: LookWhen achieves a superior accuracy-computation trade-off compared to efficient baselines (e.g., VideoMAE, VideoMamba) and upgraded versions of the same size.
  • On K400, LookWhen (70% sparse) achieves 84.6% accuracy with 40% fewer FLOPs than UMT-B800e and 78% fewer FLOPs than VideoMambaPro with comparable or better accuracy.
  • On SSv2, it achieves 72.0% accuracy at 108 GFLOPs, outperforming VideoMAEv2 (71.2% at 180 GFLOPs).
• Pareto Dominance: LookWhen Pareto-dominates (better accuracy and lower cost) in 9 out of 12 evaluation cases (6 tasks × 2 settings: linear probing and fine-tuning).
• Real-World Throughput: Measured on an NVIDIA L40S GPU, LookWhen is 6.7× faster than the dense InternVideo2-B (IV2) at equal accuracy. This gap is larger than theoretical FLOP savings suggest because IV2+merging baselines suffer from Flash Attention incompatibilities and require batch size 1 or masking.
• Ablation Studies:
  • Selection: Training on "top1-dist" (uniqueness) significantly outperforms training on attention maps (IV2 or DINOv3) or random selection.
  • Extraction: Using both video and image teachers, along with time-normalization of DINOv3 features, yields the best performance, particularly on fine-grained tasks like Diving48 and Jester.

5. Significance and Claims

The paper claims that LookWhen successfully addresses the computational inefficiency of video Transformers by learning to compute only when, where, and what is necessary.

• Efficiency: It demonstrates that sparsity can be achieved without the runtime penalties of token merging, offering a practical path to faster video recognition.
• Generalization: The framework learns general representations that are effective for both linear probing (feature extraction) and fine-tuning, often surpassing its own dense teacher (InternVideo2) in specific scenarios.
• Scalability: By decoupling the selection of tokens from the deep feature extraction, the method allows for aggressive sparsity (up to 95%) while maintaining high accuracy.

The authors note limitations, such as the current restriction to ViT-Base models due to compute constraints and the reliance on teachers with positionally-grounded representations. However, they posit that as foundation models evolve to provide better positionally-grounded outputs, the LookWhen framework will become even more effective. Future work is suggested for long videos, multi-view processing, and hyperspectral data.