💻 computer science

Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

This paper proposes a data-efficient video pre-training paradigm that freezes a powerful image foundation model to handle spatial features while training only a lightweight recurrent module for temporal reasoning, demonstrating that competitive video understanding can be achieved without the substantial data and compute costs of end-to-end video pre-training.

Original authors: Svetlana Orlova, Niccolò Cavagnero, Gijs Dubbelman

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Svetlana Orlova, Niccolò Cavagnero, Gijs Dubbelman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Don't Reinvent the Wheel

Imagine you want to build a robot that can watch a movie and understand the story. Traditionally, to teach a robot this, you have to show it millions of hours of movies from scratch. It has to learn how to recognize a "face" (spatial) and how a face moves when someone smiles (temporal) all at the same time. This is incredibly expensive, requiring massive amounts of data and computer power.

This paper asks a simple question: Do we really need to teach the robot how to see faces from scratch?

The authors suggest a smarter way: Use a robot that is already an expert at looking at still photos, and just teach it how to watch the movie.

The Analogy: The Expert Photographer and the New Director

Think of the problem like making a film:

The Frozen Image Model (The Expert Photographer):
Imagine you hire a world-famous photographer who has spent years studying millions of photos. They are an absolute master at recognizing objects, lighting, and textures in a single still image. In this paper, the researchers "freeze" this photographer. They don't let them learn anything new; they just use their existing, perfect knowledge of what things look like.
The Temporal Module (The New Director):
Now, you need someone to figure out what is happening in the video. You hire a new, lightweight "Director." This Director's only job is to watch the sequence of photos the Photographer takes and figure out the story (e.g., "The person is walking," "The ball is bouncing").

The Experiment:
The researchers tried building a video AI by taking a "frozen" expert photographer (a pre-trained image model like DINOv3) and attaching a brand-new Director (a temporal module) to it. They trained only the Director on video data, leaving the Photographer completely untouched.

What They Found

1. The Photographer is Already Perfect
They compared their "Frozen Photographer + New Director" team against traditional video AI models that try to learn both seeing and moving from scratch.

Result: The team using the frozen expert photographer actually performed better at recognizing objects and scenes than the models that tried to learn everything from scratch.
Takeaway: The "spatial" knowledge (recognizing what is in the picture) is already so good in modern image models that you don't need to waste time re-learning it for video.

2. The Director Needs Training (But Less Data)
Even with the perfect Photographer, the Director still needed to learn how to connect the dots between frames.

Result: When they trained the Director from scratch using the Photographer's output, the system became very good at understanding motion and action.
Takeaway: You don't need to retrain the whole system. You just need to train the "Director" part.

3. Data Efficiency: Doing More with Less
This is the most exciting part. Because the Photographer already knows everything about the world, the Director doesn't need to see millions of videos to learn.

Result: Their system, using a frozen image model, performed better than a massive video model even when it was trained on less than 25% of the usual data.
Analogy: It's like teaching a new director who already has a script written by a genius. They don't need to watch 1,000 movies to understand the plot; they can figure it out by watching just 250.

The "Streaming" Test

The researchers tested this in a "streaming" mode, meaning the AI had to understand the video as it happened, frame-by-frame, without looking ahead (like watching a live stream).

Result: Even in this strict, real-time test, their "Frozen Photographer + New Director" approach was competitive with, and often beat, the state-of-the-art video models that were trained on billions of video clips.

The Conclusion

The paper concludes that we are likely wasting money and energy by training video models from scratch. Instead, we should:

Freeze a powerful image model (the Photographer).
Train a small, lightweight module to handle the motion (the Director).

This approach saves a huge amount of computing power and data. The authors note that while they haven't fully pre-trained the Director on massive video datasets yet (that's the next step), their initial tests prove that this "decoupled" method is a very promising path forward for building efficient video AI.

In short: Don't teach the AI how to see; just teach it how to watch.

Technical Summary: Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

Problem Statement
Video foundation models currently achieve state-of-the-art (SotA) performance across diverse video understanding tasks but rely on end-to-end pre-training on massive video datasets (often millions to billions of clips). This approach incurs substantial costs in data collection, storage, and computational resources. Conversely, modern image foundation models have reached unprecedented capability, providing powerful spatial representations trained on billions of images. The central research question posed by this work is whether competitive video models can be constructed by reusing these pre-existing spatial representations and restricting pre-training efforts exclusively to temporal reasoning. The authors aim to explore a lightweight training paradigm that freezes a pre-trained image foundation model and trains only a recurrent temporal module to process streaming video, thereby potentially reducing the data and compute requirements for video pre-training.

Methodology
The proposed framework decouples spatial and temporal learning for video understanding, consisting of three primary components:

Frozen Image Encoder: The model utilizes a pre-trained image foundation model (primarily DINOv3, though DINOv2, SigLIP2, and supervised ViTs are also evaluated) as a spatial feature extractor. This encoder remains completely frozen during training, with no gradients flowing through it.
- Multi-depth Feature Extraction: To leverage information distributed across the depth of the frozen encoder, the framework extracts patch tokens from four equally spaced ViT depths (1/4, 1/2, 3/4, and 1). These features are adapted via trainable per-layer MLPs with residual connections and averaged to form the input representation, supplemented by CLS and register tokens.
Recurrent Temporal Module: A lightweight temporal module processes the per-frame features sequentially and causally (without access to future frames). The module maintains a hidden state across frames. The authors evaluate four distinct architectures:
- RVMRNN: A GRU-gated recurrent core with cross-attention, adopted from the RVM baseline.
- Mamba: A selective state space model (SSM) operating independently on spatial tokens.
- MambaMix: Mamba interleaved with spatial self-attention blocks to enable cross-patch interaction within frames.
- GatedMambaMix (GMMix): An extension of MambaMix incorporating a learned gating mechanism to control temporal information flow, analogous to GRU gating.
Attentive Readout: A task-specific head produces predictions. The authors employ a "streaming" protocol where the readout receives only the current frame's output ( $h_t$ ), ensuring that temporal context is strictly accumulated within the recurrent state and not compensated for by the readout.

Key Contributions

Empirical Study of Temporal Learning with Frozen Spatial Representations: The authors demonstrate that effective temporal reasoning emerges across multiple image encoders, temporal architectures, and diverse video tasks (action recognition, object/point tracking, depth estimation, camera pose estimation) even when the image backbone is fixed.
Evidence for Data-Efficient Video Pre-training: The results suggest that modern image foundation models already contain sufficient spatial capacity for video understanding. This supports a future paradigm where only the temporal module requires video pre-training, significantly decoupling spatial and temporal learning.

Experimental Results
The study evaluates the framework on benchmarks including Something-Something v2 (SSv2), Waymo Open, Perception Test (PT), ScanNet, and NuScenes.

Spatial Encoder Competitiveness: Frozen image pre-trained encoders (specifically DINOv3 and DINOv2) consistently match or outperform the video pre-trained RVM encoder. For instance, on SSv2, DINOv3 achieved 66.9% accuracy compared to the video-pre-trained RVM encoder's 62.7%. On Waymo, image encoders surpassed the RVM encoder (85.0–85.5 mIoU vs. 83.4 mIoU).
Temporal Module Performance: When paired with a frozen DINOv3 encoder, all tested temporal architectures significantly outperformed a frozen RVM baseline (which has a frozen video-pre-trained encoder and temporal core). For example, DINOv3 + GMMix achieved a normalized average of 99.4, compared to 77.7 for the frozen RVM-L.
Feature Extraction Strategy: Utilizing multi-depth features from the frozen encoder consistently improved performance across all temporal architectures compared to using only final-layer tokens.
Transferability of Temporal Dynamics: Initializing the temporal module with weights pre-trained on video (from RVM) provided performance gains even when paired with a different frozen encoder (DINOv3). This indicates that learned temporal dynamics are partially encoder-agnostic.
Data Efficiency: Training DINOv3 + GMMix on only 25% of the SSv2 training set yielded performance (56.5%) that surpassed the frozen RVM trained on 100% of the data (46.9%).
Comparison to SotA: At the ViT-L scale, the proposed model (DINOv3 + GMMix) achieved a normalized average of 99.1, exceeding video foundation models like RVM-L (89.3), V-JEPA-L (88.5), and VideoMAE-L (88.9), despite the proposed model having no large-scale video pre-training for its temporal module.

Significance and Claims
The authors position this work as a "work in progress" toward a new pre-training paradigm for recurrent video models. They claim that their findings provide initial empirical evidence that:

Large-scale video pre-training for the spatial encoder is not strictly necessary if strong image foundation models are available.
Strong video understanding is achievable without large-scale video pre-training when the spatial encoder is sufficiently powerful.
A modular approach, where a single frozen image encoder serves as a shared backbone for lightweight, task-specific temporal heads, offers a more data- and compute-efficient alternative to end-to-end video pre-training.

The paper explicitly notes that it does not yet perform large-scale video pre-training of the temporal module itself; rather, it validates the feasibility of the decoupled approach to justify the investment in such future work. The authors conclude that while the current results are promising, future work should include pre-training the temporal module on video data to fully realize the potential of this decoupled paradigm.