🤖 AI

Frozen Forecasting: A Unified Evaluation

This paper introduces a unified evaluation framework for assessing the forecasting capabilities of frozen vision backbones by training latent diffusion models to predict future features in representation space, revealing that video-pretrained models outperform image-based ones and that forecasting ability strongly correlates with perceptual quality while language supervision offers inconsistent benefits.

Original authors: Jacob C Walker, Pedro Vélez, Luisa Polania Cabrera, Guangyao Zhou, Sayna Ebrahimi, Rishabh Kabra, Carl Doersch, Maks Ovsjanikov, João Carreira, Shiry Ginosar

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jacob C Walker, Pedro Vélez, Luisa Polania Cabrera, Guangyao Zhou, Sayna Ebrahimi, Rishabh Kabra, Carl Doersch, Maks Ovsjanikov, João Carreira, Shiry Ginosar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie. A standard computer vision model is like a photographer who is incredibly good at taking a single, perfect snapshot of the present moment. It can tell you exactly what a car looks like, how deep the road is, or where a person is standing right now.

But what if you want to know what happens next? What if you need to predict where that car will be in five seconds, or how a person will move their hand? This is forecasting.

The paper "Frozen Forecasting" tackles a big problem: How do we test if a computer model is good at predicting the future?

The future is messy. It's not just one single outcome; it's a cloud of possibilities. If you throw a ball, it might land here, or there, or bounce off a wall. A good predictor needs to understand this uncertainty, not just guess the "average" outcome.

Here is the paper's story, broken down with some everyday analogies:

1. The Problem: The "Frozen" Brain

The researchers wanted to test different "brains" (AI models) to see which one is best at predicting the future.

The Models: They took famous AI models that were already trained to see and understand the present (like recognizing objects in a photo). These models are "frozen," meaning they can't be re-taught; they are fixed.
The Challenge: These models are great at describing what is, but they haven't been explicitly taught what will be.

2. The Solution: The "Crystal Ball" Diffusion

Instead of trying to teach the frozen brain new tricks, the researchers built a Crystal Ball (a Diffusion Model) that sits on top of it.

How it works: The frozen brain looks at the past few seconds of a video and turns them into a "secret code" (latent representation). The Crystal Ball then takes that code and imagines what the secret code for the future seconds might look like.
The Magic: Because the future is uncertain, the Crystal Ball doesn't just spit out one prediction. It generates many different possible futures (like rolling dice to see all the possible ways a ball could land).

3. The Test: The "Translator" (Readout Heads)

Once the Crystal Ball guesses the future "secret code," the researchers need to translate that back into something humans can understand to check if it's right.

They use a Translator (a lightweight "readout head") to turn the secret code into:
- Pixels: What the next video frames look like.
- Depth: How far away objects are.
- Point Tracks: Where specific dots on objects are moving.
- Bounding Boxes: Where cars or people will be.

4. The Big Discoveries (The "Aha!" Moments)

🏆 Video Models > Image Models

The Analogy: Imagine trying to predict the weather. A model trained only on static photos of clouds (Image Models) is okay, but a model trained on time-lapse videos of clouds moving (Video Models) is much better.
The Result: Models that learned by watching videos consistently beat models that only learned by looking at single pictures. You need to see motion to predict motion.

🎨 The "Artist" vs. The "Architect"

The Analogy: Some models are trained to be Architects (Masked models like VideoMAE). They are taught to look at a picture, cover up a part of it, and guess what's missing. This makes them great at understanding structure.
Other models are Artists (Synthesis models like WALT). They are trained to paint the next frame of a video from scratch.
The Result: The Artists were surprisingly good at predicting the future, often beating the Architects, especially at low-level tasks like predicting exact pixel colors. However, for high-level logic (like tracking a specific car), the Architects sometimes held their own.

🗣️ Language Doesn't Help

The Analogy: You might think that teaching a model to "read" subtitles (Language Supervision) would help it predict the future. "The car is speeding up" sounds like a good clue.
The Result: Surprisingly, adding language didn't make the models better at forecasting. Seeing the motion is more important than reading about it.

🎲 The "Average" Trap

The Analogy: If you ask a model to predict the weather, and it says "It will be 70°F" (the average), it might be wrong every single time if the real weather swings between 50°F and 90°F.
The Result: The researchers found that simple "average" predictions fail. The best models are the ones that generate a diverse set of possibilities (a distribution) that matches the real variety of the future.

5. The Takeaway

This paper built a universal test to see which AI models are the best "fortune tellers."

They found that to predict the future, an AI needs to have seen the flow of time (video training), not just static moments. While some models are great at the "big picture" (objects), others are better at the "fine details" (pixels). Most importantly, the future is rarely a single path; it's a web of possibilities, and the best models are the ones that can imagine all of them.

In short: If you want an AI that can drive a car or play sports, don't just give it a photo album. Give it a movie.

1. Problem Statement

Forecasting future events is a critical capability for general-purpose AI systems, yet evaluating the "correctness" of a forecast is inherently difficult due to the stochastic nature of the future. Unlike perception tasks (which analyze past/present frames), forecasting requires modeling uncertainty and multiple plausible outcomes.

Current evaluation methods for vision models often focus on:

Perception: Assessing how well a model understands static or past frames (e.g., classification, detection).
Deterministic Prediction: Many existing forecasting approaches predict a single future state, failing to capture the multimodal distribution of possible futures.
Task-Specificity: Previous works often train models from scratch for specific forecasting tasks, making it difficult to compare the intrinsic forecasting capabilities of different pre-trained backbones.

The authors aim to bridge this gap by creating a unified evaluation framework to assess the forecasting capabilities of frozen (pre-trained, non-fine-tuned) vision backbones across diverse abstraction levels (from pixels to object bounding boxes) without modifying the backbone itself.

2. Methodology

The proposed framework, "Frozen Forecasting," decouples the representation learning (the frozen backbone) from the forecasting mechanism. It operates in two main stages:

A. Architecture Overview

Frozen Backbone: A pre-trained vision model (image or video) encodes observed frames into latent representations. The backbone weights remain frozen.
Latent Diffusion Forecaster: A lightweight diffusion model is trained to predict future latent trajectories directly in the representation space of the frozen backbone.
- Input: Latent encodings of $t=4$ past frames.
- Output: Latent encodings for $T=12$ future frames (total clip length 16).
- Mechanism: A conditional denoising diffusion model (using DDIM and a cosine schedule) generates diverse future trajectories, capturing the stochastic nature of the future.
Task-Specific Readout Heads: Lightweight, attention-based heads are trained to decode the frozen representations (both observed and forecasted) into specific downstream tasks.
- Tasks: Pixel reconstruction, Depth estimation, Point tracks, and Object Bounding Boxes.
- Training: Readout heads are trained only on observed frames using standard supervised losses. They remain fixed during the forecasting evaluation.

B. Evaluation Metrics

To address the stochastic nature of forecasting, the authors propose a dual-metric evaluation strategy:

Per-Example Metrics: Measures accuracy on a per-sample basis (e.g., PSNR for pixels, IoU for boxes, Jaccard for tracks). They report statistics (Mean, Best-of-N, Min/Max) over 10 samples to capture sample quality.
Distributional Metrics:
- Fréchet Distance (FD): Computes the distance between the distribution of forecasted trajectories and the ground truth distribution in the output task space (e.g., 2D coordinates for tracks, downsampled patches for pixels). This evaluates realism and diversity.
- Variance: Measures the variance of the forecasted trajectories to ensure the model captures the inherent diversity of the future, rather than collapsing to a static average.

3. Key Contributions

Unified Framework: Introduced a standardized pipeline to evaluate forecasting across 9 diverse vision models (including DINOv2, SigLIP, VideoMAE, WALT) and 4 abstraction levels using a single diffusion-based engine.
Stochastic Evaluation: Moved beyond deterministic regression by utilizing diffusion models to generate diverse future trajectories and employing Fréchet Distance to evaluate distributional alignment.
Frozen Backbone Analysis: Demonstrated that pre-trained perception models can be effectively repurposed for forecasting without fine-tuning the backbone, isolating the forecasting capacity of the learned representations.
Comprehensive Benchmark: Established strong baselines and a new benchmark for video forecasting that spans low-level (pixels) to high-level (object motion) tasks.

4. Key Results

The study evaluated 9 models (image and video pre-trained, with and without language supervision) on 4 tasks. Key findings include:

Correlation with Perception: Forecasting performance generally correlates with perception performance, but this relationship breaks down for the strongest models. The best model for perception is not always the best for forecasting.
Synthesis vs. Masked Models:
- WALT (a video synthesis model trained with frame prediction objectives) significantly outperforms masked models (like VideoMAE) on low-level tasks (pixels, depth).
- However, for mid-level tasks (point tracks, bounding boxes), WALT's performance is comparable to or slightly exceeds masked models, suggesting that explicit frame prediction training yields robust low-level dynamics but not necessarily superior high-level semantic forecasting.
Video vs. Image Backbones: Models pre-trained exclusively on video consistently outperform those pre-trained only on static images (e.g., DINOv2, SigLIP). This highlights the necessity of temporal context for learning generalizable video representations.
Language Supervision: Models with vision-language contrastive training (e.g., SigLIP, VideoPrism) do not consistently improve forecasting performance compared to purely visual models.
Distributional Alignment: While some models achieve high per-example accuracy, they often fail to match the variance of the ground truth distribution. WALT showed strong distributional alignment (low Fréchet Distance) on pixels, while DINOv2 performed surprisingly well on depth forecasting.

5. Significance and Implications

Redefining Evaluation: The paper argues that evaluating forecasting requires distribution-aware metrics (like Fréchet Distance) rather than simple point-wise error, as the future is inherently multimodal.
Representation Learning: The results suggest that temporal supervision during pre-training is more critical for forecasting than language supervision or sheer parameter count.
Generality of Frozen Models: The framework proves that frozen video representations contain rich implicit dynamics that can be unlocked by a separate, lightweight forecasting module, offering a new paradigm for building general-purpose predictive systems without retraining massive backbones.
Limitations: The study notes that current models struggle to capture the full variance of complex motions and that the framework is currently limited to short-horizon (16-frame) contexts.

In conclusion, "Frozen Forecasting" provides a rigorous, unified methodology to benchmark how well vision models anticipate the future, revealing that while perception and forecasting are related, they require distinct representational qualities, with video-pretrained models currently holding the advantage.