💻 computer science

Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Re2Pix is a hierarchical video prediction framework that enhances temporal semantic consistency and visual fidelity in complex dynamic environments by first forecasting scene structure in a frozen vision model's feature space and then using these predicted representations to guide a latent diffusion model for photorealistic frame synthesis.

Original authors: Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict what a busy city street will look like five seconds from now. You see cars moving, pedestrians crossing, and the sun shifting.

Most current AI models try to do this by looking at the raw pixels (the tiny colored dots that make up the image) and guessing the next set of pixels. It's like trying to predict the future of a movie by guessing the exact color of every single pixel in the next frame. This is incredibly hard, often leading to blurry, weird, or "glitchy" results where a car might suddenly turn into a tree or a person might flicker in and out of existence.

Re2Pix is a new approach that solves this by breaking the problem into two distinct steps, like a director and a special effects artist working together.

The Two-Stage Process

Stage 1: The "Director" (Semantic Prediction)

First, the AI acts like a movie director who doesn't care about the lighting or the texture of the actors' clothes yet. The director only cares about the story and the layout.

What it does: It looks at the current scene and predicts the "skeleton" of the future. Where will the cars be? Where will the pedestrians walk? Will a building block the view?
The Analogy: Think of this as drawing a rough sketch or a blueprint of the future scene. It uses a powerful, pre-trained "brain" (called a Vision Foundation Model) that understands the world deeply. It predicts the meaning of the scene, not the pretty details.

Stage 2: The "Special Effects Artist" (Visual Synthesis)

Once the director has the sketch, they hand it to the special effects artist.

What it does: The artist takes that rough sketch and fills in all the beautiful details: the shiny paint on the cars, the shadows, the reflections, and the realistic textures.
The Analogy: This is like taking a black-and-white storyboard and turning it into a photorealistic movie. Because the artist already knows exactly where everything is supposed to be (thanks to the director's sketch), they don't have to guess. They just focus on making it look real.

The Big Problem: The "Teacher vs. Student" Mismatch

There was a tricky problem with this two-step idea.

During Training: The "Special Effects Artist" was taught using perfect blueprints drawn by a human teacher.
During Real Use (Inference): The "Director" has to draw the blueprint themselves. Since the Director is an AI, their sketch might be slightly messy or have small errors.

If you train the Artist to only work with perfect blueprints, they will panic and produce blurry, messy movies when given a slightly imperfect sketch from the Director. This is called the Train-Test Mismatch.

The Solution: Training with "Imperfect" Blueprints

Re2Pix introduces two clever tricks to fix this, ensuring the Artist can handle a messy sketch:

Nested Dropout (The "Blurry Sketch" Trick):
During training, the AI randomly takes the perfect blueprint and erases some of the fine details, leaving only the big, important shapes (like the outline of a car, but not the wheels). The Artist has to learn to make a great movie even when the blueprint is incomplete. This makes the Artist robust and ready for the Director's imperfect predictions.
Mixed Supervision (The "Real-World Practice"):
The AI is trained 90% of the time with perfect blueprints, but 10% of the time, it is forced to use the Director's actual (imperfect) predictions. This is like giving the Artist a "practice exam" that looks exactly like the real test. It teaches the Artist not to freak out when the sketch isn't perfect.

Why This Matters

The results are impressive, especially for self-driving cars:

Faster Training: Because the AI doesn't have to guess the layout and the details at the same time, it learns much faster (up to 7x faster for visuals and 14x faster for understanding the scene).
Better Consistency: Objects don't disappear or morph into other things. A car stays a car, and a person stays a person.
Realism: The final video looks sharp and photorealistic because the "Special Effects Artist" has a solid plan to follow.

In summary: Re2Pix stops trying to guess the future pixel-by-pixel. Instead, it first figures out the story of the future, and then paints the picture based on that story. By training the painter to handle rough drafts, it creates a system that is faster, smarter, and much more reliable for things like autonomous driving.

1. Problem Statement

Video prediction, particularly in complex dynamic environments like autonomous driving, requires models to simultaneously master two difficult tasks:

High-level semantics: Understanding scene structure, object interactions, and temporal dynamics (e.g., vehicle motion, occlusions).
Low-level appearance: Generating photorealistic, high-fidelity visual details.

Current Limitations:
Most state-of-the-art approaches use an end-to-end paradigm, predicting future frames directly in a latent space (often using Diffusion Models). This creates a fundamental limitation: semantic structure and fine-grained visual details are deeply entangled within the same latent representation. Consequently, models struggle with:

Temporal semantic inconsistencies: Object identity drift, structural degradation, and flickering.
Training inefficiency: The model must learn to infer dynamics and render appearance simultaneously, slowing convergence and increasing data requirements.
Lack of control: It is difficult to reason about or control specific components (dynamics vs. appearance) independently.

2. Methodology: Re2Pix Framework

The authors propose Re2Pix, a hierarchical framework that explicitly decouples video prediction into two distinct stages: Semantic Representation Prediction and Semantics-Guided Visual Synthesis.

Stage 1: Semantic Feature Prediction

Goal: Forecast the future structure of the scene in a feature space rather than pixel space.
Mechanism:
- A frozen Vision Foundation Model (VFM) encoder (specifically DINOv2) extracts semantic feature maps ( $h_t$ ) from context frames.
- A lightweight Masked Feature Transformer predicts future semantic features autoregressively. It takes context features as input and predicts the next frame's features, which are then fed back for subsequent predictions.
- Output: A sequence of predicted future semantic feature maps ( $\hat{h}_{M+1:K}$ ).

Stage 2: Semantics-Guided Video Generation

Goal: Render photorealistic future frames conditioned on the predicted semantics.
Mechanism:
- A Latent Diffusion Model (based on a Diffusion Transformer or DiT) operates within the latent space of a causal 3D VAE (WAN2.1).
- Conditioning: The diffusion model is conditioned on:
  1. Past context latents ( $z_{1:M}$ ).
  2. Predicted future semantic features ( $\hat{h}_{M+1:K}$ ) from Stage 1.
- Early Fusion: Semantic features and VAE latents are embedded and combined via channel-wise summation at the input level, ensuring the diffusion process is guided by the forecasted scene structure from the very first step.

Addressing the Train-Test Mismatch

A critical challenge arises because the diffusion model is trained with ground-truth semantic features but must rely on predicted (noisy) features during inference. To bridge this gap, Re2Pix introduces two robust conditioning strategies:

Nested Dropout: During training, the model stochastically truncates the semantic feature channels (e.g., keeping only the top $c$ channels out of 1152). This forces the diffusion model to rely on robust, coarse-to-fine semantic structures rather than overfitting to perfect, fine-grained feature values.
Mixed Supervision: The training data is a mixture (90% ground-truth features, 10% predicted features). This regularizes the model to handle imperfect conditioning signals without sacrificing semantic consistency.

3. Key Contributions

Hierarchical Semantic-to-Pixel Framework: Re2Pix is the first work to demonstrate that predicting VFM features can effectively guide hierarchical video diffusion, separating dynamics modeling from appearance synthesis.
Robust Conditioning Strategies: The introduction of Nested Dropout and Mixed Supervision successfully mitigates the train-test distribution shift, allowing the generator to operate reliably on imperfect, autoregressively predicted features.
Efficiency and Performance: The approach achieves significant improvements in training convergence speed and final generation quality compared to strong baselines.

4. Experimental Results

Experiments were conducted on autonomous driving benchmarks (Cityscapes, nuScenes, CoVLA, KITTI).

Performance Gains:
- Semantic Consistency: Re2Pix significantly outperforms baselines in temporal semantic fidelity (measured by mIoU for segmentation and depth estimation).
- Generation Quality: It achieves superior FID (Fréchet Inception Distance) and FVD (Fréchet Video Distance) scores, indicating more realistic and temporally coherent videos.
- Comparison: It outperforms standard diffusion baselines, REPA, and VideoREPA. It also surpasses or competes with large-scale, pre-trained world models (Vista, Cosmos-Predict-2) despite having fewer parameters and less training data.
Training Efficiency:
- Speed-up: Re2Pix accelerates training convergence dramatically. It reaches target FID/FVD scores 7× faster than the baseline and achieves 14× faster convergence for semantic metrics (segmentation mIoU).
Ablation Studies:
- Nested dropout and mixed supervision are proven essential for balancing semantic consistency with visual fidelity.
- The model is robust to the choice of VFM (DINOv2 vs. SigLIP-2).
- Performance remains strong even with reduced semantic components at inference, highlighting the informativeness of coarse semantic structures.

5. Significance

Re2Pix represents a paradigm shift in video prediction by moving away from entangled end-to-end latent prediction toward a structured, hierarchical approach.

Decoupling: By separating "what happens" (semantics) from "what it looks like" (pixels), the model achieves higher temporal consistency and avoids common artifacts like object drift.
Scalability: The framework demonstrates that leveraging frozen foundation models for intermediate representation prediction is a scalable and efficient strategy, reducing the computational burden on the generative model.
Application: The method is particularly significant for autonomous systems where reliable long-horizon reasoning and safe planning depend on accurate, semantically consistent future predictions.

In summary, Re2Pix proves that semantics-first design, supported by robust conditioning techniques, yields video prediction models that are not only more accurate and realistic but also significantly faster to train.