🤖 AI

Video Generation Models are General-Purpose Vision Learners

This paper proposes GenCeption, a feed-forward perception model built upon a pre-trained text-to-video diffusion backbone, demonstrating that large-scale video generation serves as a superior pre-training paradigm for achieving state-of-the-art, data-efficient, and general-purpose visual intelligence across diverse computer vision tasks.

Original authors: Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, Cristian Sminchisescu

Published 2026-07-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, Cristian Sminchisescu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've spent years building a super-smart robot chef. This chef is incredible at following recipes to cook a perfect steak, a fluffy omelet, or a spicy curry. But here's the catch: if you ask the chef to "tell me the temperature of the kitchen" or "count the number of eggs in the fridge," they just stare blankly. They are a master of making things, but not of understanding the world they are making.

For a long time, computer vision (the field of teaching computers to see) has been like that chef. We built separate, specialized robots for every single job: one robot to measure depth, another to track faces, a third to find edges. They were great at their specific jobs, but they couldn't talk to each other or learn new tricks without being completely rebuilt.

Then, a team of researchers at Google DeepMind and other universities asked a wild question: What if we took that robot chef, the one that learns by trying to generate perfect videos, and asked it to become a general-purpose detective instead?

Their paper, titled "Video Generation Models are General-Purpose Vision Learners," suggests that the answer is a resounding "yes." They introduce a new model called GenCeption.

The Magic Trick: From "Making" to "Knowing"

Think of the old way of training vision models like teaching a student by showing them flashcards. You show a card with a tree and say "tree," then a card with a car and say "car." It works, but it's slow and rigid.

The new method, which the authors call GenCeption, is like teaching that student by having them write a million different stories about the world. To write a story about a gorilla waving at a camera, the student has to understand how a gorilla looks, how a hand moves, how light hits fur, and how the camera angle changes over time. They have to internalize the physics of the world just to make the story make sense.

The paper argues that this process of text-to-video generation is actually the ultimate training ground for vision. By training a model to create high-quality videos from text prompts (like "A gorilla approaches waving hand"), the model accidentally learns a massive amount of "world knowledge"—how 3D space works, how objects move, and how light behaves.

The Big Shift: One Model to Rule Them All

The researchers took a massive video-generation model (based on a technology called a "diffusion model," which is like a noise-remover that slowly turns static into a clear picture) and gave it a makeover.

Instead of letting it slowly "dream" a video frame by frame (which takes forever), they tweaked it to be a feed-forward model. Think of it like this: instead of the model taking 50 steps to slowly guess what the answer is, they trained it to look at the input and spit out the answer in a single, lightning-fast step.

They then taught this single model to do everything just by changing the text prompt.

Prompt: "Depth" → The model outputs a depth map (how far away things are).
Prompt: "Surface Normal" → The model outputs the direction surfaces are facing.
Prompt: "Segmentation" → The model outlines objects.
Prompt: "3D Keypoints" → The model finds the joints of a person's body.

It's like having a Swiss Army knife where the tool you need appears automatically based on what you ask for, rather than having to carry a separate knife, screwdriver, and scissors in your pocket.

The Results: Beating the Specialists

The team tested GenCeption on a huge list of tasks, from measuring the depth of a scene to tracking a person's pose while skiing. The results were surprising.

It beats the experts: In many cases, this single "generalist" model performed just as well as, or even better than, the specialized models built specifically for those tasks. For example, it matched or beat models like DepthAnything3 (a depth expert) and SAM3 (a segmentation expert).
It's a data wizard: One of the most mind-blowing parts is how little data it needed. The authors found that GenCeption could achieve performance comparable to top-tier models like D4RT and VGGT-Ω using 7 to 500 times less training data. It's as if the model learned the rules of the game so well from its video-generation training that it didn't need to memorize every single play.
It's a generalist: The model was trained almost entirely on synthetic data (computer-generated videos of humans). Yet, when they showed it real-world videos of animals, robots, or people doing things it had never seen, it still worked. It could even track a cat's whiskers or segment hair in a video it had never seen before. This suggests the model learned the concept of "fur" or "hair" rather than just memorizing specific pictures.

What This Paper Says It Is NOT

It's important to know what this paper doesn't claim. The authors are careful to say that this isn't a magic wand that solves every problem in the world yet.

They explicitly argue against the idea that we need to build a new, custom architecture for every single vision task. They show that the old way of "specialized models" is less efficient than this unified approach.
They also note that while the model is amazing, it isn't perfect. When they tried to combine all tasks into one training session, the model got a bit confused with 3D human keypoint estimation (tracking body joints). The performance dropped slightly compared to when it was trained just for that one job. This suggests that while the "generalist" approach is powerful, there are still some kinks to work out when mixing very different types of tasks.
They also clarify that the model's ability to handle real-world videos after being trained on fake ones is an emergent behavior—a happy accident of the training method, not something they explicitly programmed.

The Bottom Line

The paper suggests that the future of computer vision might not be about building more specialized robots, but about teaching one super-robot to "imagine" the world so vividly that it understands it perfectly.

By using the power of video generation as a pre-training tool, GenCeption shows that a model can learn to see, measure, and understand the physical world just by trying to create it. It's a shift from "engineering a solution for every problem" to "creating a smart foundation that can solve them all."

The authors suggest that this path—using generative models as the base for perception—could be the key to building truly general-purpose vision intelligence, similar to how Large Language Models (LLMs) revolutionized text. It's a promising step, but as the paper notes, we are still in the early stages of figuring out exactly how to make this generalist brain perfect for every single task.

Technical Summary: GenCeption – Video Generation Models as General-Purpose Vision Learners

1. Problem Statement

Computer vision currently operates in a "specialized model" era, where distinct architectures are required for specific tasks (e.g., separate models for depth estimation, segmentation, or pose estimation). This contrasts with Natural Language Processing (NLP), where Large Language Models (LLMs) have evolved into unified foundation models capable of handling diverse tasks through a single architecture driven by next-token prediction.

The paper posits that computer vision lacks a universal pre-training objective analogous to next-token prediction. Existing vision pre-training paradigms (e.g., contrastive learning, masked autoencoders like MAE/VideoMAE) often lack inherent spatio-temporal understanding, native vision-language alignment, or the ability to scale effectively to general-purpose intelligence. The authors argue that the field needs a paradigm shift from task-specific engineering to a unified, task-agnostic architecture that can internalize the 4D physics of the world and follow natural language instructions.

2. Methodology: GenCeption

The authors introduce GenCeption, a generalist video perception model that repurposes large-scale text-to-video generative diffusion models as a pre-training base for perception tasks. The methodology is built on three core principles:

A. Multimodal Generative Pre-training as Representation Learning

Instead of training a model from scratch for perception, GenCeption utilizes a pre-trained text-to-video diffusion backbone (specifically WAN 2.1). The generative objective of creating high-fidelity video sequences conditioned on text forces the model to internalize:

Spatio-temporal priors: 3D geometry, object permanence, and physical laws.
Vision-language alignment: A native understanding of visual concepts linked to linguistic semantics.

B. Task-Agnostic Unified Architecture

GenCeption treats diverse vision tasks as a unified sequence-to-sequence mapping problem.

Unified Representation: All dense perception tasks (depth, surface normals, segmentation, dense pose, camera raymaps) are mapped into a standard 3-channel RGB space within the range $[0, 1]$ . For example, depth maps are normalized and mapped to RGB channels; camera poses are represented as "Raymaps" (spatially partitioned rotation and translation components).
Sparse Tasks: For structured outputs like 2D/3D keypoints, the model appends learnable tokens to the video latents. An MLP decodes these tokens to predict coordinates per frame.
Instruction Steering: The specific target task is defined solely by a text prompt (e.g., "Depth," "Segmentation," "Pose"), eliminating the need for task-specific heads or architectural modifications.

C. Feed-Forward Reformulation

While diffusion models typically rely on slow, iterative denoising, perception requires efficiency. GenCeption transforms the multi-step diffusion process into a single-step, feed-forward model:

The input to the Diffusion Transformer (DiT) is the clean latent of the input video (rather than noisy latents).
The conditioning timestep is fixed to $t=0$ (the end of the generative process).
The model predicts the velocity $v = \epsilon - x_0$ (based on Rectified Flow objectives). The output is negated ( $-v = x_0 - \epsilon$ ) to align with the target latent, allowing for direct, single-pass inference.

D. Training Strategy

Data: The model is trained primarily on synthetic data (7,500 videos generated from RenderPeople assets and CMU motion capture) to ensure ground truth for all modalities (depth, normals, masks, poses). Real-world datasets are used only for expression-referring segmentation.
Loss Function: A unified $L_2$ loss is applied across all tasks in the latent space (for dense tasks) or output space (for sparse tasks). Task-specific constraints (like scale invariance for depth) are handled via data preprocessing (e.g., normalizing depth maps by scene median) rather than custom loss functions.

3. Key Contributions

Paradigm Shift: The paper proposes that large-scale text-to-video generation is a superior pre-training paradigm for computer vision compared to traditional self-supervised methods (e.g., V-JEPA, VideoMAE), satisfying the requirements for spatio-temporal evolution, vision-language alignment, and scalability.
Unified Architecture: GenCeption demonstrates a single model capable of performing heterogeneous tasks (dense and sparse) using a unified backbone, head, and loss function, steered only by text prompts.
Efficiency: By converting the iterative diffusion process into a single feed-forward pass, the model achieves real-time inference speeds (e.g., 13.6 FPS for the 1.3B model) without sacrificing accuracy.
Emergent Behaviors: The model exhibits zero-shot generalization capabilities, including sim-to-real transfer and generalization to out-of-distribution (OOD) object categories (e.g., animals and robots) despite being trained exclusively on synthetic human data.

4. Experimental Results

The authors evaluate GenCeption on a diverse suite of benchmarks, comparing it against specialized state-of-the-art (SOTA) models.

Performance vs. Specialists: GenCeption matches or surpasses specialized models across multiple tasks:
- Depth Estimation: Outperforms or matches DepthAnything V3, D4RT, and VGGT-Ω.
- Surface Normals: Surpasses NormalCrafter and Lotus-2.
- Segmentation: Achieves competitive results in foreground and expression-referring segmentation (outperforming SAM3 in understanding complex sentences).
- 3D Pose: Delivers stronger 3D keypoint prediction than Genmo and TRAM.
Pre-training Comparison: Under comparable settings, the video generative pretrained backbone (WAN 2.1) significantly outperforms alternative pre-training paradigms like V-JEPA and VideoMAE V2.
Data Efficiency: The model achieves performance comparable to leading models (D4RT, VGGT-Ω) using 7× to 500× less training data.
Scaling: Preliminary results show that performance improves with increased model size (1.3B to 14B) and data volume.

5. Significance and Claims

The paper claims that video generation is not merely a synthesis tool but a foundational path toward generalist vision intelligence.

The "World Model" Hypothesis: The authors suggest that the rich spatiotemporal priors learned during large-scale video generation effectively constitute a "world model" that can be directly harnessed for perception.
Scalability: By shifting task specifications from architectural changes to data format designs (analogous to how LLMs handle diverse tasks via text), the paradigm allows for seamless scaling to an ever-expanding spectrum of visual tasks.
Emergent Generalization: The ability of a model trained on synthetic human data to generalize to real-world footage and unseen object categories (animals, robots) without explicit training suggests that generative pre-training captures fundamental physical and semantic regularities of the world.

The authors conclude that this approach paves the way for a truly unified vision intelligence for the physical world, moving beyond the limitations of specialized, task-specific computer vision models.