Technical Summary: Gen4U: Unifying Video Generation and Understanding via Diffusion

Problem Statement

Current paradigms in visual representation learning face a fundamental trade-off: they struggle to reconcile precise low-level geometry with high-level semantics. Contrastive methods and large Vision-Language Models (VLMs) excel at semantic understanding but often neglect spatio-temporal details. Conversely, pixel-level reconstruction approaches like Masked Autoencoders (MAE) preserve local geometry and motion but fail to generalize to broad semantic tasks. While recent studies suggested that frozen video diffusion models capture low-level geometry but lack high-level semantic awareness, there remains an open problem of finding a single, task-agnostic foundation video model capable of supporting both generation and diverse understanding tasks without extensive fine-tuning.

Methodology

The authors introduce Gen4U (Generation for Understanding), a framework that repurposes frozen, large-scale video diffusion models as competitive video encoders. The methodology relies on a systematic analysis of the latent space structure of diffusion models to identify optimal points for feature extraction.

1. Latent Space Analysis

The authors probe the intermediate activations of both proprietary (Veo3) and open-weight (Wan2.2-T2V-A14B) video diffusion models across network depth and noise levels. They utilize:

• Mutual k-NN (MkNN) Alignment: A zero-shot metric to compare the neighborhood structure of diffusion representations against established encoders (e.g., Gemma-2 text encoder, DINOv2, VideoMAEv2) without learned mapping.
• Probing Mechanisms: Linear probes (global pooling followed by linear projection) and attention probes (cross-attention heads with task-specific queries) to evaluate feature quality for downstream tasks.

2. Feature Extraction Strategy

The core insight is that diffusion representations evolve along a dual-axis trajectory (depth and noise).

• Semantic Bottleneck: The analysis reveals a consistent "semantic bottleneck" at approximately 60% noise level, where global semantics are most linearly separable.
• Feature Complexity: While moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered. These scattered features require attention mechanisms to decode effectively, whereas linear probes suffice for global semantics at the semantic bottleneck.
• Single Forward Pass: Gen4U extracts features from a single optimal block and noise level, requiring only one forward pass through the diffusion backbone. This circumvents the high inference costs of iterative denoising, making it computationally comparable to standard discriminative encoders.

3. Downstream Adaptation

Frozen generative representations are paired with lightweight decoders (e.g., one-block attention decoders or small language models) trained specifically for the downstream task. The pre-trained generative weights remain frozen, preserving the model's video generation capabilities.

Key Contributions

1. Latent Space Mapping: The paper maps the evolution of diffusion features, identifying optimal extraction points for understanding tasks. It reveals that state-of-the-art video diffusion models possess a highly structured latent space where visual representations evolve systematically across depth and noise.
2. Unified Paradigm: Gen4U demonstrates that frozen video diffusion models can function as general-purpose video encoders, unifying generation and understanding. The framework achieves strong performance across semantic and non-semantic objectives without altering the generative weights.
3. Dual Capability: The work shows that these models capture both rich high-level understanding (semantics) and strong low-level geometry awareness (motion, depth, pose), challenging the prior belief that diffusion models struggle with high-level semantics.

Experimental Results

The authors evaluate Gen4U on a wide spectrum of tasks using the Veo3 model:

• Video Classification: On the challenging Something-Something V2 (SSv2) dataset, Gen4U achieves a 71.3% Top-1 accuracy (72.6% with data augmentation), surpassing strong baselines like 4DS (68.2%) and V-JEPA (72.2%) in the frozen representation setting.
• Geometry Understanding:
  • Depth Estimation: On ScanNet, Gen4U achieves an AbsRel of 0.075, a 10.7% relative improvement over the frozen-feature baseline reported in prior work.
  • Camera Pose Estimation: The model achieves an End-Point Error (EPE) of 1.10, performing on par with the DINOv2 baseline (1.08).
• Captioning:
  • Video Captioning: On SSv2, Gen4U produces reasonable captions. On the VATEX dataset, it significantly outperforms VideoPrism (CIDEr 31.7) even when the LLM decoder is frozen, achieving a CIDEr of 44.8 (vs. 31.7 for VideoPrism).
  • Image Captioning: On MS-COCO, the model shows satisfactory performance, though it trails behind SigLIP-based baselines optimized specifically for text alignment. The authors note that using high-resolution inputs and noise-level data augmentation reduces this gap.

Significance and Claims

The paper claims that training purely for video generation at scale is sufficient to endow models with powerful, general-purpose features. The implicit emergence of alignment between generative representations and text/visual encoders is a significant finding not previously reported.

• Unification: Gen4U unifies the generation and understanding paradigms, suggesting that a single visual model can be shared across both domains.
• Efficiency: By extracting features from a single forward pass, the framework avoids the computational overhead of iterative denoising, making it as efficient as discriminative encoders.
• Representation Power: The results challenge the notion that diffusion models are limited to low-level geometry, demonstrating that state-of-the-art models encode rich semantic information within their intermediate features.

The authors acknowledge limitations, noting that the primary analysis relies on a proprietary model (Veo3), which limits reproducibility, though the open-weight Wan 2.2 model was used to validate the generalizability of the findings. They conclude that this work encourages further research into the representation power of diffusion models for perception tasks.