💻 computer science

Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator

Uni-ViGU proposes a generation-centric framework that unifies video generation and understanding by extending a diffusion-based video generator with a unified flow matching method and a bidirectional training mechanism, effectively overcoming the computational imbalance between these tasks to achieve competitive performance in both domains.

Original authors: Luozheng Qin, Jia Gong, Qian Qiao, Tianjiao Li, Li Xu, Haoyu Pan, Chao Qu, Zhiyu Tan, Hao Li

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luozheng Qin, Jia Gong, Qian Qiao, Tianjiao Li, Li Xu, Haoyu Pan, Chao Qu, Zhiyu Tan, Hao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a robot that is both a master painter and a sharp art critic.

Traditionally, scientists tried to build this robot by starting with a brilliant art critic (a model that understands images) and then trying to teach it how to paint. But there's a huge problem: Painting is exhausting. To create a single high-quality video, the robot has to make millions of tiny, repetitive calculations (like trying to sculpt a statue out of sand, grain by grain). Teaching a critic to do this heavy lifting is slow, expensive, and often results in messy paintings.

Uni-ViGU flips this script. Instead of teaching a critic to paint, the researchers started with a master painter (a powerful video generator) and taught it how to be a critic.

Here is how they did it, using some simple analogies:

1. The "Two-Track" Train System (Unified Flow)

Imagine a train station where two different types of trains arrive:

Video Trains: These are smooth, continuous streams of water (like a river). They need to flow gently to create a picture.
Text Trains: These are made of distinct, blocky Lego bricks. They snap together one by one.

Usually, you need two different stations to handle these. But Uni-ViGU built a super-station that handles both at once.

It treats the video like a river that needs to be smoothed out (Continuous Flow).
It treats the text like Lego bricks that need to be snapped into place (Discrete Flow).
The Magic: The station runs both trains on the same track at the same time. As the video gets clearer, the text gets more precise, and they help each other along the way.

2. The "Shared Brain, Specialized Hands" (MoE Architecture)

Think of the AI model as a factory with a central Brain and many Hands.

The Brain (Attention Layers): This part understands how things relate to each other. "The dog is chasing the ball." This logic is the same whether you are talking about a dog or a video. Uni-ViGU keeps this brain shared for both tasks.
The Hands (FFN Layers): This is where the actual work happens.
- The Video Hands are already strong because the model was pre-trained to paint. They keep their original, powerful muscles.
- The Text Hands are new and weak. So, the researchers gave the model a second pair of hands specifically for writing.

The Result: The model uses its shared brain to understand the connection between words and pictures, but it uses its specialized hands to do the heavy lifting for each task. It doesn't have to relearn how to paint; it just learns how to describe what it's painting.

3. The "Reverse Engineering" Training (Bidirectional Learning)

How do you teach a painter to be a critic? You use a two-step training camp:

Step 1: The Memory Game (Knowledge Recall)
The model is shown a video and a short, vague prompt (e.g., "A cat"). It is asked to recreate the prompt based only on the video.
- Why? Since the model already knows how to turn "A cat" into a video, it can easily reverse the process: turn the video back into "A cat." This wakes up its memory of how words and images connect.
Step 2: The Detective Work (Capability Refinement)
Now, the model is shown a video and a short prompt, but it is asked to write a long, detailed story about the video (e.g., "A fluffy orange cat with a blue collar is chasing a red ball across a green lawn...").
- Why? The short prompt isn't enough to write the long story. The model is forced to look closely at the video details (the fur, the color, the movement) to fill in the blanks. This forces it to become a sharp observer.

Why This Matters

The old way was like trying to teach a human to fly by making them run faster and faster on the ground. It's inefficient.
Uni-ViGU says: "Let's start with a bird that already knows how to fly, and teach it how to sing."

By starting with a video generator (the bird), the model already has a deep, intuitive understanding of how the visual world works. By simply adding a few "text hands" and training it to reverse-engineer its own creations, they created a system that is:

Cheaper: It doesn't need to relearn the basics of video.
Smarter: It understands the link between words and pictures better because it learned it from the inside out.
Unified: It can generate a video from a text, describe a video with text, or even generate both simultaneously, all in one package.

In short, Uni-ViGU proves that the best way to understand a video might be to learn how to make one first.

1. Problem Statement

Current unified multimodal models (MLLMs) typically follow an understanding-centric paradigm, where a pre-trained language model is extended to support visual generation. However, this approach faces a fundamental scalability bottleneck:

Computational Imbalance: Visual generation (especially video) is significantly more computationally expensive than visual understanding. Diffusion-based video generation requires iterative denoising over millions of tokens (e.g., ~74k tokens for a 5-second video), whereas text generation involves far fewer tokens.
Architectural Mismatch: Extending MLLMs to generate video often results in limited visual fidelity or requires decoupled training (e.g., freezing the MLLM and adding a generator), which prevents the generation objectives from directly benefiting the understanding capabilities.
The Core Question: Instead of forcing understanding models to generate, can generation-centric models serve as the foundation for unified intelligence? The authors argue that strong generative visual priors learned during text-to-video pretraining can be repurposed for understanding.

2. Methodology

Uni-ViGU proposes a framework that extends a state-of-the-art video generator (based on WAN2.1) to support both video generation and understanding within a single model. The methodology consists of three core components:

A. Unified Flow Matching (Uni-Flow)

To bridge the gap between continuous video data and discrete text tokens, the authors introduce a Uni-Flow process that handles both modalities in a single generative process:

Video (Continuous): Uses standard Continuous Flow Matching in the latent space of a Variational Autoencoder (VAE). It learns to transport Gaussian noise to the data distribution via linear interpolation.
Text (Discrete): Uses Discrete Flow Matching. Instead of autoregressive prediction, text tokens are mapped to continuous embeddings. The model learns to transport noise to the manifold of valid token embeddings.
Joint Optimization: The model simultaneously constructs flow paths for both modalities. Crucially, the noise levels ( $\tau_v$ for video, $\tau_t$ for text) are sampled independently. This allows the model to learn cross-modal dependencies at all stages of the denoising process (e.g., generating text from noisy video, or video from noisy text).

B. Modality-Driven Mixture-of-Experts (MoE) Architecture

To preserve the high-quality generative priors of the pre-trained video generator while enabling text generation, the authors design a specialized architecture:

Shared Attention: Self-attention and Cross-attention layers are shared across video and text tokens. This preserves the rich visual-semantic alignment learned during pretraining and enables bidirectional interaction.
Separate FFNs (Experts): Feed-Forward Networks (FFNs) are split into modality-specific experts ( $FFN_v$ $F F N_{v}$ for video, $FFN_t$ $F F N_{t}$ for text).
- $FFN_v$ : Retains the pre-trained weights to preserve generative video priors.
- $FFN_t$ : Newly initialized to learn text generation patterns.
Benefit: This design minimizes parameter overhead while ensuring that modality-specific knowledge (domain patterns) does not interfere with the shared relational reasoning.

C. Bidirectional Two-Stage Training

The training process is designed to transfer knowledge from generation to understanding and then refine it:

Stage 1: Knowledge Recall (Warm-up):
- Goal: Establish basic cross-modal alignment.
- Mechanism: The model is trained to reconstruct the input prompt ( $c$ ) given the video. To prevent the model from simply copying the prompt, condition dropout is applied (randomly dropping $c$ ). This forces the model to rely on the video latent to recover the text, effectively learning the reverse mapping (Video $\to$ Text) using the pre-trained text-to-video knowledge.
- Loss Balancing: Since video tokens vastly outnumber text tokens, a normalization factor ( $\lambda_t = |z_v|/|z_t|$ ) is applied to the text loss to ensure balanced optimization.
Stage 2: Capability Refinement:
- Goal: Develop fine-grained video understanding.
- Mechanism: The target text is replaced with detailed video captions (rich, semantic descriptions) rather than the coarse input prompt.
- Effect: The model can no longer rely on the prompt alone; it must actively attend to the video latent via shared self-attention to extract fine-grained details (objects, attributes, dynamics) to generate the detailed caption. This forces the development of genuine discriminative representations.

3. Key Contributions

Paradigm Inversion: Proposes a generation-centric unified framework, arguing that extending video generators is more scalable and principled than extending MLLMs for video generation.
Unified Flow Formulation: Introduces a novel method to perform continuous flow matching for video and discrete flow matching for text simultaneously within a single Transformer backbone, solving the modality mismatch problem.
Efficient Architecture: Designs a Modality-Driven MoE structure that shares attention layers (for alignment) but separates FFN layers (for domain specificity), allowing the reuse of pre-trained generative knowledge with minimal parameter cost.
Bidirectional Training Strategy: Develops a two-stage training mechanism (Knowledge Recall + Capability Refinement) that effectively repurposes generation priors for understanding and refines them for detailed semantic comprehension.

4. Results

Performance: Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation (maintaining the quality of the base WAN2.1 model) and video understanding (generating detailed, accurate captions).
Joint Generation: The model can perform joint video-text generation from noise, where the two modalities co-evolve. The text provides semantic guidance for the video, while the video provides visual context for the text, resulting in highly coherent and aligned pairs.
Qualitative Evidence: Visual results show that the model can generate videos with detailed, faithful captions that capture complex scenes, spatial relationships, and temporal dynamics, outperforming baselines that rely on decoupled training or autoregressive generation.

5. Significance

Scalability: By treating the video generator as the foundation, Uni-ViGU offers a scalable path toward unified multimodal intelligence, avoiding the prohibitive computational costs of training understanding-centric models to generate high-fidelity video.
Knowledge Repurposing: It validates the hypothesis that generative priors are a rich source of understanding. If a model knows how to generate a video from text, it inherently possesses the knowledge to understand the video and reconstruct the text.
Unified Architecture: It eliminates the need for dual-tower architectures or decoupled training, providing a single, coherent framework for both tasks that leverages the strengths of diffusion models for both continuous (video) and discrete (text) data.

In summary, Uni-ViGU represents a significant step forward in multimodal AI by demonstrating that a generation-first approach, supported by unified flow matching and efficient architectural design, can effectively unify video understanding and generation.