Technical Summary: UniVideo

Problem Statement

While unified multimodal models have demonstrated significant progress in text-image understanding and generation, they remain largely confined to the image domain. Existing video generation models are typically specialized for single tasks (e.g., text-to-video) and rely on text encoders that struggle with complex multimodal instructions. Furthermore, video editing methods often depend on task-specific modules or pipelines, lacking the scalability to handle diverse tasks within a single framework. Consequently, advanced capabilities such as multimodal prompting, in-context video generation, and sophisticated free-form video editing remain out of reach for unified systems.

Methodology

UniVideo is a unified framework designed to extend multimodal understanding, generation, and editing to the video domain. It employs a dual-stream architecture that integrates a Multimodal Large Language Model (MLLM) for visual understanding with a Multimodal Diffusion Transformer (MMDiT) for visual generation.

Architecture

1. Understanding Branch (MLLM): Utilizes a frozen MLLM (specifically Qwen2.5VL-7B) to process text, image, and video inputs. This branch preserves the model's original text generation and multimodal reasoning capabilities, allowing it to interpret complex instructions and ambiguous prompts.
2. Generation Branch (MMDiT): Based on the HunyuanVideo-T2V-13B backbone, this branch focuses on visual synthesis. It receives two types of inputs:
   • Semantic Stream: High-level semantic features from the MLLM, aligned via a trainable MLP connector.
   • Visual Stream: Fine-grained reconstruction signals encoded by a Variational Autoencoder (VAE).
3. Unified Conditioning: Unlike prior approaches that use task-specific bias embeddings or context adapters, UniVideo unifies diverse tasks (text-to-video, image-to-video, in-context generation/editing) under a single natural language instruction paradigm. It utilizes 3D positional embeddings to distinguish between condition latents and noisy video latents without offsetting all axes, a strategy found more effective than MRoPE for handling multiple visual inputs.

Training Strategy

The model is trained in three stages:

• Stage 1 (Connector Alignment): The MLP connector is trained while the MLLM and MMDiT remain frozen. Training involves text-to-image (T2I), text-to-video (T2V), and image reconstruction tasks to align feature dimensions.
• Stage 2 (Fine-tuning): The connector and MMDiT are fine-tuned on high-quality T2I/T2V samples with the MLLM still frozen, achieving performance comparable to the MMDiT backbone using its own text encoder.
• Stage 3 (Multi-task Training): The system is jointly trained on a broad spectrum of tasks, including in-context generation (multi-ID), in-context video editing (ID swapping, deletion, addition, style transfer), image editing, and image-to-video. The MLLM remains frozen throughout.

Key Capabilities

• Generation with Thinking: The MLLM branch interprets unconventional or hand-crafted prompts (e.g., visual annotations, diagrams) and translates them into structured plans or dense prompt tokens to guide the MMDiT.
• Zero-Shot Generalization: Despite not being explicitly trained on free-form video editing, UniVideo transfers editing capabilities learned from large-scale image editing data to the video domain, handling unseen instructions like changing materials or environments.

Key Contributions

1. Unified Framework: Introduction of UniVideo, the first model to unify understanding, generation, and editing for both images and videos within a single framework.
2. Dual-Stream Design: Proposal of a novel architecture combining MLLM reasoning with MMDiT generation, enabling the model to interpret multimodal instructions while maintaining visual consistency and identity.
3. Generalization: Demonstration that the unified framework generalizes to unseen task compositions (e.g., combining editing with style transfer) and transfers editing capabilities from images to videos without ad-hoc designs.
4. Empirical Validation: Systematic study of design choices (connectors, generators, conditioning strategies) providing evidence for the effectiveness of the proposed architecture.

Experimental Results

Extensive benchmarks demonstrate that UniVideo achieves competitive or superior performance compared to state-of-the-art task-specific baselines across multiple domains:

• Visual Understanding: Achieves strong scores on MMBench (83.5), MMMU (58.6), and MM-Vet (66.6), retaining the capabilities of its frozen MLLM backbone.
• Image Generation & Editing: Outperforms many specialized text-to-image models and unified multimodal models on GenEval. In image editing (ImgEdit-Bench, GEdit-Bench), it surpasses several specialized editing models in semantic consistency and overall quality.
• Video Generation: Matches the performance of the HunyuanVideo backbone on VBench for text-to-video tasks.
• In-Context Tasks:
  • Generation: Outperforms baselines (VACE, Kling1.6, Pika2.2) in subject consistency and prompt following for both single-ID and multi-ID scenarios.
  • Editing: Operates in a mask-free setting (requiring no explicit mask inputs), yet achieves superior or competitive performance in identity alignment, video quality, and prompt following compared to task-specific experts like UNIC and AnyV2V.

Significance

The paper claims that UniVideo represents a significant step toward building multimodal AI assistants capable of seamless communication through language, images, and video. By unifying diverse video tasks under a single instruction paradigm, the model overcomes the limitations of specialized pipelines. Its ability to generalize to novel task compositions and transfer editing skills from images to videos highlights the potential of unified frameworks to handle complex, open-ended multimodal interactions. Furthermore, the "generation with thinking" capability allows the system to interpret complex visual prompts, bridging the gap between human intent and precise video synthesis.