💻 computer science

CogOmniControl: Reasoning-Driven Controllable Video Generation via Creative Intent Cognition

CogOmniControl is a reasoning-driven framework that enhances controllable video generation by integrating a specialized CogVLM for accurate creative intent cognition with a unified CogOmniDiT generator and a closed-loop Best-of-N selection mechanism, achieving superior performance on professional benchmarks compared to existing models.

Original authors: Hongji Yang, Songlian Li, Yucheng Zhou, Xiaotong Zhao, Alan Zhao, Chengzhong Xu, Jianbing Shen

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongji Yang, Songlian Li, Yucheng Zhou, Xiaotong Zhao, Alan Zhao, Chengzhong Xu, Jianbing Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a film director trying to explain a complex scene to a team of animators. You have a rough sketch (a storyboard), a clay model of a character, and a few vague notes like "make it feel rainy and sad."

In the past, AI video generators were like animators who could only follow strict, pixel-perfect instructions. If you gave them a rough sketch, they would get confused, produce a muddy mess, or ignore your emotional notes entirely. They lacked the "creative brain" to understand why you wanted the scene to look a certain way.

CogOmniControl is a new system designed to fix this. It acts like a super-smart creative director who sits between you (the user) and the animation team (the AI generator). Here is how it works, broken down into simple steps:

1. The Problem: The "Translation Gap"

Current AI video tools are great at making things look real, but they struggle when you give them abstract or messy instructions (like a hand-drawn sketch or a clay model).

The Old Way: You give a sketch; the AI tries to copy the lines exactly but misses the feeling or the story.
The Result: The video looks wrong, the character's face changes (identity drift), or the movement is jerky.

2. The Solution: A Two-Part Team

The authors built a system with two main characters, working together:

Character A: The "Creative Director" (CogVLM)

Think of this as a highly trained film director who has watched thousands of professional anime production videos.

What it does: It doesn't just look at your sketch; it understands your intent. If you show a sketch of a character in the rain, this "Director" doesn't just see lines. It reasons: "Okay, the character is sad, the ground should be wet, the rain should ripple, and the lighting should be gloomy."
The Magic: It turns your vague, abstract ideas into a dense, detailed plan. It acts like a translator, converting your "creative intent" into a clear set of instructions that the computer can actually follow.
Training: They taught this Director using real data from professional animation studios (storyboards and clay renders), not just random internet videos. This makes it an expert in "how things should look" rather than just "what things look like."

Character B: The "Animator" (CogOmniDiT)

This is the actual video generator, the one that paints the pixels.

What it does: It takes the detailed plan from the Creative Director and the original sketch, and it builds the video.
The Magic: Because it is listening to the Director's detailed plan, it knows exactly how to move the character, how the clothes should flutter, and how the light should change. It doesn't just guess; it follows a logical roadmap.

3. The "Harness" System: The Quality Control Loop

Here is the cleverest part. Usually, you generate a video and hope it's good. CogOmniControl adds a Quality Control Harness.

The Idea: Before the final video is shown, the Creative Director (CogVLM) acts like a producer. It says, "Okay, we need to check three things: Is the character's face consistent? Is the rain moving naturally? Is the lighting right?"
The Process:
1. The system generates several versions of the video (like trying out 4 different takes).
2. The Director picks specific "Inspectors" (evaluators) based on what the scene needs. If the scene has a specific character, it picks an "Identity Inspector." If it's a storm, it picks a "Physics Inspector."
3. These inspectors grade the videos.
4. The system picks the best one (Best-of-N) and shows that to you.

4. The New "Test Drives" (Benchmarks)

To prove their system works, the authors didn't just use standard tests. They built two new "driving tracks" called CogReasonBench and CogControlBench.

These tracks are filled with real-world professional challenges (like turning a rough storyboard into a final movie scene).
They found that their system, CogOmniControl, performed better than all other open-source models and came very close to the best expensive, private systems.

Summary Analogy

Imagine you want to bake a very specific, complex cake.

Old AI: You hand it a napkin with a scribble of a cake. It tries to copy the scribble exactly, resulting in a burnt, misshapen mess.
CogOmniControl: You hand the napkin to a Master Chef (CogVLM). The Chef reads your scribble, understands you want a "moist chocolate cake with a raspberry center," and writes a precise recipe. Then, the Baker (CogOmniDiT) follows that recipe perfectly. Finally, the Chef tastes a few batches, picks the best one using a specific checklist, and serves you the perfect cake.

The paper claims this approach bridges the gap between "vague human ideas" and "precise computer execution," making it possible to generate high-quality, professional-looking videos from simple sketches and abstract ideas.

Technical Summary: CogOmniControl

1. Problem Statement

Recent diffusion-based video generative models have achieved significant progress in photorealism and motion fluency. However, they remain fragile when faced with abstract, sparse, or complex control conditions common in professional production workflows (e.g., storyboard sketches, clay renders). Existing approaches typically fall into two categories:

Adapter-based methods: These inject conditions via additional modules but often lack the flexibility to handle diverse, non-pixel-aligned constraints, leading to artifacts and identity drift.
Generic VLM-coupled methods: These integrate general Vision-Language Models (VLMs) to interpret conditions. However, generic VLMs struggle to comprehend underlying creative intents in professional contexts, often failing to formulate generation plans grounded in domain-specific knowledge. Furthermore, their reasoning outputs can introduce noise, and there is a misalignment gap between the VLM's abstract reasoning and the final pixel-level video generation.

The core challenge is bridging the gap between abstract user conditions and faithful, high-quality controllable video generation that aligns with genuine creative intent.

2. Methodology

The authors propose CogOmniControl, a reasoning-driven framework that factorizes controllable video generation into two distinct stages: Creative Intent Cognition and Generation. The framework consists of two primary components: CogVLM and CogOmniDiT.

2.1 CogVLM: Creative Intent Cognition

CogVLM is a specialized Vision-Language Model trained to act as a "professional director." Its role is to ingest multimodal inputs (Control Video, Reference Image, Textual Description) and cognize the user's creative intent, transforming sparse cues into dense, logical reasoning outputs.

Training Strategy: CogVLM undergoes a two-stage training process:
1. Supervised Fine-Tuning (SFT): Trained on authentic anime production data (storyboards, clay renders) to learn professional workflows.
2. Reinforcement Fine-Tuning (RFT): Utilizes a Holistic Reward and Fact Verification Reward mechanism (based on LLM-as-a-Judge) to optimize for Creative Intent, Physical Plausibility, Information Integrity, and Motion Description. This ensures the reasoning is factually grounded and avoids hallucinations.
Output: CogVLM produces a reasoning output that details the generation plan and, crucially, identifies a set of evaluators (tools) from a predefined library suitable for the specific input conditions.

2.2 CogOmniDiT: Unified Video Generation

CogOmniDiT is a Video Diffusion Transformer (DiT) that unifies diverse control signals within a single sequence.

Input Sequence: It concatenates noisy latents, reference image latents, control video latents, and the high-level semantic embeddings from CogVLM.
Training: It employs a three-stage SFT strategy (in-context generation, connector alignment, joint training) followed by RFT. The RFT aligns the DiT's pixel-level output with the high-level reasoning of CogVLM using rewards for condition following and video quality.
Mechanism: By leveraging in-context learning, CogOmniDiT allows conditions and latents to model themselves and each other via self-attention, ensuring precise injection of control signals.

2.3 Closed-Loop Verification (Evaluator Harness)

A unique feature of CogOmniControl is its "harness-like" architecture. Instead of using a fixed set of evaluators for all generations, CogVLM dynamically selects specific evaluators (e.g., Identity Consistency, Physical Dynamics, Storyboard Annotation Following) based on the input conditions.

Best-of-N Selection: The system generates multiple video candidates ( $N$ ) and uses the CogVLM-selected evaluators to score them. The candidate with the highest score is selected, creating a closed-loop Reasoning-Generation-Verification pipeline.

3. Benchmarks and Data

To address the lack of benchmarks for professional creative intent, the authors introduce two new datasets derived from real-world professional animation production pipelines:

CogReasonBench: Designed to evaluate the VLM's ability to cognize creative intent and perform reasoning from multimodal inputs.
CogControlBench: Designed to evaluate the controllable video generation quality and condition following under abstract and sparse conditions.
Unlike existing benchmarks that often use simulated or synthetic data, these benchmarks utilize native data from professional workflows (storyboards, clay renders) and their corresponding final videos.

4. Experimental Results

Experiments were conducted on the proposed benchmarks and compared against state-of-the-art open-source models (e.g., VACE, VINO, OmniWeaving) and proprietary systems (e.g., Kling, Seedance).

CogVLM Performance: On CogReasonBench, the RFT-trained CogVLM significantly outperformed generic VLMs (e.g., Qwen3-VL-Thinking) in Multimodal Intent, Information Integrity, and Motion Description, demonstrating the necessity of domain-specific training.
CogOmniControl Performance: On CogControlBench, CogOmniControl achieved the highest average score (0.727) among open-source models, surpassing VINO (0.686) and VACE-Wan2.1 (0.665). It narrowed the performance gap with the strongest proprietary system, Seedance2.0 (0.750).
Best-of-N Impact: Employing the Best-of-N selection with adaptively chosen evaluators further improved performance (0.742), validating the effectiveness of the closed-loop harness architecture.
Qualitative Results: Visual comparisons show that CogOmniControl produces videos with better visual consistency, smoother motion, and more accurate adherence to abstract controls (like clay renders) compared to adapter-based methods which often suffer from artifacts and semantic misalignment.

5. Significance and Claims

The paper claims that CogOmniControl represents a shift from simple condition injection to reasoning-driven generation. Its significance lies in:

Bridging the Cognitive Gap: By training a specialized CogVLM on professional data, the framework successfully interprets abstract conditions and aligns them with high-level creative intent, a task where generic VLMs fail.
Closed-Loop Architecture: The integration of adaptive evaluator harnesses allows for a self-correcting generation process, enabling Best-of-N selection that is tailored to the specific creative intent of each input.
Professional Relevance: By utilizing real-world production data and benchmarks, the framework addresses the specific needs of professional workflows (storyboarding, clay rendering) rather than just general text-to-video tasks.

The authors conclude that CogOmniControl consistently outperforms existing open-source controllable video generators and demonstrates that reasoning-driven frameworks can effectively handle the complexities of professional video production.