💻 computer science

VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System

VideoCoCo introduces an agentic dual-engine framework that leverages executable Blender code as a process-level chain of thought to generate physically consistent videos by first creating a deterministic spatiotemporal draft and then refining it into a photorealistic output, significantly outperforming existing baselines on physical consistency benchmarks.

Original authors: Haodong Li, Tianfei Ren, Xiaoxiao Ma, Chunmei Qing, Zhen Fang, Sipeng He, Ziyu Guo, Haoyu Wu, Juanxi Tian, Yihang Zou, Ruichuan An, Dongzhi Jiang, Boxue Yang, Ji Xie, Xu Huang, Wenhao Yan, Jialv Zou

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Haodong Li, Tianfei Ren, Xiaoxiao Ma, Chunmei Qing, Zhen Fang, Sipeng He, Ziyu Guo, Haoyu Wu, Juanxi Tian, Yihang Zou, Ruichuan An, Dongzhi Jiang, Boxue Yang, Ji Xie, Xu Huang, Wenhao Yan, Jialv Zou, Zhengrong Yue, Yaxin Luo, Xiaotong Li, Yuzhu Wang, Junyan Ye, Jinjing Zhao, Zehui Chen, Lin Chen, Renye Yan, Feng Zhao, Pheng-Ann Heng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot how to bake a cake just by whispering, "Make it delicious." The robot might understand the word "delicious," but without a recipe, it has no idea whether to mix the eggs before the flour, how long to bake it, or why the cake shouldn't turn into a brick. This is the current struggle in the world of Artificial Intelligence video generation. Scientists are building models that can turn text descriptions into moving pictures, but these AI "dreamers" often struggle with the laws of physics. They might make a ball float upward instead of falling, or a glass shatter before it hits the ground, because the AI is guessing the sequence of events rather than truly understanding how the world works.

To fix this, researchers are looking for a way to make the AI "think" before it acts, a concept known as Chain-of-Thought. Instead of jumping straight to the final picture, the AI is encouraged to create an intermediate plan. Think of it like an architect drawing blueprints before building a house, or a chef writing a recipe before cooking. The big question is: what kind of blueprint is best? Is it a list of words? A few rough sketches? Or something more precise? This is the puzzle the paper VideoCoCo sets out to solve, aiming to make AI-generated videos not just look pretty, but actually obey the rules of reality.

The Magic of "Code-as-CoT": A Two-Engine Dream Team

Meet VideoCoCo, a new system that acts like a super-smart director and a talented artist working together to make videos that don't break the laws of physics. The researchers realized that when AI tries to guess how a scene evolves just from a text prompt, it often gets the "story" wrong. So, they gave the AI a new tool: executable code.

Think of the process like this:

The Director (The Coding Agent): When you give the AI a prompt like "a butter melting in a hot pan," the Director doesn't just imagine the scene. Instead, it writes a Blender program (a set of computer instructions). This code is like a strict, unbreakable recipe. It explicitly tells the computer: "Place a block of butter here. Turn on the heat. Make the butter soften, slump, and spread over exactly 5 seconds." Because this is code, it must run exactly as written. It's not a guess; it's a simulation.
The Sandbox (The Simulation Engine): The computer runs this code in a safe, isolated playground called a "sandbox." The result isn't a pretty movie yet; it's a deterministic draft. Imagine a low-quality, white-clay animation. It looks like a rough sketch made of play-dough, but it is perfectly accurate in its movement. The butter melts exactly how physics says it should, because the code forced it to happen.
The Artist (The Generative Video Engine): Now, the second engine steps in. This is the artist who takes that rough, white-clay sketch and paints over it to make it look like a real, high-definition movie. Because the artist already has the perfect "play-dough" version to copy, they don't have to guess how the butter should move. They just focus on making it look shiny, golden, and delicious.

Why This Approach is a Game-Changer

The paper argues that previous methods tried to do too much at once. Some AI models tried to write a text plan (like "butter melts"), but text is vague. Others tried to guess the next frame of a video, but that's like trying to paint a masterpiece by looking at a blurry photo of the previous one.

VideoCoCo says: "Stop guessing the physics. Just simulate it."

By using code as the "Chain of Thought," the system separates the logic of the movement from the beauty of the image. The code handles the "what happens and when," while the video editor handles the "what it looks like." This is a huge shift because it turns a guessing game into a two-step process where the first step is 100% reliable.

The Proof is in the Pudding (and the Physics)

The researchers tested their system on two major challenges: PhyGenBench and VBench-2.0. These are like exams for AI, specifically testing if the videos follow real-world rules like gravity, heat, and how materials behave.

The Results: Before VideoCoCo, the baseline AI (called OmniWeaving) scored an average of 0.475 on the physics test. After adding the "Code-as-CoT" system, the score jumped to 0.558. On the other test (VBench-2.0), the score skyrocketed from 52.18% to 77.88%.
The "Aha!" Moment: The biggest improvements happened in areas where AI usually fails the most: thermal dynamics (like heat and melting) and material dynamics (how things break or flow). This proves that the code draft was actually teaching the AI how physics works, not just making the video look prettier.

What the Paper Rules Out

It's important to note what VideoCoCo is not. The paper explicitly argues against the idea that we can just train a bigger AI to "learn" physics from millions of videos. They found that without an explicit, executable plan (the code), the AI still struggles to get the sequence of events right. They also show that simply having a rough text plan or a few keyframes isn't enough; the plan needs to be executable code that the computer can actually run to create a draft.

The Catch and the Future

While the results are impressive, the paper is honest about the limitations. The system is currently a bit slower because it has to write code, run a simulation, and then paint the video. It's like having a brilliant architect and a painter, but it takes a little longer than just snapping a photo. Also, the system is limited by how complex the Blender simulator can be; super-complex things like swirling, chaotic water might still be tricky to simulate perfectly right now.

However, the authors suggest that this "Code-as-CoT" approach is a powerful new way to think about AI. Instead of hoping the AI magically understands the world, we can give it a tool to build a model of the world first, and then let it create the art based on that solid foundation. It's a reminder that sometimes, the best way to create something beautiful is to first build something that works.

Technical Summary: VideoCoCo

Problem Statement

Current text-to-video (T2V) models have achieved high visual fidelity but struggle with physically consistent dynamics. The core issue, termed Causal Opacity, arises because text prompts compress complex physical events into highly abstract semantic descriptions. Consequently, models must implicitly infer the entire spatiotemporal evolution of a scene without explicit guidance on the underlying physical principles.

Existing Chain-of-Thought (CoT) approaches attempt to mitigate this by introducing intermediate representations such as textual plans, layouts, or keyframes. However, these intermediates are often non-executable (lacking precise physical constraints) or temporally sparse (failing to capture continuous evolution). Furthermore, methods relying on test-time search or visual-state reasoning often remain descriptive or selective rather than instantiating a complete, inspectable process. This leaves a gap in generating videos where the temporal evolution strictly adheres to physical laws.

Methodology: VideoCoCo

The authors propose VideoCoCo, an agentic dual-engine framework that utilizes executable code as a process-level Chain-of-Thought. The system decouples process-level reasoning from high-fidelity visual realization through two distinct stages:

1. Executable Simulation Engine (Process Reasoning)

Instead of asking the video generator to infer dynamics, a coding agent synthesizes a self-contained Blender Python program based on the user prompt.

Code-as-CoT: The agent writes code that explicitly defines objects, physical properties, and temporal evolution. This representation is chosen because it is explicit (no underspecification), executable (commits to a concrete process), and inspectable.
Sandboxed Execution: The generated code is executed in an isolated Blender environment to produce a deterministic spatiotemporal draft.
Draft Characteristics: The output is a low-fidelity "white-clay" simulation. It lacks photorealistic textures and lighting but is temporally dense, providing a precise structural scaffold of what happens and when, ensuring physical consistency before any pixels are synthesized.

2. Generative Video Engine (Visual Realization)

The second engine focuses solely on translating the instantiated process into a photorealistic video, avoiding the need to re-reason about dynamics.

Instruction Construction: An instruction agent composes an editing instruction ( $e$ ) based on the original prompt ( $p$ ) and the rendered draft ( $d$ ). This instruction explicitly describes appearance (materials, lighting, style) while avoiding redefinition of the motion already present in the draft.
Draft-Conditioned Editing: A video editing model ( $G_\theta$ ) takes the draft ( $d$ ) and the instruction ( $e$ ) as conditions to generate the final high-fidelity video ( $\hat{v}$ ). The draft anchors the spatiotemporal structure, while the model handles the photorealistic appearance.

3. Data Construction: VideoCoCo-3K

To train the video editor to accept simulated drafts (which differ significantly from natural video distributions), the authors constructed VideoCoCo-3K.

Pipeline: For 3,000 prompts, the simulation engine generates drafts, and an instruction agent creates editing prompts.
Teacher Distillation: A high-fidelity teacher editor (Seedance 2.0) generates the target photorealistic videos ( $y$ ) conditioned on the drafts and instructions.
Result: A dataset of triplets $(d, e, y)$ that aligns physically grounded processes with their photorealistic realizations.

4. Training and Inference

Training: The editor is trained using a standard conditional denoising objective on the VideoCoCo-3K dataset. Experiments compare full fine-tuning versus parameter-efficient LoRA adaptation.
Inference: The system operates end-to-end from a text prompt alone. The coding agent generates the code, the sandbox renders the draft, the instruction agent composes the prompt, and the adapted editor produces the final video. No ground-truth drafts or human annotations are required at inference time.

Key Contributions

VideoCoCo Framework: An agentic dual-engine system that uses executable Blender code as a process-level Chain-of-Thought, separating spatiotemporal simulation from photorealistic editing.
VideoCoCo-3K Dataset: A curated dataset of 3,000 draft–instruction–target triplets generated via an agentic pipeline, providing the necessary supervision to adapt generative video engines to simulated drafts.
State-of-the-Art Performance: The framework achieves the best average scores on both PhyGenBench and VBench-2.0, demonstrating that executable code serves as an effective, controllable, and inspectable intermediate representation for physically consistent video generation.

Experimental Results

The paper evaluates VideoCoCo against a wide range of closed-source (e.g., Sora, Kling, Gen-3) and open-source (e.g., HunyuanVideo, CogVideoX, Wan2.2) baselines.

PhyGenBench: VideoCoCo improves the OmniWeaving baseline average score from 0.475 to 0.558, achieving the highest overall score. It shows significant gains in Material (+0.133) and Thermal (+0.078) dynamics, categories where appearance-driven priors typically fail.
VBench-2.0: The framework improves the average physical plausibility score from 52.18% to 77.88% (a gain of 25.70 points). It achieves the best scores in Mechanics (92.31%) and Thermotics (72.92%).
Ablation Studies:
- Draft Contribution: Even without fine-tuning the editor (Tuning-Free), the executable draft alone improves the average score from 0.475 to 0.506, confirming that the draft supplies the physical dynamics.
- Adaptation Strategy: LoRA adaptation yields the best results (0.558), outperforming full fine-tuning (0.535). The authors attribute this to the narrow nature of the task (restyling a simulation), where LoRA preserves the base model's strong visual priors while learning the draft-to-realistic mapping, whereas full fine-tuning risks overfitting.

Significance and Claims

The paper claims that executable code provides a superior intermediate representation for video generation compared to textual plans or sparse visual states. By externalizing the causal process into a deterministic simulation, VideoCoCo addresses the "Causal Opacity" of text-to-video generation.

The authors emphasize that this approach:

Decouples the difficult tasks of faithful dynamics and photorealistic appearance, allowing specialized engines to handle each.
Ensures that the physical evolution is explicitly defined and inspectable before visual synthesis begins.
Demonstrates that "Code-as-CoT" can be extended from image preview to complete temporal processes, offering a pathway toward more controllable and physically grounded generative world modeling.

The paper acknowledges limitations, noting that the approach introduces inference latency and is bounded by the expressiveness of the Blender simulator, particularly for highly complex phenomena like turbulent fluids. Future work is suggested to explore integrating specialized physical engines (e.g., Taichi) and knowledge distillation to internalize these priors directly into end-to-end models.