💻 computer science

CoT-Edit: Let CoT Guide Instruction Video Editing

CoT-Edit introduces a plan-guided-edit framework that leverages a Chain-of-Thought-enhanced multimodal large language model to generate precise spatial priors and attribute-rich directives, thereby enabling text-driven video editing in complex scenes with improved object localization, physical consistency, and temporal coherence.

Original authors: Sen Liang, Fengbin Guan, Youliang Zhang, Xin Li, Zhibo Chen

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sen Liang, Fengbin Guan, Youliang Zhang, Xin Li, Zhibo Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a robot to edit a home movie. You don't want to spend hours learning complex software; you just want to say, "Make the dog fly like a superhero," and have it happen. This is the dream of instruction-based video editing, a field where computers try to understand your natural language commands and change the pixels in a video to match. But here's the catch: computers are notoriously bad at "common sense." If you tell a robot to "add a UFO flying in a circle," it might just paste a UFO image onto the sky, ignoring gravity, or it might accidentally turn the wrong dog into a cat if there are two dogs in the scene. It's like giving a chef a recipe that says "add some spice" without telling them which spice or how much—the result is often a mess. To fix this, researchers are trying to bridge the gap between vague human words and precise, physical actions in a video, ensuring that if a ball is thrown, it follows the laws of physics, and if you point to a specific object, the computer knows exactly which one you mean.

Enter CoT-Edit, a new method that acts like a brilliant, step-by-step project manager for video editing. Instead of just shouting an instruction and hoping the computer gets it right, CoT-Edit forces the AI to "think before it acts." The authors propose a three-step framework called Plan–Guide–Edit. First, the Planner (a super-smart AI brain) looks at the video and your text command. It doesn't just guess; it uses a technique called Chain-of-Thought (CoT) to break the task down. It performs structured reasoning steps to analyze the scene: "What is the dog doing? Where is it? If I add a UFO, where should it fly to look real?" It then generates a sequence of normalized bounding boxes to mark the objects and writes a more detailed, "enriched" instruction that includes physical rules, like "the UFO must follow a curved path."

Next, the Guide takes these bounding boxes and turns them into actual masks—think of them as digital stencils or cut-out shapes that tell the computer exactly where to work. Because the computer now has a clear map (the boxes) instead of just a vague idea, it doesn't have to guess which dog to change or where the new object should land. Finally, the Editor (the artist) uses these stencils and the detailed instructions to paint the new content into the video. It's like the difference between a painter being told "paint a bird" and being given a specific outline of a bird, a list of colors, and a rule that says "the bird must be sitting on the branch, not floating in mid-air."

The paper finds that this "think-first" approach works significantly better than previous methods. In tests, CoT-Edit was much better at picking the right object when there were many similar ones (like choosing the yellow dog over the brown one) and at making new objects move in physically realistic ways (like a ball bouncing correctly). The researchers trained their system in two stages: first, they taught the mask-maker and the editor separately using a mix of public datasets; then, they trained them together on about 100,000 high-quality video editing pairs. When they tested it, CoT-Edit outperformed other top methods in almost every category, including how smooth the motion looked and how well it followed the user's instructions. The authors suggest that by explicitly planning the "where" and "how" before the "what," they can create video edits that are not only visually impressive but also make logical sense to the human eye.

Technical Summary: CoT-Edit: Let CoT Guide Instruction Video Editing

1. Problem Statement

Text-driven instruction-based video editing faces significant challenges in complex real-world scenarios, particularly when dealing with multiple coexisting objects, densely distributed similar entities, or dynamic interactions. Purely textual prompts often fail to capture precise spatial relationships and physical constraints, leading to:

Target Ambiguity: Difficulty in selecting the intended object among similar ones (e.g., changing the "yellow dog" when multiple dogs are present).
Physical Implausibility: Failure to respect motion trajectories, scale, or contact relationships (e.g., an object flying in an impossible pattern or ignoring gravity).
Localization Drift: Inaccurate spatial grounding resulting in edits that affect the wrong regions or lack temporal stability.

Existing methods often rely on large-scale datasets and high-capacity models to implicitly learn these constraints, yet they frequently suffer from insufficient spatial grounding. Furthermore, generating masks solely from ambiguous text propagates semantic uncertainty to the spatial layer, compromising edit quality.

2. Methodology: The Plan–Guide–Edit Framework

The authors propose a modular framework that explicitly bridges high-level semantic intent and low-level spatial execution through three core stages: Plan, Guide, and Edit.

3.1. Planner: Semantic–Spatial Planning with CoT

The Planner acts as a semantic-spatial translator, utilizing a Chain-of-Thought (CoT)-enhanced Multimodal Large Language Model (MLLM). Instead of direct single-pass output, the MLLM performs a structured, multi-step reasoning process on input keyframes and text instructions:

Task Parsing & Cross-Frame Perception: Identifies editing intent (e.g., addition, swap) and tracks objects across frames, distinguishing camera motion from object motion.
Physics & Temporal Consistency Modeling: Explicitly reasons about physical constraints (gravity, occlusion, contact) and temporal continuity (displacement, scale changes) to ensure kinematic plausibility.
Output Generation: Produces two structured outputs:
- Temporal Bounding Box Sequence: A sequence of normalized bounding boxes ( $\{b_t\}$ ) aligned with keyframes, providing explicit spatial anchors.
- Enriched Instruction: An augmented text description incorporating target attributes, spatial relations, and scene constraints.

3.2. Guide: Box-Guided Mask Generation

The Guide branch transforms the difficult global problem of mask generation into a local refinement problem using the Planner's outputs as hard constraints.

Box-Conditioned Generation: The bounding box sequence serves as a hard constraint, while the enriched instruction and video features act as soft constraints.
Spatiotemporal Self-Attention: The branch shares a hierarchical architecture with the Editor, utilizing self-attention for temporal consistency and cross-attention to inject semantic and spatial priors.
Bidirectional Correction:
- Reverse-Connector: Feeds high-level semantic understanding from the Editor back to the Guide to recover details in thin structures or occluded regions.
- Mask-Connector: Injects refined mask features back into the Editor to ensure low-level spatial guidance is maintained across layers.
Output: Generates a temporally consistent binary mask sequence ( $\{M_t\}$ ).

3.3. Editor: Diffusion-Based Editing

The Editor executes the actual appearance modification and content generation using a diffusion backbone (Wan2.2 5B).

Multi-Condition Fusion: The Editor integrates the enriched instruction (via Qwen-VL features), the precise spatiotemporal masks, and the original video latent representation.
Cross-Attention Mechanism: Specialized cross-attention modules inject the MLLM's organized semantic knowledge and the mask features into the diffusion process.
Compositing: A mask-guided strategy applies edits strictly within the mask regions while preserving the original video outside, ensuring temporal coherence and semantic faithfulness.

Training Strategy

The framework employs a two-stage training approach to reduce data requirements and improve stability:

Modular Training: The Mask (Guide) and Editor branches are trained separately on mixed image/video segmentation and editing datasets.
Joint Training: The branches are jointly trained on a high-quality internal dataset of 100k video editing pairs with precise mask annotations.

3. Key Contributions

Plan–Guide–Edit Paradigm: A novel framework that decouples semantic intent from spatial execution, parsing targets and localizing them in space and time before the diffusion editing process begins.
CoT-Enhanced MLLM Planner: A reasoning module that proactively resolves textual ambiguities and generates physically consistent spatial priors (bounding boxes) and enriched instructions.
Box-Conditioned Mask Prediction: A mechanism that converts explicit spatial priors into high-quality, temporally consistent masks, significantly easing the burden on the diffusion editor.
Bidirectional Collaboration: A design featuring Reverse-Connectors and Mask-Connectors that allow the Editor and Guide to mutually refine each other's outputs, enhancing boundary stability and physical plausibility.

4. Experimental Results

The authors evaluated CoT-Edit against state-of-the-art baselines (e.g., InsV2V, StableV2V, Lucy-1.1, OmniVideo) using both automatic metrics and user studies.

Quantitative Performance: CoT-Edit achieved state-of-the-art performance across multiple metrics, including:
- Visual Quality: Improved FVD (1015.67 vs. baselines >1400), Temporal Consistency (TC), Motion Smoothness (MS), and Aesthetics (AES).
- Instruction Following: Higher CLIPScore and Gemini-rated scores for physical plausibility, spatial relations, and overall editing quality.
- Spatial Reasoning: Notable gains in handling complex instructions involving multiple similar objects and physical laws (e.g., parabolic motion).
Qualitative Improvements: Visual comparisons demonstrated superior target localization, better preservation of non-edited regions, and the ability to generate physically plausible motions (e.g., bouncing balls) that text-only methods failed to achieve.
Ablation Studies:
- CoT: Removing the Chain-of-Thought reasoning led to failures in generating physically plausible motion trajectories. The study compared variants including Qwen3-VL-32B and Gemini 2.5 Pro (both with and without CoT) to verify effectiveness.
- Mask Branch: The inclusion of the Mask branch and bidirectional connectors significantly improved spatial localization and instruction following scores. The study compared configurations using Qwen3-VL-8B as an instruction enhancement module within the Editor branch.

5. Significance and Claims

The paper claims that the primary significance of CoT-Edit lies in its establishment of a reliable semantic–spatial link through structured planning. By explicitly modeling spatial constraints and physical priors before generation, the framework:

Decouples Localization from Generation: Relieving the diffusion model of the burden of spatial grounding, which simplifies the generation task and reduces the need for massive aligned datasets.
Ensures Physical Consistency: The explicit reasoning about scale, kinematics, and contact relationships prevents common failure modes like object drift or impossible physics.
Improves Efficiency and Stability: The modular, staged training approach clarifies learning objectives for each component, leading to more stable training and efficient inference compared to end-to-end text-to-video editing models.

The authors conclude that this approach enables precise, physically consistent, and temporally coherent video editing, addressing the critical gap between high-level natural language instructions and low-level pixel-level execution.