💻 computer science

VideoCoF: Unified Video Editing with Temporal Reasoner

VideoCoF introduces a unified video editing framework that employs a "see, reason, then edit" Chain-of-Frames approach to predict edit-region latents as explicit reasoning tokens, thereby achieving precise instruction-to-region alignment and motion consistency without requiring user-provided masks while maintaining high efficiency with minimal training data.

Original authors: Xiangpeng Yang, Ji Xie, Yiyuan Yang, Yue Ma, Yan Huang, Min Xu, Qiang Wu

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiangpeng Yang, Ji Xie, Yiyuan Yang, Yue Ma, Yan Huang, Min Xu, Qiang Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a video of a busy street, and you want to edit it. Maybe you want to remove a specific person, add a dog, or change the color of a car.

The Problem with Current AI:
Right now, most AI video editors are like two different types of workers:

The "Expert" Worker: This worker is amazing at precision. If you give them a mask (a digital stencil) saying "remove this person," they do it perfectly. But they are slow and annoying because you have to draw the stencil yourself for every single edit.
The "Generalist" Worker: This worker is fast and doesn't need a stencil. You just say, "Remove the person," and they try. But they are often confused. They might remove the wrong person, or they might accidentally erase the background because they don't know exactly where to look.

The Solution: VideoCoF
The researchers behind VideoCoF (Video Chain-of-Frames) decided to teach the AI a new way of thinking. Instead of just guessing or needing a stencil, they made the AI follow a three-step process: See → Reason → Edit.

Think of it like hiring a Smart Director for your movie instead of a simple camera operator.

The "See, Reason, Edit" Process

Here is how VideoCoF works, using a simple analogy:

1. SEE (The Setup)
The AI looks at the original video. It's like the director watching the raw footage to understand the scene.

2. REASON (The "Ghost" Sketch)
This is the magic part. Before the AI actually changes the video, it first generates a "Reasoning Frame."

The Analogy: Imagine the AI is a painter. Before it paints the final masterpiece, it first draws a faint, gray sketch on the canvas to show exactly where the changes will happen. It's like the AI is saying to itself, "Okay, I see the instruction 'remove the woman in beige pants.' I am going to highlight her in a ghostly gray color first to make sure I'm looking at the right person."
Why this matters: By forcing the AI to "think" about where the edit is before doing it, it solves the confusion problem. It stops the AI from erasing the wrong person.

3. EDIT (The Final Touch)
Once the AI has "reasoned" and highlighted the correct spot, it performs the actual edit (removing the woman, adding the dog, etc.) with high precision.

The "Time-Travel" Trick (RoPE Alignment)

There is another cool trick in the paper. Usually, if you train an AI on short videos (like 30 seconds), it gets confused when you ask it to edit a long video (like 5 minutes). It loses track of time and the motion gets messy.

VideoCoF uses a clever numbering system (called RoPE Alignment) to fix this.

The Analogy: Imagine you are teaching a child to count steps. If you teach them "Step 1, Step 2, Step 3" for a short walk, they might get lost on a long hike.
VideoCoF teaches the AI to reset its "step counter" for the new video parts. It tells the AI: "The original video is steps 1 to 30. The 'thinking' part is step 0. The new edited part starts fresh at step 1 again."
The Result: This allows the AI to edit videos that are 16 times longer than what it was trained on, without the motion getting blurry or the characters teleporting.

Why is this a big deal?

No Stencils Needed: You don't have to draw masks. You just type what you want, and the AI figures out the "where" and "what" on its own.
Super Efficient: The researchers trained this powerful model on only 50,000 video examples. Other top models needed 1 million examples to get even close to this quality. It's like learning to drive a race car by reading a small manual instead of driving for 10 years.
Handles Complex Scenes: If there are two people wearing the same shirt, VideoCoF can tell them apart (e.g., "Remove the one on the left") because it "reasoned" about the location first.

In Summary:
VideoCoF is like giving a video editor a thinking cap. Instead of blindly guessing or needing a human to point at the screen, the AI first pauses to "think" about exactly where the action should happen, sketches a mental map, and then executes the edit perfectly. This makes video editing faster, smarter, and capable of handling much longer and more complex videos than ever before.

1. Problem Statement

Existing video editing methods face a fundamental trade-off between precision and unification:

Expert Models: These models (e.g., those using adapters) achieve high precision by relying on user-provided masks or specific priors. However, they are task-specific, require additional inputs, and lack a unified framework for diverse editing tasks.
Unified In-Context Learning Models: These models (e.g., ICVE, UNIC) concatenate source and target tokens temporally to perform editing without masks. While unified and mask-free, they suffer from weak instruction-to-region mapping. Without explicit spatial cues, they struggle with multi-instance recognition, spatial reasoning (e.g., "remove the person on the left"), and precise localization, leading to hallucinations or editing the wrong objects.

The core challenge is: Can we achieve the precision of expert models and the unification of in-context learning without requiring user-provided masks?

2. Methodology: VideoCoF

The authors propose VideoCoF, a framework inspired by Chain-of-Thought (CoT) reasoning in Large Language Models. Instead of directly generating the edited video, VideoCoF enforces a "See $\rightarrow$ Reason $\rightarrow$ Edit" procedure.

A. Chain of Frames (CoF)

The core innovation is the introduction of Reasoning Tokens (edit-region latents) that the model must predict before generating the target video tokens.

Process:
1. See: The model receives the source video tokens.
2. Reason: The model predicts a "reasoning frame" (a visual grounding map) that highlights where the edit should occur based on the text instruction.
3. Edit: Using the source and the predicted reasoning map, the model generates the final edited video.
Visual Grounding Format: The authors found that a soft, gradually highlighted grayscale region is the most effective format for reasoning tokens. This acts as a visual "thought process" that explicitly links the text instruction to the spatial location, replacing the need for manual masks.
Architecture: The framework uses a Video Diffusion Transformer (VideoDiT). The input sequence is a temporal concatenation of:
- Source Video Latents ( $z_s$ )
- Reasoning Video Latents ( $z_r$ )
- Target/Edit Video Latents ( $z_e$ )
  During training, only the reasoning and target parts are noised and denoised; the source remains clean to anchor the context.

B. RoPE Alignment Strategy for Length Extrapolation

A critical component for handling variable video lengths is a novel Rotary Position Embedding (RoPE) design.

The Problem: Naive temporal concatenation (e.g., indices $0 \to 2F-1$ ) causes the model to overfit to a fixed mapping between source and target frames, failing to generalize to longer videos. It also causes "index collisions" where the reasoning frame and the first target frame share the same temporal index, leading to artifacts.
The Solution: The authors reset the temporal indices to avoid collision and ensure alignment:
- Source Video: Indices $[1, F]$
- Reasoning Frame: Index $0$ (isolated to provide spatial guidance without temporal interference).
- Target Video: Indices $[1, F]$
  This design allows the model to generalize to video lengths far exceeding the training duration (e.g., $4\times$ to $16\times$ ) while maintaining motion alignment.

C. Data Curation

To train this reasoning capability, the authors constructed a high-quality dataset of 50k video pairs (Source, Reasoning, Target).

Pipeline: They utilized a multi-stage pipeline involving:
- Qwen-VL for multi-instance identification.
- Grounding-SAM2 for precise instance segmentation.
- Minimax-Remover and VACE-14B (guided by GPT-4o) to generate synthetic editing pairs (addition, removal, swap, style transfer).
Filtering: A rigorous filtering process using Dover Score (aesthetic) and VIE Score (fidelity) ensures high-quality training data.

3. Key Contributions

Chain of Frames (CoF): The first framework to introduce a "reasoning-then-editing" paradigm for video diffusion models, enabling precise instruction-to-region alignment without user masks.
RoPE Alignment Strategy: A novel positional encoding scheme that resolves index collisions and enables robust length extrapolation (up to $16\times$ training length) while preserving motion consistency.
VideoCoF-Bench: A new benchmark containing 200 high-quality videos with complex, instance-level editing instructions (multi-instance removal, addition, swapping, style transfer) to evaluate fine-grained reasoning.
Data Efficiency: Demonstrates that high-performance video editing can be achieved with only 50k training pairs, significantly outperforming baselines trained on millions of samples.

4. Experimental Results

The model was evaluated on VideoCoF-Bench against state-of-the-art baselines (ICVE, VACE, LucyEdit, etc.).

Quantitative Performance:
- Instruction Following: VideoCoF achieved 8.97 (GPT-4o score), significantly outperforming ICVE (7.79) and VACE (7.47).
- Success Ratio: VideoCoF achieved 76.36%, compared to 57.76% for ICVE and 26.60% for VACE.
- Data Efficiency: VideoCoF (50k data) outperformed ICVE (1M pre-training + 150k fine-tuning) in instruction following and success rate.
Qualitative Performance:
- Multi-instance Handling: Successfully distinguished between similar objects (e.g., removing the "man on the left" vs. "man on the right"), whereas baselines often failed or edited the wrong instance.
- Length Extrapolation: The model successfully edited videos up to 513 frames (16x the 33-frame training length) with minimal degradation in motion alignment or visual quality, a capability where naive baselines failed completely.
Ablation Studies:
- Removing the "Reasoning" step (Naive Temporal In-Context) dropped the success ratio by ~10%.
- Using the specific RoPE design ( $[1-F, 0, 1-F]$ ) was crucial for motion alignment and preventing artifacts in the first frame.
- The progressive gray mask format for reasoning tokens yielded the best results compared to black or red masks.

5. Significance

VideoCoF represents a paradigm shift in video editing by bridging the gap between generative reasoning and spatial precision.

Unified Framework: It eliminates the need for task-specific masks or adapters, offering a single model capable of diverse editing tasks (add, remove, swap, style).
Scalability: The ability to extrapolate to extreme video lengths ( $16\times$ ) makes it practical for real-world applications where video duration varies.
Efficiency: By proving that a "reasoning-first" approach yields superior results with minimal data (50k vs. millions), it suggests a more efficient path for training future video foundation models, reducing the computational cost and data requirements for high-fidelity editing.

In summary, VideoCoF solves the "mask-free precision" problem by forcing the model to "think" about where to edit before what to edit, achieving state-of-the-art performance with remarkable data efficiency.