💻 computer science

Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing

This paper introduces Goku, a large-scale dataset of 2 million instruction-aligned video editing pairs covering complex multi-task and structural manipulations, along with the Goku-Edit model and Goku-Bench benchmark, to advance instruction-based video editing capabilities beyond simple appearance changes.

Original authors: Sen Liang, Cong Wang, Zhentao Yu, Fengbin Guan, Zhengguang Zhou, Teng Hu, Youliang Zhang, Yuan Zhou, Xin Li, Qinglin Lu, Zhibo Chen

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sen Liang, Cong Wang, Zhentao Yu, Fengbin Guan, Zhengguang Zhou, Teng Hu, Youliang Zhang, Yuan Zhou, Xin Li, Qinglin Lu, Zhibo Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magic video editor that can change anything in a movie just by you typing a sentence like, "Make the dog run faster and turn the sky purple." For a long time, these magic tools were like clumsy apprentices: they could swap a shirt color or remove a stray object, but if you asked them to move a character across the room or change the camera angle, they would get confused, freeze, or make a mess.

The paper introduces Goku, a massive new project designed to train these magic editors to become true masters. Here is the breakdown of what they did, explained simply:

1. The Problem: The "One-Trick Pony" Datasets

Before this, the training data for these AI video editors was like a library full of books about only one topic: changing the color of a car. If you wanted to learn how to drive a car (move a subject) or change the scenery (camera movement), the library had no books. The AI models were stuck because they only knew how to do simple "paint-by-numbers" tasks.

2. The Solution: The "Goku" Library

The researchers built Goku, a massive dataset containing 2 million video pairs. Think of this as building a giant, high-tech training gym for AI.

What's inside? Unlike previous libraries, this one has complex challenges. It includes videos where the camera pans around, where a person moves from one side of the room to another, and where multiple changes happen at once (e.g., "Swap the shirt AND add a hat").
How did they make it? They didn't just film these videos; they built a robot factory to create them. They used advanced AI to break down complex requests into small, manageable steps. For example, to make a dog run, they first taught the AI how to make a dog run in a single image, then how to make that image move smoothly into a video.
The Quality Control: To ensure the videos weren't garbage, they installed a "triple-check" security system. Every single video was checked three times by a super-smart AI (Gemini) to make sure the instructions were followed, the movement looked real, and the video didn't glitch. They threw away about 88% of the attempts to keep only the best 2 million.

3. The New Tool: Goku-Edit

With this new library, they built a new video editor called Goku-Edit.

The Brain: Instead of just reading text, this editor uses a "Multimodal Large Language Model" (MLLM). Think of this as giving the editor a brain that can actually understand the story and the instructions, not just match keywords.
The Two-Handed Approach: The biggest innovation is that the editor has two "hands" working together:
- Hand A (The Architect): This hand draws a rough map (a mask) of where the changes should happen. It focuses on structure and movement.
- Hand B (The Painter): This hand focuses on the details, colors, and textures.
- Why this matters: By separating the "where" from the "what," the editor doesn't get confused. It knows exactly where to move the dog without accidentally painting the dog's fur blue.
The "Spatial CFG": This is a fancy term for a safety net. It ensures that when the editor makes a change, it doesn't accidentally spill over and ruin the rest of the video. It keeps the changes tight and precise.

4. The Test: Goku-Bench

To prove their new editor is the best, they created a new test called Goku-Bench.

Instead of simple tests, they gave the AI 1,000 difficult challenges, like "Move the camera to the left while the cat jumps."
They used 7 new ways to grade the results, checking things like "Did the physics make sense?" and "Did the camera move smoothly?"
The Result: Goku-Edit beat all other open-source models, improving its ability to follow instructions by up to 8%. It was particularly good at moving things around and handling complex, multi-step requests where other models failed.

Summary

In short, the paper says: "We built a massive, high-quality training library (Goku) that teaches AI how to do complex video editing, not just simple color swaps. We built a new editor (Goku-Edit) that uses a two-part system to understand structure and details separately, and we proved it works better than anything else on a new, harder test (Goku-Bench)."

They did not claim this technology is ready for hospitals or specific clinical uses; they focused entirely on making video editing smarter, more flexible, and more capable of handling complex creative requests.

Technical Summary: Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing

1. Problem Statement

Existing instruction-based video editing (IVE) datasets and models are largely confined to single-task appearance editing (e.g., object removal, attribute swapping, or style transfer). They fail to address the complex, multi-faceted requirements of real-world scenarios, specifically lacking support for:

Structural manipulations: Precise control over subject movement and camera motion.
Multi-task editing: Simultaneous execution of multiple editing instructions (e.g., adding an object while changing the style).
Reference-guided editing: Editing based on user-provided reference images with pose or lighting variations.

Current datasets often suffer from narrow task definitions, insufficient filtering rigor leading to low-quality or static samples, and a lack of benchmarks capable of evaluating these complex structural and multi-task capabilities.

2. Methodology

The paper proposes a comprehensive solution comprising a large-scale dataset (Goku), a robust data synthesis pipeline, a novel editing model (Goku-Edit), and a rigorous benchmark (Goku-Bench).

A. The Goku Dataset (2 Million Pairs)

Goku is a high-quality, instruction-aligned dataset containing 2 million video editing pairs. It expands the scope of IVE to include:

Basic Edits: Add, Remove, Swap, Alter, Style Transfer.
Structural Edits: Subject Movement (action/position variation) and Camera Movement (pan, zoom, etc.).
Complex Edits: Multi-task editing (2–5 tasks simultaneously) and Reference-based editing.

Data Synthesis Pipeline:
To generate this scale of data while maintaining semantic precision and temporal coherence, the authors designed an automated pipeline:

Pre-processing: Curated 1 million high-quality clips from Koala-36M using aesthetic scoring, motion analysis, and OCR-based watermark removal.
Instruction Generation: Leveraged Gemini2.5-Pro (an MLLM) to generate diverse, structured editing instructions and identify operable objects.
Task-Specific Generation:
- Appearance: Used VACE for swapping/altering and a duality approach (Remove $\to$ Swap roles $\to$ Add) for insertion.
- Style Transfer: Decomposed into stylizing the first frame (Flux), extracting depth maps, and propagating style via VACE to ensure temporal consistency.
- Structural: Decomposed complex tasks. For Subject Movement, action variation used Wan2.2, while position variation combined Flux (frame-level shift) and Wan2.2 (video generation). Camera Movement utilized RecamMaster.
- Multi-Task: Executed as a sequential chain of single-task edits.
Progressive Filtering System: A three-tier quality control system powered by Gemini2.5-Pro and geometric checks:
- Tier 1: Source video filtering (aesthetics, motion, watermarks).
- Tier 2: Condition verification (mask IoU, semantic consistency of instructions vs. targets).
- Tier 3: Post-synthesis validation (inter-frame consistency, frequency artifacts, photorealism). This tier filters out ~88% of synthesized samples to ensure high fidelity.

B. Goku-Edit Model Architecture

Goku-Edit is a dual-branch framework designed to handle complex instructions and structural control:

Dual-Branch Design:
- Main Branch: Generates the edited video ( $V_e$ ) conditioned on text, source video, and reference image.
- Auxiliary Branch: Predicts the editing mask ( $M$ ) to provide spatial grounding. It operates on a spatially downsampled version of the source video.
- Encoder: Both branches utilize a frozen Multimodal Large Language Model (MLLM) (Qwen3VL-8B) as the text encoder to enhance semantic understanding of complex instructions.
RoPE-Aligned Spatial Cross-Attention:
- To resolve the spatial misalignment between the high-resolution video branch and the low-resolution mask branch ( $1/n$ resolution), the authors propose scaling the mask branch's position indices by factor $n$ before RoPE computation.
- This maps mask tokens to a unified physical coordinate system, eliminating macro-level offsets and preserving locality bias.
- Bidirectional Attention: Includes both Mask-to-Video (M2V) and Video-to-Mask (V2M) attention. V2M allows the mask branch to refine its predictions based on the evolving state of the main branch, preventing boundary incoherence.
Spatial Enhanced CFG (SpatialCFG):
- A training-free inference strategy that amplifies cross-branch spatial constraints.
- It contrasts coupled predictions (with cross-attention) against decoupled baselines (with null latents) to isolate and amplify the spatial signal injected by the mask branch.
- This suppresses editing spillover and improves boundary consistency without additional training.

C. Goku-Bench

A comprehensive benchmark featuring 1,000 human-verified test cases covering diverse subjects, scenes, and task difficulties (including low-light and fast motion).

Metrics: Includes 7 editing-specific metrics:
1. Physical Rule Fidelity (PR): Adherence to real-world physics.
2. Spatial Relationship Accuracy (SR): Correct spatial arrangement.
3. Instruction Following (IF): Completeness of multi-task execution.
4. Overall Editing Quality (EQ): Visual naturalness and consistency.
5. Subject Motion (SuM): Realism of subject trajectory.
6. Camera Motion (CM): Accuracy of camera movement types.
7. Style Transfer (ST): Feature similarity for style accuracy.

3. Key Results

Evaluated on Goku-Bench, Goku-Edit demonstrates significant improvements over state-of-the-art open-source models (e.g., LucyEdit, InsViE, Omni-Video) and competes favorably with commercial models (Runway Gen-4, Luma Ray3).

Instruction Following: Achieved up to +8% improvement over other open-source models.
Structural & Multi-Task Performance: Significantly outperformed baselines in PR (Physical Rule Fidelity), SR (Spatial Accuracy), SuM (Subject Motion), and CM (Camera Motion), validating the dataset's ability to train models on complex structural tasks.
Ablation Studies:
- Data Quality: Models trained on filtered Goku data significantly outperformed those trained on unfiltered data or existing datasets (InsV2V, InsViE), confirming the value of the progressive filtering system.
- Architecture: Removing the MLLM encoder, dual-branch design, RoPE alignment, or SpatialCFG resulted in measurable drops in performance, particularly in spatial accuracy and instruction following.
- Downsampling Factor: An optimal downsampling factor of $n=4$ was found to balance texture detail and structural signal.
User Study: Goku-Edit ranked first among all methods in a user study regarding instruction following, video quality, and temporal coherence.

4. Significance and Claims

The paper claims that Goku represents a paradigm shift in instruction-based video editing by moving beyond simplistic attribute changes to multi-task and structural manipulations.

Dataset Contribution: It is the first dataset to explicitly cover structural deformations (subject/camera movement) and multi-task editing at a million-scale, addressing a critical gap in current IVE research.
Methodological Contribution: The proposed Goku-Edit architecture demonstrates that decoupling structural control (mask branch) from appearance rendering (main branch), combined with MLLM-based reasoning and spatial alignment, is essential for high-fidelity complex editing.
Benchmark Contribution: Goku-Bench establishes a new standard for evaluation, introducing metrics that specifically target physical plausibility and spatial reasoning, which were previously overlooked.

The authors conclude that their work provides the necessary data, model, and evaluation framework to bridge the gap between current generative AI capabilities and the complex, creative demands of real-world video production.