💻 computer science

ImVideoEdit: Image-learning Video Editing via 2D Spatial Difference Attention Blocks

ImVideoEdit is an efficient video editing framework that achieves high-fidelity, temporally consistent results by learning exclusively from image pairs through a decoupled spatiotemporal approach, utilizing a Predict-Update Spatial Difference Attention module and a Text-Guided Dynamic Semantic Gating mechanism while freezing pre-trained 3D attention modules.

Original authors: Jiayang Xu, Fan Zhuo, Majun Zhang, Changhao Pan, Zehan Wang, Siyu Chen, Xiaoda Yang, Tao Jin, Zhou Zhao

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiayang Xu, Fan Zhuo, Majun Zhang, Changhao Pan, Zehan Wang, Siyu Chen, Xiaoda Yang, Tao Jin, Zhou Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical video camera that can generate incredible movies from scratch. But now, you want to edit an existing movie: maybe you want to change the weather from a stormy night to a sunny morning, or swap a vintage car for a futuristic drone, all while keeping the actors' movements and the camera's shake exactly the same.

Doing this is usually incredibly hard, expensive, and slow. It's like trying to rewrite a novel while the author is still typing it, without messing up the plot or the characters' voices.

Enter ImVideoEdit. This paper introduces a clever new way to teach computers how to edit videos, and it does so with a surprising twist: it learns entirely from pictures, not videos.

Here is the simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Heavy Video" Bottleneck

Usually, to teach a computer to edit videos, you need thousands of hours of "Before and After" video clips.

The Old Way: Imagine trying to teach a student to fix a moving car engine by showing them thousands of videos of mechanics fixing engines. It takes forever, costs a fortune, and the student might get confused by the moving parts.
The Result: Existing models often get "motion sickness." When they try to change one thing (like the sky), the whole video starts flickering, the background drifts away, or the people start melting.

2. The Big Idea: "Freeze the Motion, Fix the Picture"

The authors realized something brilliant: Changing the look of a video is mostly about changing the picture, not the movement.

If you change a car from red to blue, the car still drives the same way.
If you change the weather, the rain still falls the same way.

So, instead of teaching the computer with expensive videos, they taught it using 13,000 pairs of static images (a "before" photo and an "after" photo).

The Analogy: Think of the video model as a dance instructor who already knows how to move perfectly (the "temporal dynamics"). ImVideoEdit doesn't try to teach the instructor how to dance again. Instead, it just teaches the instructor how to change their costume while they are dancing.

3. The Secret Sauce: The "Predict-Update" Mechanism

How do you change the costume without stopping the dance? The paper introduces a special module called Predict-Update Spatial Difference Attention.

The Analogy: Imagine you are painting a moving train.
1. Predict Phase: You first look at the train and guess where the new paint should go (a rough sketch).
2. Update Phase: You then look at the difference between your rough sketch and the actual train, and you carefully fill in the tiny details (the high-frequency stuff like sparks or texture).
3. The Magic: This happens in a "parallel lane." The main video engine (the train) keeps moving smoothly, while this new "painting lane" works on the details. Because they are separate, the train doesn't wobble.

4. The "Smart Gatekeeper": Text-Guided Gating

Sometimes, you only want to change part of the scene. If the prompt says "Make the drone broken," you don't want to break the clouds or the people in the background.

The Analogy: Think of a bouncer at a club. The "Text-Guided Dynamic Semantic Gating" is the bouncer. It reads your instruction ("Break the drone") and decides exactly which pixels get the "change" permission and which ones stay locked down. It ensures the edit is precise and doesn't accidentally ruin the rest of the scene.

5. The Results: Fast, Cheap, and Amazing

Because they didn't need to process millions of video hours, the training was incredibly fast and cheap.

The Stats: They trained it for just 5 days (5 epochs) on a relatively small dataset of images.
The Outcome: The result is a model that edits videos as well as (and sometimes better than) massive models that took months to train on huge video datasets. It keeps the video stable (no flickering) and follows your instructions perfectly.

Summary

ImVideoEdit is like a specialized costume designer for video.

It doesn't try to relearn how to film a movie (the camera work and movement are already perfect).
It learns how to swap outfits, change the scenery, or fix props just by looking at photos.
It uses a "Predict-Update" system to make sure the changes are smooth and a "Smart Gatekeeper" to make sure only the right things get changed.

The result? You can now edit videos with the precision of a surgeon and the speed of a photo editor, without needing a supercomputer or a library of video clips.

1. Problem Statement

Current video editing models based on 3D Diffusion Transformers (3D DiTs) face three critical challenges:

Data Scarcity & Cost: Training robust video editing models requires massive, high-quality paired video datasets (source video, target video, and text instructions), which are prohibitively expensive and time-consuming to construct.
Temporal Instability: Directly fine-tuning or injecting features into the tightly coupled 3D spatio-temporal attention modules of pre-trained models often disrupts learned motion priors. This leads to artifacts such as background drift, temporal flickering, and structural distortions.
Reliance on External Masks: Many existing solutions rely on external segmentation models or manual masks to isolate editing regions, compromising the elegance of end-to-end, text-driven editing and limiting zero-shot capabilities.

The core insight of the paper is that most video editing tasks are fundamentally decoupled spatiotemporal processes: the temporal dynamics (motion) should be preserved from the pre-trained model, while only the spatial content needs to be modified.

2. Methodology

The authors propose ImVideoEdit, a framework that learns video editing capabilities entirely from image pairs (treating images as single-frame videos), thereby bypassing the need for video training data.

A. Dataset Construction

To overcome data scarcity, the authors constructed a curated dataset of 13,000 high-quality image pairs using a three-stage pipeline:

Scene-Conditioned Prompt Construction: Generating diverse base scenes and editing tasks (e.g., style transfer, object addition/removal) using LLMs (GPT-5.3) and validating them with VLMs (Gemini 3.1 Pro) for semantic plausibility.
Paired Image Synthesis: Using text-to-image (Qwen-Image) and image editing models to generate source and edited image pairs with consistent visual fidelity.
Data Filtering: Automated and human-in-the-loop filtering to ensure instruction faithfulness and visual quality.

B. Architectural Design

The framework is built upon a frozen pre-trained 3D DiT backbone (Wan2.1-T2V-1.3B) to preserve robust spatiotemporal priors. The core innovation lies in two parallel modules:

Predict-Update Spatial Difference Attention Module:
- Concept: This module decouples spatial learning from temporal dynamics. It treats images as single-frame videos ( $F=1$ ) and processes them through a dedicated 2D attention branch.
- Predict Phase: Extracts a coarse-grained spatial structural alignment from the reference latent.
- Update Phase: Calculates the high-frequency spatial residual (difference) between the predicted state and the target. This two-step process allows for progressive refinement of spatial features without interfering with the frozen 3D temporal layers.
Text-Guided Dynamic Semantic Gating:
- Problem: The spatial module lacks inherent text awareness, and naive addition of residuals can cause global structural shifts.
- Solution: A gating mechanism uses cross-attention between the spatial residuals and text embeddings to generate a dynamic mask ( $G \in (0,1)$ ).
- Function: This gate adaptively modulates the intensity of the structural update, ensuring that edits are localized to the regions specified by the text prompt while preserving the rest of the scene.

3. Key Contributions

Video-Free Training Paradigm: Demonstrates that high-fidelity video editing can be achieved by training exclusively on static image pairs, significantly reducing computational costs and data acquisition barriers.
Predict-Update Spatial Difference Attention: A novel architecture that extracts and injects spatial differences in a coarse-to-fine manner, effectively decoupling spatial reconstruction from temporal consistency.
Text-Guided Dynamic Semantic Gating: An implicit, mask-free mechanism that enables precise, prompt-driven semantic modulation without relying on external segmentation models.
Curated Dataset: The release of a 13K image-pair dataset specifically designed for learning spatial transformations in video editing.

4. Experimental Results

The model was trained for only 5 epochs on 13K image pairs using 8 NVIDIA A100 GPUs (consuming ~20GB VRAM per GPU), yet it outperforms or matches much larger models.

Quantitative Performance (VLM Evaluation):
- ImVideoEdit (1.3B parameter base) achieved a Total Score of 65.24, outperforming VACE-1.3B (56.79) and approaching the performance of the 13B/14B parameter models like ICVE (65.04) and DITTO (61.82).
- It significantly excelled in Instruction Adherence and Visual Fidelity, proving that image-based training is sufficient for complex semantic editing.
Temporal Consistency:
- Despite training on images, the model maintained strong temporal consistency (0.990 on VBench Motion Smoothness), validating that freezing the 3D backbone successfully preserves motion dynamics.
Qualitative Analysis:
- In tasks like background replacement, object removal, and style transfer, ImVideoEdit produced coherent, localized edits. In contrast, baselines often suffered from global appearance shifts, identity blending, or partial edits.
User Study:
- Human evaluators rated ImVideoEdit highly (3.06/5.0), comparable to the 5B parameter Kiwi-Edit (3.08) and significantly better than VACE-1.3B (2.18), confirming its perceptual quality.

5. Significance

ImVideoEdit represents a paradigm shift in video editing research by proving that temporal dynamics and spatial editing can be learned separately. By leveraging the "video-free" training approach, it:

Democratizes Access: Makes high-quality video editing accessible to researchers with limited computational resources (trainable on a single 3090 GPU).
Solves Data Bottlenecks: Eliminates the need for expensive, large-scale paired video datasets.
Improves Stability: Avoids the "motion collapse" often seen when fine-tuning 3D DiTs directly, offering a more robust solution for real-world content creation.

The paper concludes that the decoupling of spatial learning (via image pairs) and temporal preservation (via frozen 3D priors) is a highly effective strategy for the next generation of efficient, text-driven video editing tools.