💻 computer science

VicEdit: Learning to Edit Videos from Visual In-Context Examples

The paper introduces VicEdit, a unified framework that advances video editing by leveraging a new large-scale dataset (VicEdit-400K) and novel techniques like Modality-Adaptive Semantic Distillation and Dual-Context Injection to effectively integrate visual in-context examples with textual instructions for superior editing performance.

Original authors: Yuji Wang, Teng Hu, Yuheng Chen, Ran Yi, Han Feng, Weijian Cao, Chengjie Wang, Lizhuang Ma, Jiangning Zhang

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuji Wang, Teng Hu, Yuheng Chen, Ran Yi, Han Feng, Weijian Cao, Chengjie Wang, Lizhuang Ma, Jiangning Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For years, the dream of editing video has been to simply tell a computer what to do. If you wanted to change the color of a car in a movie scene or replace a cloudy sky with a sunset, you could type a sentence like "make the sky orange," and the software would try its best. This approach, known as instruction-based editing, has made remarkable progress. It relies on powerful artificial intelligence models that understand language and can generate images and videos from scratch. However, a fundamental problem remains: words are often too vague to capture the specific look and feel of a scene. A text description can say "red car," but it cannot easily convey the exact shade of red, the way the light hits the metal, or the specific motion of the vehicle. When a computer tries to guess these details from a sentence alone, the result often looks wrong, with colors that drift, objects that appear in the wrong place, or movements that feel stiff and unnatural.

To solve this, researchers have begun to explore a different way of teaching computers: showing them examples instead of just telling them. This concept, called in-context learning, is similar to how a human apprentice learns a craft. Instead of reading a manual on how to paint a wall, the apprentice watches a master painter apply a specific brushstroke to a specific surface. By observing the relationship between the original wall and the finished result, the apprentice learns the precise technique. In the world of video editing, this means providing the computer with visual references—such as a single image, a pair of images showing a before-and-after, or even a short video clip of the desired change. The challenge has been that no single system could handle all these different types of visual clues at once, and there was no large collection of examples to teach the system how to use them effectively.

A team of researchers has now bridged this gap with a new system called VicEdit. Their work represents a significant shift from relying solely on text descriptions to using visual examples as the primary guide for editing. To build this system, the team first created a massive library of training data. They generated 400,000 high-quality examples, a dataset they named VicEdit-400K. This collection is unique because it covers ten different types of editing tasks, from changing the style of a whole scene to adding or removing specific objects. Crucially, each example in this library is paired with the right kind of visual reference: a single image for style changes, a pair of images for spatial changes, or a pair of videos for complex motion. This vast library allowed them to teach the computer how to interpret visual cues with a level of precision that text alone could never provide.

The core of their new system is a method that allows the computer to adapt its understanding based on the type of visual clue it receives. When the computer sees a single image, it learns to focus on textures and colors. When it sees a pair of images, it learns to understand how objects move from one position to another. When it sees a pair of videos, it learns to follow the flow of time and motion. The researchers designed a process that extracts these specific lessons from the visual examples and combines them with the written instructions. This ensures that the computer does not just follow the text blindly but uses the visual examples to anchor the changes in reality. For instance, if a user asks to change a water bottle into a glowing galaxy, the text provides the idea, but a visual example of a glowing galaxy ensures the computer knows exactly what that looks like, preventing it from creating a generic or distorted result.

The results of this approach are striking. When tested against existing methods, the new system consistently produced higher-quality videos that were more faithful to the user's intent. In tasks where other systems struggled to keep objects in the right place or maintain the correct style, this new method succeeded. It could seamlessly blend a new object into a scene, change the background without distorting the foreground, or apply complex artistic styles that looked natural and coherent. The system proved that by showing the computer what to do, rather than just telling it, the quality of the editing improves dramatically. This work establishes a new standard for video editing, demonstrating that visual examples are a powerful and necessary tool for guiding artificial intelligence. The researchers have made their dataset and system available for others to use, opening the door for more precise and creative video editing tools in the future.

Technical Summary: VicEdit: Learning to Edit Videos from Visual In-Context Examples

Problem Statement

Instruction-driven video editing has advanced significantly through text-based interfaces, leveraging large language models (LLMs) and diffusion architectures. However, the authors identify a fundamental limitation: unimodal textual instructions inherently struggle to convey fine-grained textures, precise spatial transformations, and complex temporal dynamics. Text-only baselines often suffer from:

Stylistic Drift: Linguistic descriptors fail to precisely anchor specific visual textures or styles.
Spatial Ambiguity: Text specifies what to edit but not where, lacking the spatial anchors provided by image pairs.
Static Dynamics: Complex motion patterns in creative editing exceed linguistic capacity, a gap not addressed by existing reference-based works that typically support only static single-image modalities.

Furthermore, existing datasets and benchmarks are confined to text or single-image drivers, lacking the large-scale, multi-modal resources necessary to train models capable of heterogeneous visual reasoning.

Methodology: The VicEdit Framework

The paper proposes VicEdit, a unified framework that transitions video editing from descriptive "telling" to prescriptive "showing" via Visual In-Context Editing. This paradigm accepts heterogeneous reference inputs: Single Images ( $R_{si}$ ), Image Pairs ( $R_{ip}$ ), and Video Pairs ( $R_{vp}$ ).

1. Data Curation: VicEdit-400K

To support this paradigm, the authors introduce VicEdit-400K, the first large-scale dataset for visual in-context video editing.

Scale & Diversity: Contains 400K high-quality samples across 10 distinct task types (e.g., style transfer, object manipulation, spatiotemporal synthesis) and three reference modalities.
Construction Pipeline: An automated four-stage pipeline ensures quality:
1. Source Data Filtering: Uses LAION-Aesthetic and Qwen3-VL to filter for semantic consistency and aesthetic quality.
2. Instruction Synthesis: MLLMs generate modality-aware instructions strictly aligned with visual references.
3. Visual Reference Generation: Utilizes state-of-the-art generative models for images and a Semantic Knowledge Graph with spectral clustering for video pairs to ensure temporal fidelity.
4. Semantic Verification: Multi-dimensional verification via MLLMs to filter for instruction following, temporal consistency, and reference alignment.

2. Model Architecture

VicEdit integrates a frozen Multi-Modal Large Language Model (MLLM) for semantic understanding with a Video Diffusion Transformer (DiT) for generation, connected by two lightweight modules:

Modality-Adaptive Semantic Distillation (MASD):
- Challenge: Existing methods use fixed learnable queries, which cannot adapt to the structural differences between modalities (e.g., spatial shifts in image pairs vs. global textures in single images).
- Solution: MASD modulates base queries ( $Q_{base}$ ) with learnable modality-specific embeddings ( $E_m$ ) via an additive shift: $Q_{vis} = Q_{base} + E_m$ .
- Mechanism: This decomposes the search space into a common editing manifold and modality-specific sub-regions, allowing the model to distill modality-specific semantic tokens ( $T_{vis}$ ) from the MLLM hidden states.
Dual-Context Injection (DCI):
- Challenge: Concatenating visual tokens and text instructions often leads to misalignment, where the text branch extracts generic features that ignore the specific perceptual instantiation provided by the visual reference.
- Solution: DCI introduces a semantic shift mechanism. It uses the modality-conditioned queries ( $Q_{vis}$ ) from MASD to generate a shift vector ( $\Delta_{inst}$ ) via a lightweight MLP.
- Mechanism: This shift calibrates the base instruction queries ( $Q_{inst}$ ) to produce visually-aware queries ( $Q'_{inst} = Q_{inst} + \Delta_{inst}$ ). These are then used to extract instruction tokens ( $T_{inst}$ ). Finally, $T_{vis}$ and $T_{inst}$ are concatenated into dual-context tokens ( $T_{dual}$ ) and injected into the Video DiT via cross-attention.

3. Training Strategy

VicEdit is trained in three progressive stages on the Wan-TI2V-5B backbone:

Base Editing: Fine-tuning the DiT on text-instructed tasks to establish fundamental instruction-following.
In-Context Branch: Training the MASD and DCI modules (with DiT frozen) to learn balanced modality embeddings and semantic shifts.
Joint Fine-Tuning: Unfreezing the DiT to jointly optimize the entire pipeline, aligning visual conditioning signals with the generative process.

Key Contributions

Paradigm Shift: Formulation of a unified Visual In-Context Video Editing paradigm that leverages dense semantic cues from single images, image pairs, and video pairs.
VicEdit-400K: The creation of the first large-scale dataset (400K samples) covering three reference modalities and ten task types, providing a systematic foundation for visual in-context learning.
VicEdit Framework: A unified architecture featuring Modality-Adaptive Semantic Distillation (MASD) for extracting modality-specific semantics and Dual-Context Injection (DCI) for synergistically fusing visual and textual contexts.
VicEdit-Bench: A standardized evaluation protocol comprising 200 zero-shot instances to rigorously assess heterogeneous reference following.

Experimental Results

Extensive evaluations were conducted on OpenVE-Ext (basic editing) and VicEdit-Bench (visual in-context editing).

Quantitative Performance: VicEdit achieves state-of-the-art (SOTA) performance across all metrics. On VicEdit-Bench, it scores 3.45 overall, surpassing the second-best method (Kiwi-Edit, 2.97) by a significant margin (16% relative improvement). It outperforms text-only baselines (VACE, Lucy-Edit, DITTO, NovaEdit, VideoCoF) and the single-image reference baseline (Kiwi-Edit) across all subtasks, including Global Style, Local Change, and Creative Edit.
Qualitative Analysis: Visual comparisons demonstrate that VicEdit resolves stylistic drift, spatial ambiguity, and static dynamics issues prevalent in baselines. For instance, in "local_add" tasks, VicEdit correctly handles spatial relationships and scale, whereas baselines often produce structural collapse or poor fusion.
Ablation Studies:
- MASD: Significantly enhances detail fidelity by adaptively distilling modality-specific priors, preventing semantic blurring.
- DCI: Crucial for refining instruction-following precision by calibrating text queries against visual context, resolving natural language ambiguities.
- Training Strategy: The phased approach ensures the model maintains robust base editing capabilities while mastering complex in-context tasks.

Significance and Claims

The paper claims that VicEdit establishes visual in-context learning as a powerful and controllable paradigm for video editing. By bridging the perceptual gap between text descriptions and visual reality, the framework enables fine-grained controllability that text-only instructions cannot achieve.

The authors emphasize that their work addresses systemic deficiencies in both architectural reasoning (lack of flexibility in heterogeneous contexts) and data resources (lack of large-scale multi-modal datasets). While the current design targets semantic-level editing, the paper notes that future work could explore pixel-level precision and further dataset scaling. The proposed framework and dataset are presented as a systematic foundation for advancing the field from descriptive instruction to prescriptive visual demonstration.