💻 computer science

MiVE: Multiscale Vision-language features for reference-guided video Editing

MiVE introduces a reference-guided video editing framework that leverages hierarchical, multiscale features from Qwen3-VL within a unified self-attention Diffusion Transformer to overcome modality gaps and spatial detail loss, achieving state-of-the-art performance in preserving motion and executing precise edits.

Original authors: Tong Wang, Meng Zou, Chengjing Wu, Xiaochao Qu, Luoqi Liu, Xiaolin Hu, Ting Liu

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tong Wang, Meng Zou, Chengjing Wu, Xiaochao Qu, Luoqi Liu, Xiaolin Hu, Ting Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a home video of your friend walking down the street. You want to change their hair color to bright pink, but you want to keep their walk, the background, and everything else exactly the same. This is the challenge of Reference-Guided Video Editing.

The paper introduces a new tool called MiVE (Multiscale Vision-language features for reference-guided video Editing) that solves this problem better than current methods, including expensive commercial systems.

Here is how MiVE works, explained through simple analogies:

The Problem: The "Translator" vs. The "Architect"

Current video editing tools usually try to do two things separately:

The Translator: A system that reads your text instructions (e.g., "make hair pink").
The Architect: A system that looks at the video and the reference photo to understand what to change.

The paper argues that these two systems often talk past each other. The "Translator" understands the idea of pink hair but doesn't know exactly where the hair is. The "Architect" sees the hair but might not fully grasp the specific instruction. When they try to combine their work late in the process, the result is often blurry, inconsistent, or changes things you didn't want to change.

The Solution: MiVE's "All-Hands Meeting"

MiVE changes the game by using a single, powerful brain (a Vision-Language Model called Qwen3-VL) to do everything at once. But here is the secret sauce: MiVE listens to different "voices" within that brain.

The authors discovered that deep learning models have layers, like floors in a skyscraper:

The Lower Floors (Early Layers): These are like architects with a magnifying glass. They see tiny, specific details: the exact shape of a hair strand, the texture of a shirt, or the edge of a shadow.
The Top Floor (Final Layers): This is like the CEO. They understand the big picture and the meaning: "This is a person," "This is pink hair," "This is a sunny day."

Old methods only listened to the CEO (the final layer). They knew what to do but missed the details, resulting in fuzzy edits.
MiVE holds a meeting where both the architects with magnifying glasses and the CEO speak at the same time.

How It Works: The "Unified Stage"

Instead of having the text, the reference photo, and the video talk to each other through a complicated relay race (which causes delays and errors), MiVE puts them all on a single stage.

The Input: You give MiVE the original video, a text instruction, and a reference photo (a picture of what you want the result to look like).
The Mix: MiVE takes the "magnifying glass" details and the "big picture" meaning from the reference and instruction.
The Dance: It mixes all this information together in one big pool. The video frames don't just "listen" to the instructions; they dance with them. This ensures that when the video changes the hair color, it knows exactly which pixels to touch and which to leave alone, preserving the original movement perfectly.

The Results: Why It Wins

The paper tested MiVE against other top academic models and a famous commercial system (Kling O1).

In Simple Scenarios: MiVE could change an object's color or remove a person without blurring the background.
In Complex Scenarios: When the video had fast movement, shadows, or people walking behind objects, MiVE was the only one that kept the person's face recognizable and the lighting realistic.

The authors say MiVE is the best because it doesn't just guess; it uses the fine details (from the early layers) to ensure precision and the big ideas (from the final layers) to ensure the instruction is followed correctly.

Summary

Think of MiVE as a master editor who doesn't just read the script (the text) and look at the photo (the reference) separately. Instead, MiVE has a superpower: it can see the entire movie and the entire script at the same time, paying attention to both the broad story and the tiny details simultaneously. This allows it to make changes that look natural, stay consistent, and follow your instructions perfectly.

Technical Summary: MiVE (Multiscale Vision-language features for reference-guided video Editing)

1. Problem Definition

The paper addresses reference-guided video editing, a task that requires generating an edited video sequence based on three inputs: a source video ( $x_{src}$ ), a text instruction ( $x_{text}$ ), and a reference image ( $x_{ref}$ ). The reference image is typically the first frame of the source video, modified via an external image editing model to reflect the desired changes.

The core challenge lies in balancing two competing objectives:

Faithful Propagation: The model must accurately propagate the visual attributes and appearance changes specified in the reference image across all frames of the video.
Preservation of Integrity: The model must maintain the original motion dynamics, semantic integrity, and unedited content of the source video, even in regions occluded in the reference image.

Existing methods face limitations in multimodal fusion:

Decoupled Encoders: Approaches using separate encoders for text (e.g., T5) and vision (e.g., SigLIP) suffer from modality gaps, where late-stage cross-attention fails to bridge the semantic divide between instructions and visual content.
Unified Encoders with Single-Layer Output: Recent Vision-Language Models (VLMs) used as unified encoders (e.g., Kling O1) typically rely solely on final-layer representations. The authors observe that this discards fine-grained spatial details essential for precise editing, as deeper layers encode global semantics while earlier layers capture localized spatial structures.

2. Methodology: MiVE

The authors propose MiVE, a framework that repurposes VLMs as multiscale feature extractors and integrates them into a unified self-attention Diffusion Transformer (DiT). The architecture consists of three core components:

A. Multiscale Context Extraction

Motivated by the observation that VLM layers encode complementary information hierarchically, MiVE extracts features from both the first and last layers of a pre-trained VLM (specifically Qwen3-VL-8B).

Diagnostic Insight: Through a cross-modal attention analysis, the authors demonstrate that early VLM layers precisely localize spatial details (e.g., human silhouettes), while final layers capture holistic semantics.
Implementation: Features from the first layer ( $\phi_1$ ) and last layer ( $\phi_L$ ) are projected into the DiT latent space via lightweight adapters. These multi-level features are concatenated and fused to create stationary conditional tokens ( $c$ ) that serve as a semantic anchor throughout the denoising process.

B. Reference-Aware Latent Encoding

To preserve structural integrity while anchoring generation to the reference appearance, the method encodes visual inputs (source video, target video, and reference image) into a shared latent space using a pre-trained VAE.

Input Construction: During training and inference, the model constructs a unified input by concatenating two temporally augmented branches along the channel dimension: a "noisy target" branch and a "control" branch.
Anchoring: The clean reference latent ( $z_{ref}$ ) is prepended to both branches. This ensures the model maintains a consistent appearance anchor throughout the diffusion process, preventing the reference style from drifting.

C. Unified Self-Attention Backbone

Unlike decoupled architectures that fuse modalities via separate cross-attention branches, MiVE processes all tokens—conditional context ( $c$ ) and visual latent tokens ( $v$ )—within a single unified self-attention manifold.

Mechanism: The conditional tokens and visual tokens are concatenated into a single sequence $u^{(0)} = [c; v]$ .
Adaptive Modulation: The model employs per-token adaptive layer normalization (AdaLN). "Clean" tokens (context and reference patches) are modulated as fully denoised ( $t=0$ ), acting as stable anchors, while "noisy" tokens (remaining video patches) are modulated according to the current timestep.
Benefit: This design eliminates the modality mismatch inherent in cross-attention designs, enabling bidirectional fusion and deep integration of fine-grained spatial details with global semantic context within a shared space.

3. Key Contributions

Multiscale Feature Extraction: The paper identifies and leverages the hierarchical nature of VLMs, extracting complementary features from early (spatial) and late (semantic) layers to address the trade-off between detail preservation and instruction adherence.
Unified Self-Attention Architecture: MiVE introduces a unified token modeling approach that processes conditional and visual tokens jointly, avoiding the limitations of asymmetric cross-attention and facilitating better modality alignment.
Reference-Aware Latent Encoding: The method employs a specific latent concatenation strategy that prepends the reference frame to both noisy and control branches, ensuring stable appearance propagation without requiring explicit segmentation masks.

4. Experimental Results

The authors evaluated MiVE on a curated benchmark of 60 videos (720p), divided into a Simple Subset (localized editing) and a Complex Subset (holistic scene transformations, rapid motion, occlusions).

Quantitative Performance:
- Automated Evaluation: Using Gemini-3-Flash across six dimensions (Instruction Adherence, Content Consistency, Temporal Smoothness, Physical Realism, Visual Aesthetics, Style Coherence), MiVE achieved the highest scores in the Complex Subset across all dimensions and ranked first or second in the Simple Subset.
- Human Preference: In a user study with 30 participants, MiVE ranked highest in human preference for both subsets, outperforming leading academic methods (VACE, LucyEdit, VideoCof, Wan-Animate) and the commercial system Kling O1.
Ablation Studies:
- Architecture: Replacing the unified self-attention with decoupled encoders or cross-attention mechanisms resulted in significant performance drops, validating the necessity of the unified manifold.
- Layer Selection: Using only early layers or only late layers yielded inferior results compared to the combined "First + Last" approach, confirming the benefit of multiscale extraction.

5. Significance and Claims

The paper claims that MiVE represents a state-of-the-art approach to reference-guided video editing. Its significance lies in:

Overcoming Modality Gaps: By unifying conditional and visual tokens in a self-attention manifold, the model achieves deeper cross-modal reasoning than previous decoupled or cross-attention-based approaches.
Balancing Fidelity and Semantics: The multiscale extraction strategy successfully resolves the tension between preserving fine-grained spatial details (essential for identity and texture) and understanding global semantic instructions.
Practical Applicability: The method operates without explicit segmentation masks, making it suitable for complex editing scenarios where obtaining accurate masks is difficult or impossible.

The authors position MiVE as a robust solution for applications in film post-production, content creation, and visual effects, while acknowledging the potential for misuse (e.g., deepfakes) and advocating for responsible use and detection mechanisms.