💻 computer science

Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework

The paper proposes Smart-Insertion-V, a closed-loop feedback dual-stream framework that integrates video insertion with image style transfer and employs specialized modules like Dual-World-View RoPE and a Decoupled Guidance Module to achieve photorealistic, harmonious object insertion even under severe stylistic domain gaps.

Original authors: Xiao Cao, Yansong Qu, Xiangzhen, Chang, Wen Xiao, Jiakui Hu, Heyuan Li, Jialun Liu, Zhiyong Huang, Xuelong Li

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiao Cao, Yansong Qu, Xiangzhen, Chang, Wen Xiao, Jiakui Hu, Heyuan Li, Jialun Liu, Zhiyong Huang, Xuelong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a home video of your family having dinner, and you want to magically add a squirrel sitting on the table. The problem is, if you just "paste" a photo of a squirrel into the video, it looks fake. It might be the wrong size, the lighting won't match, and it won't look like it actually belongs there.

This paper introduces a new tool called Smart-Insertion-V that solves this problem. Think of it as a "smart editor" that doesn't just paste the object in; it learns how to make the object look like it was always part of the scene.

Here is how it works, broken down into simple concepts:

1. The Problem with Old Methods

Previous tools tried to do this in two separate steps, like a relay race where the baton is dropped:

Step 1: First, they tried to edit a single photo of the squirrel to match the lighting of the dinner table.
Step 2: Then, they tried to put that edited photo into the video.

The paper argues this is messy. If the first step makes a small mistake (like the squirrel's fur looking a bit too bright), that mistake gets amplified in the second step, making the whole video look weird. It's like trying to build a house by first building a brick, then a wall, then a roof, but never checking if the bricks fit the wall.

2. The Solution: A "Dual-Stream" Team

Instead of a relay race, Smart-Insertion-V uses a two-person team working together in real-time:

The Video Stream: This person focuses on the movie itself, making sure the squirrel moves naturally with the camera and the other people.
The Image Stream: This person focuses on the squirrel photo, instantly changing its style (lighting, color, texture) to match the dinner table.

The Magic Trick: These two streams talk to each other constantly. As the Image Stream figures out how to make the squirrel look "dinner-table-ready," it instantly tells the Video Stream, "Hey, use this version of the squirrel!" This ensures the final result is perfectly harmonized.

3. The "Closed-Loop" Feedback

Imagine you are drawing a picture, and every few seconds, a smart assistant looks at your sketch and says, "Your arm is a bit too long, fix it," and you immediately correct it before finishing the drawing.

Smart-Insertion-V does this. During the generation process, it takes a quick "snapshot" of what it's creating, checks if the squirrel looks right, and uses that check to correct the video while it is still being made. This prevents errors from piling up.

4. The "Dual-World" Map (Dual-RoPE)

When you mix different instructions (like "make the video" and "change the photo style") into one computer brain, they can get confused. It's like trying to listen to two different radio stations at the same time; the music gets garbled.

The authors invented a special "map" called Dual-World-View RoPE. Think of it as giving the computer two different colored notebooks:

Notebook A (Zero Offset): For the actual video frames.
Notebook B (Shifted Offset): For the reference photo and style instructions.

By shifting the "coordinates" of the instructions, the computer knows exactly which note belongs to which song. This prevents the "style" of the squirrel from accidentally leaking into the background of the video (like making the table look like fur).

5. The "Training Gym" (Data Curation)

To teach this AI, you need a massive amount of practice data. But finding videos where an object is perfectly inserted is rare. So, the team built an automated factory:

They took thousands of existing videos.
They used AI to "erase" an object (like a person) to create a clean background.
They took a photo of a similar object, changed its style wildly (to make it look very different from the video), and then taught the AI how to fix that mismatch.

This created a huge library of "before and after" examples, allowing the AI to learn how to fix style mismatches on its own.

Summary

Smart-Insertion-V is like a master chef who doesn't just throw ingredients into a pot. Instead, they have one assistant tasting the sauce (the Image Stream) while another stirs the pot (the Video Stream), constantly adjusting the heat and seasoning until the dish is perfect. The result is a video where the inserted object looks so real and natural that you might forget it wasn't there originally.

Technical Summary: Smart-Insertion-V

Problem Definition

The paper addresses the challenge of mask-free video object insertion, a task requiring the seamless integration of a raw reference object into a source video without user-provided spatial masks. While recent diffusion-based video generation models have advanced significantly, existing methods struggle when the reference object exhibits severe stylistic domain gaps (e.g., lighting, texture, or artistic style) relative to the source scene.

Current approaches fall into three categories, each with distinct limitations:

Explicit Spatial Guidance: Methods relying on user-defined masks or control points introduce annotation burdens and often fail to reason about scene context, leading to implausible object attributes (scale, perspective).
Mask-Free Methods: While eliminating manual annotations, these often generate contextual disharmony because raw references are processed without explicit constraints to adapt their style to the target video.
Cascaded Strategies: Pipelines that first edit the reference image and then insert it into the video suffer from error accumulation. Artifacts or inaccurate style adaptations from the initial editing phase propagate and amplify during video generation, degrading temporal coherence and visual quality.

The core challenge is to achieve spatial plausibility, stylistic harmonization, and temporal consistency simultaneously, particularly when the reference and source domains are visually incompatible.

Methodology: Smart-Insertion-V

The authors propose Smart-Insertion-V, an end-to-end, dual-stream framework designed to concurrently perform video insertion and image style transfer. The architecture integrates three primary components:

1. Dual-Stream Framework with Closed-Loop Feedback

The framework consists of two parallel streams:

Image Stream: Synthesizes a style-harmonized reference image ( $\hat{I}_{tar}$ ) based on the first frame of the source video ( $V^0_{src}$ ) and the raw reference ( $I_{ref}$ ). This stream captures the stylistic transition trend from the raw reference to the target environment.
Video Stream: Performs the core insertion task, conditioned on the source video, the raw reference, and the intermediate features generated by the image stream.

Closed-Loop Feedback: During inference, the framework employs a feedback mechanism. The image stream generates a one-step estimate ( $\hat{x}^{onestep}_0$ ) which is analyzed by a Vision-Language Model (VLM) branch to identify stylistic or spatial imperfections. The VLM produces refinement guidance embeddings that replace initial predictions, allowing both streams to iteratively refine the generation path.

2. Decoupled Guidance Module (DGM)

To address spatial perception and style discrepancies without sacrificing temporal dynamics, the DGM utilizes two distinct branches:

VLM Reasoning Branch: A Vision-Language Model (adapted via an MLP) comprehends the scene's spatial layout, lighting, and style. It infers necessary adjustments for the reference object (scale, orientation, location) based on coarse text instructions.
Native Text Branch: Retains the native T5 text encoder to process general video descriptions, focusing specifically on motion guidance and preserving temporal priors.
This decoupling ensures that spatial-style reasoning does not interfere with the model's ability to maintain consistent motion dynamics.

3. Dual-World-View RoPE (Dual-RoPE)

A critical innovation to prevent feature entanglement and style leakage (where the inserted object's style corrupts unedited regions) is the Dual-World-View Rotary Position Embedding.

Mechanism: Instead of adding task-indicator dimensions or treating conditions as extra frames (which incurs high training overhead), Dual-RoPE applies distinct spatial-temporal offsets to different inputs.
Implementation: Target latents are anchored at zero-offset positions. Strong conditions (e.g., source video, target reference) are offset spatially (along the width axis), while weak conditions (e.g., raw reference) are offset temporally. This isolates feature spaces, allowing the model to process diverse conditioning signals efficiently without heavy computational costs.

Data Curation

Recognizing the scarcity of paired data for raw-to-harmonized object insertion, the authors developed a scalable data curation pipeline:

Source: Utilizes massive open-source text-to-video datasets.
Synthesis: Automatically generates training quadruplets (Source Video, Target Video, Raw Reference, Harmonized Reference) by identifying removable objects, removing them via inpainting, and applying randomized style transfers to the removed objects to create "raw" references with severe domain gaps.
Verification: A rigorous dual-verification mechanism (automated scoring by agents) filters the synthesized data to ensure high quality and strict alignment between the source, target, and reference states.

Experimental Results

The method was evaluated against four baselines (TeleOmni, UniVideo, UniVideo-Q, VACE) and their cascaded variants.

Quantitative Performance: Smart-Insertion-V outperformed all baselines across all metrics, including:
- FVD (Fréchet Video Distance): 1.055 (lowest is best), compared to 1.189 for the next best baseline.
- Video-LPIPS: 0.125 (lowest is best).
- Harmonic Score: 0.842 (highest is best), significantly outperforming the next best score of 0.693.
Qualitative Performance: Visual comparisons demonstrate superior object style harmonization and spatial plausibility. Unlike baselines that often distort surrounding objects or fail to adapt the object's style, Smart-Insertion-V maintains the integrity of unedited regions while seamlessly integrating the object.
Ablation Studies: Removing the dual-stream design, the feedback mechanism, or Dual-RoPE resulted in measurable performance degradation, confirming the necessity of each component.

Significance and Claims

The paper claims that Smart-Insertion-V establishes a new paradigm for mask-free video editing by:

Solving the Domain Gap: Successfully integrating raw, stylistically incompatible references into videos through concurrent style transfer and insertion.
End-to-End Robustness: Eliminating the error accumulation inherent in cascaded pipelines by unifying the tasks in a single framework.
Efficient Feature Disentanglement: Introducing Dual-RoPE to handle complex conditioning signals without the heavy training overhead associated with previous methods.
Data Scalability: Providing a scalable pipeline and a high-quality open-source dataset to bridge the data gap for this specific task.

The authors conclude that their approach achieves the most harmonious insertion results to date, effectively balancing spatial reasoning, stylistic adaptation, and temporal consistency.