🤖 AI

Controllable Video Object Insertion via Multi-View Priors

This paper proposes a video object insertion framework that leverages multi-view object priors and view-consistent conditioning to overcome limitations in existing methods, such as identity drift and boundary artifacts, thereby achieving superior visual quality, identity consistency, and seamless foreground-background integration.

Original authors: Qi Xia, Peishan Cong, Yichen Yao, Ziyi Wang, Yaoqin Ye, Yuexin Ma

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qi Xia, Peishan Cong, Yichen Yao, Ziyi Wang, Yaoqin Ye, Yuexin Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director on a movie set, but instead of building giant sets, you are working inside a computer screen. You have a video of a busy street, and you want to add a brand-new character, like a giant elephant, walking right through the traffic. This is the world of video object insertion, a branch of computer vision where scientists teach AI to paste new things into existing videos. The tricky part is that videos are alive; the camera moves, the sun shifts, and the objects turn around. If you just paste a flat picture of an elephant into a moving video, it looks like a sticker that doesn't belong. It might look weird when the elephant turns its head, or it might float above the ground instead of walking on it. To fix this, earlier methods tried to use just one photo of the elephant or a simple text description, but that's like trying to guess what the back of a statue looks like just by seeing its front. The result is often a wobbly, glitchy mess that changes its shape or color as the video plays.

Now, meet the new team of researchers from ShanghaiTech University who decided to solve this "sticker problem" with a clever trick. They realized that to make an object look real in a moving video, you need to know what it looks like from every angle, not just one. Their solution is like giving the AI a 360-degree tour of the object before it even tries to put it in the video. Instead of showing the AI a single flat photo, they first turn that photo into a 3D model and then take hundreds of "snapshots" of it from different sides. They call this their Multi-View Priors. Think of it as the difference between handing a painter a single photo of a face versus giving them a live model they can walk around.

But having all these extra photos isn't enough; the AI needs to know which photo to use at the exact right moment. If the elephant in the video turns left, the AI needs to grab the "left-side" photo from its memory bank, not the "front" one. The researchers built a special View-Consistent Conditioning Module that acts like a super-smart librarian. When the video camera moves, this librarian instantly finds the perfect angle of the elephant to match the scene. They also added a "quality check" system. Sometimes, turning a flat photo into a 3D model can go wrong, creating weird, blurry, or missing parts. The AI is taught to ignore these bad snapshots and focus only on the clear ones, ensuring the elephant doesn't suddenly turn into a blob of pixels.

Finally, to make sure the elephant doesn't look like it's floating in mid-air or glowing with a weird halo, they added an Integration-Aware Consistency Module. This part of the system acts like a depth-sensing detective. It checks how deep the elephant is compared to the trees and cars in the background, making sure the elephant gets hidden behind a tree if it walks behind one, and casts the right shadows. They also added a rule to make sure the elephant doesn't flicker or jitter between frames, keeping its movement smooth and natural.

When they tested their new framework, the results were impressive. In their experiments, their method produced videos that looked much more realistic than previous attempts. The inserted objects stayed true to their original look (identity consistency) even when the camera spun around, and they blended into the background without looking like a cut-and-paste job. For example, when they tested on a dataset called DAVIS, their method achieved a score of 23.22 for image clarity (PSNR) and 0.9026 for structural similarity (SSIM), beating other top methods. They even showed that their system could work with just a text description, turning words into a 3D model and then into a video, proving that you don't always need a perfect photo to start. While the system isn't perfect and still relies on the quality of the initial 3D conversion, the researchers suggest that using these multi-view "memory banks" is a major step forward in making video editing feel like magic rather than a glitchy puzzle.

Technical Summary: Controllable Video Object Insertion via Multi-View Priors

Problem Statement
Video object insertion aims to integrate user-specified objects into existing dynamic scenes while maintaining visual coherence. Current methods typically rely on text prompts or a single reference image to define the object's appearance. This approach creates significant limitations:

Identity Drift and Distortion: Single-view observations fail to constrain the object's appearance under viewpoint changes, leading to identity drift, geometric distortion, and artifacts at unseen angles.
Integration Failures: Many existing approaches do not explicitly model foreground-background interactions. Consequently, camera motion and viewpoint shifts often result in incorrect layering (e.g., objects appearing in front of occluding background elements), boundary artifacts, and temporal flickering.
Underconstrained Generation: The lack of multi-view information makes it difficult to preserve the inserted object's identity and ensure plausible occlusion relationships in dynamic environments.

Methodology
The authors propose a video object insertion framework that addresses these limitations by incorporating 3D-derived multi-view priors alongside specialized conditioning and consistency modules. The framework operates on a source video sequence, a foreground reference image, a text prompt, and spatiotemporal control signals (masks or bounding boxes).

The architecture consists of three primary components:

Multi-View Reference Generation:
- A single 2D reference image is "lifted" into a 3D textured mesh using a pre-trained 3D reconstruction tool (Hunyuan3D 2.0).
- The mesh is rendered into a set of multi-view images ( $N$ views) using uniformly distributed camera extrinsics. This provides the generative model with richer appearance and geometric information from various perspectives, rather than relying on a single static view.
View-Consistent Conditioning Module:
- Dual-Path Reference Injection: To effectively utilize multi-view data without compromising stability, the module employs two complementary paths:
  - Identity-Preserving Latent Injection (IPLI): A shared visual encoder processes the source video and the multi-view renderings. The concatenated representations are injected directly into the denoising DiT blocks to anchor the object's appearance across frames.
  - Multi-View Feature Bank (MVFB): The multi-view images are encoded into a searchable knowledge base of key-value pairs. Through cross-attention, the model dynamically retrieves viewpoint-relevant textures and geometric cues from this bank, allowing for adaptive appearance generation as the object rotates or moves.
- Quality-Aware Weighting: Recognizing that 3D lifting can produce imperfect reconstructions (e.g., missing geometry, warped textures), the framework implements a robustness mechanism:
  - Synthesized Artifact Augmentation: Training retains naturally imperfect renderings to expose the model to realistic reconstruction noise.
  - Semantic Consistency Check (SCC): A CLIP-based module computes cosine similarity between the text prompt and each rendered view. A consistency score modulates the reference scale ( $\alpha_{ref}$ ) and scales individual feature tokens. This suppresses the influence of semantically misaligned or noisy views while retaining reliable evidence.
Integration-Aware Consistency Module:
- Geometric Grounding: Two auxiliary heads (Depth Head and Contour Head) operate in the latent space to improve spatial realism.
  - The Depth Head is supervised by pseudo-ground-truth depth maps to enforce correct depth ordering and occlusion relationships.
  - The Contour Head minimizes segmentation mask discrepancies to produce sharper object boundaries and reduce color bleeding.
- Temporal Consistency Optimization: A temporal loss is applied during fine-tuning using Lucas-Kanade optical flow. This warps decoded predictions toward subsequent frames to minimize inter-frame flickering and ensure motion continuity.

Key Contributions

Multi-View Prior Framework: A novel approach that lifts 2D references into multi-view representations to improve appearance consistency and foreground-background integration in dynamic scenes.
View-Consistent Conditioning: A dual-path mechanism combining direct latent injection for identity stability and a feature bank for view-adaptive retrieval, supported by a quality-aware weighting system to handle imperfect 3D reconstructions.
Integration-Aware Consistency: The introduction of latent depth and contour supervision, alongside temporal constraints, to explicitly model occlusion, sharpen boundaries, and reduce flickering.
Comprehensive Evaluation: Validation across multiple benchmarks (DAVIS, VIPSeg, MagicBench) demonstrating improvements in visual quality, controllability, identity preservation, and perceptual integration.

Experimental Results
The framework was evaluated against seven baseline methods (including AnyV2V, UniVideo, VACE, MagicMotion, LoRA-Edit, and PISCO) on DAVIS 2017, VIPSeg, and MagicBench.

Quantitative Performance: The proposed method achieved state-of-the-art results in reconstruction fidelity (PSNR: 23.22, SSIM: 0.9026) and perceptual quality (lowest LPIPS and FVD scores). It also outperformed baselines in spatial controllability (Mask_IoU and Box_IoU).
Reference-Free Metrics: In terms of perceptual video quality, the method achieved the highest Subject Consistency and Temporal Flickering scores, indicating superior identity preservation and smoother transitions.
Ablation Studies: Removing any core component (Multi-view priors, Dual-Path injection, SCC, or Geometric Grounding) resulted in measurable declines in performance, confirming the necessity of each module. For instance, removing the Multi-View Feature Bank led to identity drift during rotation, while removing the Semantic Consistency Check caused the model to follow flawed 3D reconstructions.
Text-Driven Capability: A text-driven variant (Ours t2i), which synthesizes the reference image from text before 3D lifting, remained competitive, demonstrating the pipeline's robustness to domain shifts.

Significance and Claims
The paper claims that the proposed framework effectively addresses the central challenge of preserving an inserted object's identity across different viewpoints while achieving perceptually plausible integration into dynamic scenes. By combining multi-view priors with integration-aware consistency, the method overcomes the limitations of single-view conditioning, specifically reducing identity drift, geometric distortion, and boundary artifacts. The authors position this work as a significant step toward realistic and controllable video editing for applications in film post-production, immersive AR/VR content creation, and autonomous driving simulation, where objects must remain visually coherent with complex scenes over time. The framework supports both coarse bounding-box control and fine-grained mask control without requiring full-scene reconstruction.

More like this