💻 computer science

Vorch-IR: Long-Form Unified Multimodal Identity Replacement Video Generation

Vorch-IR is a unified multimodal framework built on LTX2 that enables flexible single- and dual-person identity replacement with optional background editing in long-form videos by jointly conditioning on driving videos, reference images, and textual instructions, while overcoming data scarcity through an automatic synthesis pipeline and extending to minute-long generations via a temporal overlapping inference strategy.

Original authors: Yaole Wang, Xiaoyu Chen, Xin Ma, Yang Ding, Gang Yue, Jingjing Chen, Lin Ma, Yaohui Wang

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yaole Wang, Xiaoyu Chen, Xin Ma, Yang Ding, Gang Yue, Jingjing Chen, Lin Ma, Yaohui Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are holding a remote control for reality, but instead of changing the channel, you want to swap the actors in a movie while keeping the script, the camera angles, and the dance moves exactly the same. This is the dream of "video identity replacement," a hot topic in the world of artificial intelligence. For a long time, AI could generate new videos from scratch, but editing an existing one was like trying to change a character's face in a live-action film without messing up the lighting or the movement. Early attempts required the AI to be handed a detailed map of where the person was standing, a skeleton drawing of their pose, or a mask to cover their face. It was like asking a chef to cook a meal only if you gave them a pre-cut list of ingredients and a diagram of the kitchen. These methods were rigid and hard to use. The big question researchers are asking now is: Can we teach an AI to understand a simple sentence like "Swap the dancer on the left with this photo" and just get it, without needing a complex map or a skeleton?

Enter Vorch-IR, a new AI system that acts like a super-smart, magical film editor. Think of it as a director who doesn't need a script supervisor or a stunt coordinator. You give Vorch-IR three things: the original video (the "driving" video), a few photos of the new people you want to insert (the "references"), and a simple text note telling the AI who goes where. The magic is that the photos don't need to match the video's pose or position. If the video shows a person dancing with arms up, and your photo shows them sitting down, Vorch-IR figures out how to make the sitting person dance anyway. It handles single people, two people dancing together, and even swapping the background scenery, all with one single model.

The researchers behind Vorch-IR built this system on top of a powerful video generator called LTX2. They taught it to look at the video, the photos, and the text instructions all at once. Instead of using rigid maps, the AI uses a "vision-language" brain (a type of AI that understands both pictures and words) to figure out the connection. It's like the AI reads your note, looks at the photo, and says, "Ah, you want this person to take the place of the one on the left," and then it uses its internal "self-attention" to blend the new face onto the moving body perfectly.

One of the biggest hurdles in this field is data. To teach an AI to swap faces, you need thousands of examples of "before" and "after" videos. Since these don't exist in the real world, the team built a robot pipeline to make them. They took real videos, used other AI tools to swap the faces in the first frame, and then used a motion-guided robot to make the rest of the video follow the new faces. They then filtered out the bad attempts (like when the AI accidentally gave the dancer three arms) to create a perfect training set. This allowed them to train one model to do everything: swapping one person, swapping two people, and even changing the background, all without needing separate tools for each job.

But what about making a whole movie? Most AI video generators get confused or blurry after a few seconds. Vorch-IR uses a clever trick called "temporal overlapping inference." Imagine trying to paint a long mural. Instead of painting one small square, drying it, and then painting the next (which might make the colors look different), Vorch-IR paints overlapping sections at the same time and blends them together seamlessly. This allows it to generate videos that are a full minute long without the characters' faces drifting or the motion getting weird, all without needing to generate the video one clip at a time.

The team tested Vorch-IR against other top AI models. In head-to-head comparisons, Vorch-IR suggested that it was better at keeping the new person's face looking exactly like the photo (identity preservation) and keeping the background consistent, while still moving smoothly. In human tests, people preferred Vorch-IR's results over other methods, especially when it came to making the new character look real and physically plausible. The paper suggests that while some older models might be slightly better at specific things like perfect pose matching in very specific scenarios, Vorch-IR offers a much more flexible and unified way to edit videos, proving that you don't need complex maps to get great results. It's a step toward a future where editing a video is as easy as typing a sentence.

Technical Summary: Vorch-IR

Problem Statement

Video identity replacement aims to transfer the identities of one or more subjects into a driving video while preserving the original motion, expressions, and temporal structure. Despite recent advances in diffusion-based video generation, existing methods face three primary limitations:

Rigid Structural Controls: Many approaches rely on explicit masks, pose maps, or spatial alignment between reference images and the driving video, complicating integration into general multimodal editing systems.
Single-Identity Focus: Most systems are designed for single-target replacement. Multi-person replacement requires independent identity control and unambiguous association between multiple references and their respective targets, which is rarely addressed.
Data Scarcity: Large-scale paired training data for multi-person and background replacement settings are not readily available, hindering joint learning of these tasks.

Methodology

Vorch-IR is a unified framework built upon the LTX2 architecture that addresses these limitations through a single model capable of handling four settings: single-person replacement, dual-person replacement, and both variants with optional background replacement.

1. Unified Multimodal Conditioning

The model accepts a driving video ( $D$ ), an ordered set of reference images ( $R$ ), and a textual editing instruction ( $u$ ).

Spatial Decoupling: Reference images are treated as independent conditions; they do not need to match the pose, layout, or spatial configuration of the driving video.
Instruction-Grounded Binding: A textual instruction specifies which reference image corresponds to which subject or the background (e.g., "Reference 1 replaces the person on the left"). This replaces the need for geometric correspondence or masks.
Dual-Path Architecture:
- Dense Visual Fusion (Self-Attention): The driving video, all reference images, and the noisy target latent are concatenated into a unified token stream. Self-attention fuses these tokens, preserving fine-grained motion from the driving video and appearance details from the references. Task-ID embeddings distinguish the source video, reference slots, and target.
- Semantic Binding (Cross-Attention): A vision-language model (Gemma-based) jointly encodes the instruction, sampled driving frames, and references. This context is injected via cross-attention to bind each indexed reference to its intended target role defined by the text, enabling semantic routing without explicit localization modules.

2. Automatic Data Construction Pipeline

To overcome the lack of paired training data, the authors developed a pipeline that synthesizes supervision for all four settings:

Editing: The first frame of a source clip is edited to replace identities (and optionally the background) while preserving subject locations and coarse poses.
Propagation: A motion-guided video animation model propagates these edited appearances along the source motion trajectories.
Filtering: A vision-language model filters out failures (e.g., malformed bodies, identity mismatches).
Extraction: The edited first frame is cropped to form identity references, and the background is extracted for background-replacement samples.
This process generates aligned training tuples (driving video, references, instruction, target video) without manual collection.

3. Long-Form Inference Strategy

While trained on short clips (up to 8 seconds), Vorch-IR supports minute-long generation using a Temporal Overlapping Inference strategy adapted from Bidirectional Latent Fusion (BLF):

The full-length target latent is covered with overlapping temporal windows.
At each denoising step, the model predicts denoised windows conditioned on the corresponding source segments and global references.
Predictions are aggregated via normalized temporal weights (linear or cosine cross-fade).
A single global Euler update is applied to the entire latent sequence.
This approach avoids autoregressive continuation, thereby reducing identity drift and error accumulation over long horizons.

Key Contributions

Unified Formulation: A unified framework that formulates single-person, dual-person, and background replacement as a single instruction-guided multimodal video generation problem.
Indexed Reference Interface: A novel interface that spatially decouples references from the driving video, utilizing dense visual self-attention for content transfer and instruction-grounded cross-attention for semantic binding.
Scalable Data Pipeline: An automatic pipeline that synthesizes paired supervision for multi-person and background replacement, eliminating the need for task-specific data collection.
Long-Sequence Extension: An adaptation of overlapping latent fusion that extends short-clip models to minute-long generation without autoregressive feedback loops.

Results

The authors evaluated Vorch-IR on single-person replacement (comparing against Wan-Animate, HunyuanCustom, MoCha, and SCAIL-2) and provided qualitative results for dual-person and background settings.

Automatic Metrics: On the authors' benchmark, Vorch-IR achieved the highest Subject Consistency (0.9096) and Background Consistency (0.9332). It also ranked highly in imaging quality (0.7031) and aesthetic quality (0.6396). On the XDance benchmark, it demonstrated strong generalization, ranking second in DINOv2 and CLIP-I similarity.
Human Evaluation (GSB): In pairwise comparisons, Vorch-IR received a higher "Good" than "Bad" rating across all dimensions (motion, physical plausibility, identity, video quality) against all baselines. Its most significant advantage was in Identity Consistency, where it was preferred by 65.4% of evaluators over HunyuanCustom and 77.0% over Wan-Animate.
Trade-offs: While Vorch-IR excels in identity preservation, some specialized baselines retained slight advantages in specific metrics like temporal flickering or pose matching, indicating a trade-off between strong identity transfer and other specific visual qualities.

Significance

The paper claims that Vorch-IR represents a significant step toward flexible, general-purpose video editing. By removing the dependency on spatially aligned references, masks, or pose inputs, the model simplifies the integration of identity replacement into broader multimodal systems. The ability to handle multiple identities and background changes within a single, instruction-driven model, combined with a scalable data pipeline and long-form inference capability, addresses critical bottlenecks in current video generation research. The work demonstrates that high-fidelity identity transfer can be achieved without sacrificing motion coherence or requiring complex structural controls.