💻 computer science

Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation

The paper introduces Stand-In, a lightweight and plug-and-play framework that achieves high-fidelity identity preservation in video generation by integrating a conditional image branch with restricted self-attention, enabling superior performance with minimal additional parameters and seamless compatibility with various AIGC tasks.

Original authors: Bowen Xue, Zheng-Peng Duan, Qixin Yan, Wenjing Wang, Hao Liu, Chun-Le Guo, Chongyi Li, Chen Li, Jing Lyu

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bowen Xue, Zheng-Peng Duan, Qixin Yan, Wenjing Wang, Hao Liu, Chun-Le Guo, Chongyi Li, Chen Li, Jing Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a movie where your favorite celebrity (or your friend, or even a cartoon character) stars in a scene you describe, like "dancing in the rain" or "drinking coffee in Paris."

In the world of AI video generation, this is incredibly hard. Usually, if you ask an AI to make a video of a specific person, the result looks like a stranger who vaguely resembles them, or the person's face melts into a blob as they move.

Enter Stand-In, a new tool from researchers at Tencent and Nankai University. Think of Stand-In not as a heavy, expensive construction crew, but as a lightweight, plug-and-play "ghost" actor that can slip into any movie set and play the lead role perfectly.

Here is how it works, broken down with simple analogies:

1. The Problem: The "Heavy Suit" vs. The "Magic Costume"

Most previous methods to keep a person's face consistent in a video were like trying to fit a giant, heavy suit of armor onto a dancer.

The Old Way: To get the face right, you had to retrain the entire AI model (the "dancer") from scratch. This took massive computing power, huge amounts of data, and often broke the model's ability to follow your instructions (like "make it rain").
The Stand-In Way: Instead of rebuilding the dancer, Stand-In adds a tiny, magical "costume" (about 1% of the model's size) that tells the AI exactly who to look like. It's so light that the AI doesn't even notice the extra weight.

2. The Secret Sauce: The "Restricted Self-Attention"

How does the AI know which pixels belong to the "face" and which belong to the "background" without getting confused?

Imagine you are in a crowded room (the video) and you are trying to listen to a specific friend (the reference photo) while ignoring everyone else.

The Mistake (Vanilla Attention): In older AI models, the "friend" in the photo would start listening to the crowd. The AI would get confused, thinking the background trees or other people were part of the face, causing the identity to warp.
The Stand-In Fix (Restricted Attention): Stand-In puts a soundproof glass wall between the "Friend" and the "Crowd."
- The Friend (Reference Image) can only look at themselves. They stay perfectly still and unchanged.
- The Crowd (The Video) can look at the Friend to copy their face, but the Friend cannot look at the Crowd to get distracted.
- This ensures the face stays exactly the same, while the body and background move naturally.

3. The "Address System": Conditional Position Mapping

To make sure the AI doesn't mix up the "Friend" with the "Crowd," Stand-In gives them different addresses.

Imagine the video frames are a city grid. The video characters live in "Downtown."
Stand-In tells the AI: "The Reference Photo lives in a special, invisible neighborhood called 'The Cloud'."
By giving the photo a completely separate address, the AI knows: "Oh, that pixel is the identity guide, not part of the moving scenery." This prevents the AI from accidentally stretching the face or making the background look weird.

4. Why It's a Game Changer

Lightweight: It only needs 2,000 training pairs (a tiny dataset compared to the millions others use) and adds almost no extra computing cost.
Plug-and-Play: You can take this "magic costume" and put it on any video generator. It works like a universal USB drive.
Versatile: It doesn't just work for humans. Because it learns the concept of an identity rather than just memorizing faces, you can use it to make a teddy bear dance, a cartoon character act out a scene, or even swap faces in existing videos seamlessly.

The Bottom Line

Before Stand-In, making a video with a specific person's face was like trying to paint a portrait while juggling chainsaws—hard, dangerous, and often messy.

Stand-In is like handing the artist a perfect, pre-drawn stencil. They can still paint the background, the lighting, and the action however they want, but the face is guaranteed to be exactly who you asked for, every single time. It's fast, it's cheap, and it just works.

1. Problem Statement

Generating high-fidelity videos that strictly adhere to a user-specified identity (e.g., a specific person's face) is a critical challenge in generative AI. Existing approaches suffer from significant limitations:

Face-Encoder Methods: Early approaches rely on explicit face encoders to extract features. These often lack flexibility and fail to capture the fine-grained facial details necessary for high-quality video reconstruction.
Full-Parameter Training: Recent methods (e.g., HunyuanCustom, Phantom) fine-tune the entire diffusion transformer. While effective, these require massive computational resources, huge datasets, and lack compatibility with other AIGC tools or pre-trained models.
The Gap: There is a need for a lightweight, data-efficient, and plug-and-play solution that preserves identity without destroying the pre-trained prior of the base video generation model.

2. Methodology: Stand-In Framework

The authors propose Stand-In, a framework that injects identity control into a pre-trained video generation model (specifically Wan2.1-14B) with minimal architectural changes.

A. Core Architecture

Conditional Image Branch: Instead of using an external face encoder, Stand-In utilizes the pre-trained VAE of the video generation model itself. The reference image is encoded directly into the same latent space as the video.
Token Concatenation: Image tokens and video tokens are concatenated along the sequence dimension and processed jointly through the Diffusion Transformer (DiT) blocks.
Static Conditioning: The reference image is treated as a static condition. Its denoising timestep is fixed at $s_{ref} = 0$ , ensuring it remains invariant while the video undergoes denoising.

B. Key Technical Innovations

To merge identity information without disrupting the model's generative capabilities, two specific mechanisms are introduced:

Restricted Self-Attention (RSA):
- Problem: Standard self-attention allows image queries to attend to video keys (and vice versa), causing the static identity to be "polluted" by dynamic video content or ignored entirely.
- Solution: The attention mechanism is modified so that image queries cannot attend to video keys.
  - Image tokens ( $Q_I, K_I, V_I$ ) and Video tokens ( $Q_V, K_V, V_V$ ) are computed independently.
  - For video queries ( $Q_V$ ), the Key and Value matrices are concatenated with the image counterparts ( $[K_V, K_I]$ and $[V_V, V_I]$ ).
  - This ensures video frames can "look at" the identity reference, but the reference image remains a static, unchanging guide.
- Optimization: Low-Rank Adaptation (LoRA) is applied only to the Query/Key/Value projections of the image tokens, keeping the total trainable parameters extremely low.
- Efficiency: Since the image Key/Value matrices are constant ( $s_{ref}=0$ ), they can be cached (KV Caching) during inference, minimizing computational overhead.
Conditional Position Mapping (CPM):
- Problem: Simply concatenating tokens in a shared coordinate system confuses the model, breaking the pre-trained positional priors and leading to unstable spatial layouts.
- Solution: A specialized 3D Rotary Positional Embedding (RoPE) strategy is used to create disjoint coordinate spaces:
  - Temporal: Image tokens are assigned a fixed temporal index of -1, while video tokens occupy non-negative indices. This establishes the image as a temporally invariant condition.
  - Spatial: Image tokens are mapped to a dedicated spatial subspace $[H_V, H_V+H_I) \times [W_V, W_V+W_I)$ , separate from the video's spatial grid.
- Benefit: This geometric separation prevents false spatial correlations and preserves the backbone model's pre-trained positional understanding.

C. Data Strategy

Dataset: A custom dataset of 2,000 high-resolution human video sequences was constructed.
Preprocessing: Videos are resampled to 25 FPS, cropped to 832×480, and faces are extracted, resized to 512×512, and background-parsed (white background) to prevent leakage.
Training: The model is trained for only 3,000 steps with a batch size of 48.

3. Key Contributions

Lightweight & Efficient: The method adds only ~1% additional trainable parameters (153M for a 14B model) and requires only 2,000 training pairs, yet outperforms full-parameter training methods.
Plug-and-Play Design: The framework does not alter the main DiT architecture, allowing seamless integration into various tasks (e.g., pose-guided generation, stylization, face swapping) without retraining.
Novel Attention Mechanism: The introduction of Restricted Self-Attention and Conditional Position Mapping solves the conflict between maintaining a static identity and generating dynamic video content.
Generalization: Despite being trained on human data, the model generalizes zero-shot to non-human subjects (cartoons, objects) and diverse ethnicities/ages.

4. Experimental Results

The authors evaluated Stand-In against state-of-the-art (SOTA) closed-source (Kling, Hailuo) and open-source (ConsistID, Phantom, VACE) methods.

Quantitative Metrics (OpenS2V Benchmark):
- Face Similarity: Stand-In achieved 0.724, significantly outperforming the next best (VACE-14B at 0.647).
- Naturalness: Scored 3.922 (out of 5), surpassing all competitors, indicating high visual realism without AI artifacts.
- Prompt Following: Achieved 20.594, ranking first among open-source methods and second overall, proving identity control does not degrade text adherence.
User Study: In a blind test with 20 participants, Stand-In received the highest scores for both Face Similarity (4.10) and Video Quality (4.08).
Ablation Studies:
- Replacing RSA with Vanilla Self-Attention dropped Face Similarity from 0.724 to 0.422.
- Replacing CPM with Shared Position Mapping dropped Face Similarity from 0.724 to 0.536 and caused spatial instability.

5. Significance and Applications

Stand-In represents a paradigm shift in identity-preserving video generation by proving that high-fidelity control does not require massive fine-tuning. Its significance lies in:

Democratization: Making high-quality identity video generation accessible with minimal compute resources.
Versatility: The framework is successfully applied to:
- Subject-Driven Generation: Creating videos of non-human subjects (e.g., teddy bears, anime characters).
- Pose-Guided Generation: Enhancing existing pose-control frameworks (like VACE) with better identity retention.
- Video Stylization: Applying artistic styles (e.g., Ghibli, Redline) while keeping the subject's identity intact.
- Face Swapping: Achieving high-quality, temporally consistent face swapping via zero-shot inpainting.

In conclusion, Stand-In offers a robust, efficient, and highly compatible solution for identity control in video generation, setting a new standard for performance-to-parameter efficiency in the field.