💻 computer science

AnyID: Ultra-Fidelity Universal Identity-Preserving Video Generation from Any Visual References

AnyID is a novel framework for ultra-fidelity identity-preserving video generation that overcomes the limitations of single-reference methods by unifying heterogeneous visual inputs through a scalable omni-referenced architecture and employing a primary-referenced generation paradigm with differential prompting to achieve precise attribute-level controllability.

Original authors: Jiahao Wang, Hualian Sheng, Sijia Cai, Yuxiao Yang, Weizhan Zhang, Caixia Yan, Bing Deng, Jieping Ye

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiahao Wang, Hualian Sheng, Sijia Cai, Yuxiao Yang, Weizhan Zhang, Caixia Yan, Bing Deng, Jieping Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a short movie starring your favorite actor, but you only have a few scattered photos and a single old home video of them. You want them to look exactly like themselves, but you want them to be doing new things, wearing different clothes, and standing in different places.

The Problem with Old Methods:
Previous AI tools were like a strict, one-track director. They would say, "Okay, I only have this one photo of your actor. I'll guess what they look like from the side, what their voice sounds like, and how they move." Because they only had one clue, they often got it wrong. The actor might end up looking like a stranger, or their face might warp when they turned their head. It was a "best guess" that often failed.

The Solution: AnyID
The paper introduces AnyID, a new AI system that acts like a super-observant director who gathers all your clues before filming. Here is how it works, broken down into simple concepts:

1. The "Photo Album" Approach (Omni-Referenced)

Instead of forcing the AI to guess based on just one photo, AnyID lets you throw in anything: a selfie, a portrait, a video clip, or even a mix of all three.

The Analogy: Imagine trying to describe a friend to a painter. If you only show the painter one photo of your friend's face, the painter might get the nose right but mess up the ears. But if you show them a photo album with your friend smiling, frowning, looking left, and looking right, the painter can build a perfect 3D mental model of your friend. AnyID does this digitally, combining all your references to create a "perfect 3D memory" of the person's identity.

2. The "Anchor and Edit" Strategy (Primary Reference & Differential Prompts)

This is the smartest part of the system. When you want to change something (like the background or the clothes), old AIs often accidentally change the person's face or hair too.

The Analogy: Think of the Primary Reference as the "Anchor." It's the one photo you say, "This is exactly how their hair and face should look."
The Differential Prompt: Instead of writing a whole new description of the scene (which confuses the AI), you just tell the AI what to change.
- Old way: "A man with short black hair wearing a red jacket standing on a beach." (The AI might accidentally make his hair red or change his face).
- AnyID way: "Change the jacket to red and move the background to a beach." (The AI knows to keep the hair and face exactly as the "Anchor" photo shows, only changing the specific things you asked for).

3. The "Human Critic" (Reinforcement Learning)

Training an AI to make videos is like teaching a dog. You can show it thousands of examples, but sometimes it still gets the details wrong (like making the hair look plastic or the movement jerky).

The Analogy: After the AI learns the basics, the researchers let a group of humans play "Judge." They show the AI two videos: one that looks good and one that looks slightly off. The humans pick the winner. The AI then learns, "Oh, humans prefer the one with the smoother hair shine." It repeats this thousands of times, essentially "studying for the test" until it perfectly matches human taste.

Why This Matters

For Creators: You can finally make consistent characters for stories, games, or social media without needing a Hollywood budget or a team of animators.
For Everyone: It solves the "uncanny valley" problem. The characters don't look like weird, glitchy robots; they look like real people moving naturally.

In a Nutshell:
AnyID is like a magic movie studio that takes your messy collection of photos and videos, builds a perfect 3D model of the person, and then lets you direct them to do anything you want, while guaranteeing they still look exactly like themselves. It's the difference between a blurry guess and a high-definition reality.

1. Problem Statement

Current identity-preserving video generation methods suffer from two fundamental limitations:

Single-Reference Ill-Posedness: Most existing approaches rely on a single static image (usually a face) as the identity reference. This creates an ill-posed problem because a 2D snapshot cannot capture a person's full 3D structure, dynamic expression patterns, or diverse viewpoints. Consequently, models struggle to maintain identity fidelity when the generated video involves significant changes in head angle, expression, or lighting, often leading to "identity drift" or artifacts.
Rigid Control: Traditional methods lack fine-grained controllability. They often struggle to modify specific attributes (e.g., changing clothing while keeping the hairstyle and background constant) without inadvertently altering the identity or causing visual distortions.
Input Inflexibility: Real-world users possess diverse collections of media (faces, portraits, full videos), but current models are not designed to ingest and unify these heterogeneous formats effectively.

2. Methodology

AnyID proposes a comprehensive framework to address these issues through four core technical components:

A. Scalable Omni-Referenced Architecture

Instead of relying on specialized external encoders (like separate face or CLIP encoders) which create bottlenecks, AnyID utilizes a unified VAE encoder to process all visual inputs.

Input Unification: It accepts a set of $N$ free-form references (static images or video clips) of varying modalities.
Standardization: All inputs are resized/padded to a uniform resolution and encoded into latent representations.
Temporal Concatenation: The encoded reference latents are concatenated along the temporal dimension with the noisy target latent.
Timestep Manipulation: To distinguish between references and the target, references are assigned a static timestep of $t=0$ (noise-free), while the target is assigned $t>0$ . The model learns to process these differently, discarding the reference outputs and only using the target output for loss calculation.

B. Primary-Referenced Generation & Differential Prompts

To solve the conflict of inconsistent attributes across multiple references (e.g., different hairstyles in different photos), AnyID introduces a Primary-Referenced Generation Paradigm:

Canonical Anchor: One reference is designated as the "primary" anchor, defining the static attributes (hairstyle, clothing, accessories) that must be preserved.
Differential Prompting: Instead of describing the entire target scene, the system uses a differential prompt that describes only the changes relative to the primary reference.
Automated Construction: A two-stage pipeline using a Vision-Language Model (VLM) and a Large Language Model (LLM) generates these prompts. The VLM describes the primary reference and target; the LLM compares them, prunes unchanged elements (based on a similarity threshold), and outputs the final differential prompt.

C. Human-Centric Reinforcement Learning (RL)

Standard supervised training (MSE loss) often results in overly smooth outputs that degrade fine-grained details. AnyID employs Direct Preference Optimization (DPO) to align the model with human perception:

Dual-Track Evaluation: A preference dataset is constructed where annotators compare video pairs based on two distinct criteria:
1. Identity Fidelity: Does the video preserve the person's dynamic identity across angles and expressions?
2. Prompt Controllability: Does the video accurately reflect the requested changes while keeping unmentioned elements consistent?
Pareto Optimization: Only pairs where one video is superior in both dimensions are used for training, ensuring the model learns to balance fidelity and control.

D. Data Curation Pipeline

The authors constructed a large-scale dataset based on PortraitGala, filtering for high-quality, single-subject clips. They synthesized diverse training instances by randomly sampling frames to create "face," "portrait," and "video" references from the same ID-group, ensuring the model learns robust identity invariance.

3. Key Contributions

Universal Identity Formulation: A novel task formulation that accepts heterogeneous, multi-modal references (faces, portraits, videos) to resolve the ill-posed nature of single-image identity modeling.
Omni-Referenced Architecture: A scalable design that unifies diverse inputs using a single VAE encoder, avoiding architectural bottlenecks and enabling high-fidelity generation.
Differential Prompting Strategy: A mechanism to decouple static identity attributes from dynamic modifications, allowing precise, attribute-level control without identity drift.
RL-Driven Optimization: The application of human-centric reinforcement learning (DPO) specifically tuned for identity fidelity and prompt controllability, significantly outperforming pixel-level loss baselines.

4. Experimental Results

The authors evaluated AnyID against state-of-the-art baselines (ConsisID, FantasyID, Phantom, SkyReels-A2) on a new benchmark covering Identity-Preserving (IPT2V) and Identity-Element-Preserving (IEPT2V) tasks.

Quantitative Performance:
- Identity Fidelity: Achieved 73.22% on Holi-Arc and 67.52% on Holi-Cur, significantly outperforming all baselines.
- Prompt Controllability: Scored 86.56% (Appearance), 60.31% (Motion), and 84.79% (Background), demonstrating superior ability to follow instructions.
- Visual Quality: Led in both Static (82.03%) and Dynamic (91.12%) scores.
Qualitative Analysis:
- AnyID successfully preserved identity across extreme angle changes and expression shifts where baselines failed (identity drift).
- It effectively handled "copy-paste" errors common in other models, correctly preserving unmentioned elements (like clothing) while changing others.
User Study: In a study with 20 participants, AnyID won >70% of matchups in identity fidelity and prompt controllability, and >60% in element consistency and visual quality.

5. Significance

AnyID represents a paradigm shift in human-centric video generation. By moving away from the restrictive single-reference assumption, it unlocks creative flexibility, allowing users to leverage their existing, diverse media collections to generate high-quality content. The introduction of differential prompting solves the critical challenge of attribute-level control, making the technology viable for professional applications like virtual avatars, personalized storytelling, and film production. Furthermore, the integration of human feedback via RL ensures the generated content aligns with human perceptual standards rather than just mathematical metrics, setting a new benchmark for ultra-fidelity generative AI.