🤖 AI

FaithfulFaces: Pose-Faithful Facial Identity Preservation for Text-to-Video Generation

FaithfulFaces is a novel framework for text-to-video generation that addresses identity distortion in complex dynamic scenes by introducing a pose-shared identity aligner and a curated diverse dataset to maintain robust facial identity consistency across large pose variations and occlusions.

Original authors: Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng, Bing Ma, Kai Yu, Sen Liang, Wenyue Li, Tianxiang Zheng, Qinglin Lu, Zhen Cui

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng, Bing Ma, Kai Yu, Sen Liang, Wenyue Li, Tianxiang Zheng, Qinglin Lu, Zhen Cui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a short movie where a specific person (let's call him "Bob") is boxing. You have a single photo of Bob, and you want the AI to generate a video of him punching, dodging, and moving around.

The problem with current AI tools is that they are like amnesia patients. When Bob turns his head to the left, the AI forgets what Bob looks like from that angle. It might suddenly turn him into a different person, or his face might melt into a weird blob. If he puts his hand in front of his face, the AI panics and distorts his features.

The paper "FaithfulFaces" introduces a new system designed to solve this. Here is how it works, explained simply:

1. The Core Problem: The "Single-View" Trap

Most existing AI models only look at your reference photo and say, "Okay, I see Bob's face straight on." They don't understand that Bob's face is a 3D object that can turn, tilt, and twist. When the video asks for a side profile, the AI tries to guess, and it usually guesses wrong, leading to a distorted face.

2. The Solution: A "Pose-Faithful" Translator

The authors built a new framework called FaithfulFaces. Think of this framework as a super-smart translator that sits between your photo and the video generator.

The Old Way: The translator just says, "Here is Bob's face."
The FaithfulFaces Way: The translator says, "Here is Bob's face, and here is exactly how his head is tilted, turned, and rotated in 3D space."

3. How It Works: The "Pose Dictionary"

The secret sauce of FaithfulFaces is a component called the Pose-Shared Identity Aligner.

Imagine a giant library (a dictionary) filled with cards.

The Old Library: Had cards for "Bob's face" but no cards for "Bob's face turned left" or "Bob's face looking up."
The FaithfulFaces Library: Has a special Pose Dictionary. It learns that "Bob's face" is the same person whether he is looking left, right, up, or down.

When the system sees your photo, it doesn't just copy the pixels. It:

Measures the Pose: It calculates the exact angle of the head (like using a 3D protractor to measure the tilt).
Consults the Dictionary: It looks up how "Bob" should look at that specific angle.
Aligns the Identity: It ensures that even if the angle changes, the "soul" of the face (the identity) stays consistent.

4. The "Training Gym"

To teach this system, the researchers couldn't just use random videos. They needed videos where people were moving their heads wildly.

They built a special dataset pipeline (a factory line) that hunted down thousands of videos.
They filtered out boring videos where people stood still.
They kept only the videos where people were boxing, dancing, or turning their heads significantly.
They fed these "wild movement" videos to the AI so it could learn: "Okay, when the head turns 90 degrees, the nose moves here, but the eyes still look like Bob's."

5. The Result: A Faithful Actor

In their tests, they asked the AI to generate a video of a person boxing (a scenario full of fast head movements and hands blocking the face).

Competitors (like Kling or ConsisID): The boxer's face would morph, look like a different person, or lose its shape when he turned his head.
FaithfulFaces: The boxer kept looking like the same person, with clear features, even when he was punching, dodging, or looking up.

Summary Analogy

If making a video with current AI is like trying to draw a person from a single photo and then guessing what they look like from the side (often resulting in a caricature), FaithfulFaces is like having a 3D sculptor who knows exactly how the person's face is built. No matter how the person moves, the sculptor ensures the clay retains the correct shape and identity.

The paper claims this method is the best at keeping the person's face looking real and consistent, even when they are doing complex, dynamic actions.

Technical Summary: FaithfulFaces

Problem Statement

Identity-preserving text-to-video (IPT2V) generation aims to create diverse videos from a reference face image and text prompts while maintaining consistent human facial identity across frames. Despite recent advancements in large-scale video foundational models (e.g., HunyuanVideo, CogVideoX, Wan), existing methods struggle in complex dynamic scenes. Specifically, when subjects undergo significant facial pose variations (e.g., turning from frontal to profile) or experience facial occlusions, current approaches often produce severe identity distortion, structural collapse, or loss of facial details. The core limitation identified is that existing encoders typically capture only a single-view pose from the input reference image, failing to represent global facial pose information necessary for robust generation under dynamic conditions.

Methodology

The authors propose FaithfulFaces, a pose-faithful facial identity preservation learning framework designed to address these limitations. The methodology consists of three primary components:

1. Pose-Shared Identity Aligner

The core innovation is a specialized encoder, $\phi_{gfp}$ , designed to extract global facial pose representations from a single-view input image.

Architecture: The aligner tokenizes input face images into sequential embeddings. To provide explicit pose cues, it integrates Euler angle embeddings (pitch, yaw, roll) regressed by a pre-trained pose estimator (6DRepNet) directly into the face embeddings.
Pose-Shared Dictionary: The system employs a learnable, pose-shared dictionary matrix $D$ . The combined embeddings are projected into this dictionary space via correlation calculations and max-pooling to generate dictionary weights. These weights are then used to reconstruct refined global facial pose representations ( $S_{p1}, S_{p2}$ ).
Training Objective: The aligner is trained using a Pose Variation–Identity Invariance Constraint. This utilizes a contrastive learning paradigm (InfoNCE loss) where face images of the same identity but different poses are treated as positive pairs, while different identities serve as negative pairs. This forces the model to align diverse poses into a unified identity space while maximizing shared identity information across pose variations.

2. Dataset Construction Pipeline

Recognizing the lack of suitable training data, the authors developed a specialized pipeline to curate a high-quality video dataset featuring substantial facial pose diversity:

Filtering: Videos are filtered to ensure single-subject presence and exclude those without faces.
Pose Selection: A pose estimation step calculates the variation of Euler angles throughout a video. Only videos with a total pose variation ($Var$) exceeding a threshold of 120 (indicating significant transitions like frontal-to-profile) are retained.
Prompt Generation: Text prompts are generated using a Multimodal Large Language Model (MLLM) and manually refined to describe appearance, actions, and backgrounds.
Result: A dataset of 51,624 samples containing video, text prompts, cropped faces, and Euler angles.

3. Integration with Generative Models

The framework operates on a foundational text-to-video model (specifically VACE-14B in their experiments). The refined global facial pose representations are injected into the noisy video generation process alongside the text prompt and the original reference image. The system uses LoRA (Low-Rank Adaptation) to fine-tune the foundational model, allowing it to leverage the new pose-faithful priors without altering the core pre-trained knowledge.

Key Contributions

Pose-Faithful Learning Paradigm: The paper introduces FaithfulFaces, a framework that systematically addresses the limitations of existing IPT2V methods in handling complex facial dynamics by explicitly modeling global facial pose.
Pose-Shared Identity Aligner: A novel module that encodes global facial pose representations from single-view images using a pose-shared dictionary and a pose variation–identity invariance constraint, enhanced by explicit Euler angle embeddings.
Specialized Dataset Pipeline: The development of a rigorous data collection and processing pipeline that constructs a task-oriented video dataset with significant facial pose diversity, ensuring robust model training.

Experimental Results

The authors conducted extensive quantitative and qualitative evaluations against state-of-the-art open-source and commercial methods (including ConsisID, VACE, Kling, Vidu, and HunyuanCustom).

Quantitative Performance: FaithfulFaces achieved state-of-the-art performance across four metrics:
- Identity Preservation: Significantly outperformed baselines in FaceSim-Cur (0.568 vs. 0.484 for the next best) and FaceSim-Arc (0.542 vs. 0.456).
- Visual Quality: Achieved the lowest FID score (164.24), indicating higher fidelity to real face distributions.
- Text Alignment: Attained the highest CLIPScore (33.93).
Qualitative Performance: Visual comparisons demonstrated that FaithfulFaces maintains clear facial structures and consistent identity details even during intense movements (e.g., boxing routines) and occlusions, whereas baseline methods exhibited severe distortion or identity collapse.
Ablation Studies: Experiments confirmed that both the Identity Aligner and the Euler Angle Embedding are critical. Removing the aligner caused identity collapse, while removing Euler embeddings reduced discriminability. The study also showed the method's robustness to non-frontal input views and minor Euler angle perturbations.

Significance and Claims

The paper claims that FaithfulFaces represents a significant step forward in IPT2V by solving the specific challenge of identity distortion under large pose variations. By shifting from single-view identity encoding to a pose-faithful approach that leverages global pose priors, the method enables foundational generative models to produce videos with superior identity consistency and structural clarity. The authors emphasize that their approach provides a theoretical guarantee (via mutual information bounds) that the learned global representation preserves identity information across poses, preventing the "collapse" observed in prior works. The work is presented as a solution to a critical bottleneck in generating realistic, dynamic human videos for applications in film, advertising, and personalized avatars, while acknowledging the ethical implications regarding deepfakes and misinformation.