💻 computer science

Beyond Facial Consistency: Personalized Person Image Generation with Holistic Identity Preservation

This paper introduces a dual-branch framework enhanced by a Dynamic Balancing Scaling (DBS) strategy to resolve the trade-off between facial fidelity and overall appearance consistency in personalized person image generation, alongside the release of the Pexels-100 benchmark for holistic identity evaluation.

Original authors: Yuxuan Xiao, Shanshan Zhang, Jian Yang, Shengcai Liao

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxuan Xiao, Shanshan Zhang, Jian Yang, Shengcai Liao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director in a movie studio, but instead of hiring actors, you have a magical camera that can conjure up any character you can describe. You want a hero who looks exactly like your best friend—same smile, same eyes, same quirky nose—but wearing a space suit on Mars, or perhaps riding a dragon in a medieval forest. This is the dream of "personalized image generation." For a long time, AI artists had to choose between two frustrating options: either make the character look exactly like your friend's face, but forget what they were wearing or how their body looked; or dress them in the perfect outfit and pose, but end up with a stranger's face. It was like trying to bake a cake where you could get the frosting perfect or the sponge perfect, but never both at the same time. This paper dives into that specific corner of artificial intelligence, tackling the tricky problem of keeping a person's "whole self" consistent—their face and their entire outfit and vibe—when the AI is asked to put them in new situations.

The researchers behind this study, led by Yuxuan Xiao and colleagues, decided to stop choosing sides. They started by building a "Naive Dual-Branch" system, which is essentially a two-lane highway for the AI. One lane focuses on the big picture (the clothes, the body shape, the hairstyle), and the other lane zooms in on the tiny details of the face. They found that while this two-lane approach was a good start, it was a bit chaotic. The "face lane" was driving way too fast and taking over the whole car, while the "appearance lane" was getting lost in the rearview mirror. The result? Great faces, but the clothes would sometimes morph into something weird, or the body shape would get squished.

To fix this traffic jam, the team invented a clever traffic control system called Dynamic Balancing Scaling (DBS). Think of it as a smart conductor for an orchestra. The conductor uses two main tricks. First, they use Adaptive Temporal Gating, which is like a dimmer switch that changes the volume of the face and body musicians at different moments during the song. Early in the process, the conductor might let the face play a steady, reliable tune to set the identity, but as the song progresses, they gently turn up the volume on the body and clothes so they don't get drowned out. Second, they introduced Region-Aware Optimization. Imagine the AI is painting a picture, and it usually paints the whole canvas with the same brush pressure. This new trick tells the AI, "Hey, be super careful and precise when painting the face, but don't forget to give the clothes and background just as much attention." This ensures the AI doesn't obsess over the eyes and ignore the shirt.

The team tested their new method on a custom benchmark they created called Pexels-100, which features 100 different full-body images to see how well the AI could keep a person's identity intact across various scenes. The results were promising. Their method managed to strike a much better balance than other open-source tools, keeping the face recognizable while also making sure the clothes and body looked right. In fact, their approach performed so well that it even rivaled some expensive, closed-source commercial systems. However, the authors are careful to note that this isn't a magic wand for every problem yet. If you ask the AI to draw a person doing a complex gymnastic flip or showing intricate hand movements, the image might still get a little wobbly or distorted, because the underlying AI model still struggles with those tricky physical structures. But for most everyday scenarios, this new "conductor" helps the AI finally sing in harmony, keeping the whole person looking like themselves, not just their face.

Technical Summary: Beyond Facial Consistency: Personalized Person Image Generation with Holistic Identity Preservation

Problem Statement
Personalized person image generation aims to synthesize images matching a user's identity based on a reference image and text prompts. Current methods face a fundamental trade-off: they typically emphasize either subject consistency (preserving global appearance, clothing, and body shape) or face consistency (preserving local facial structure and details), but rarely both simultaneously. Existing approaches often suffer from instability when attempting to combine these, leading to scenarios where facial identity is preserved at the expense of overall appearance, or vice versa. This imbalance arises from the heterogeneous representation spaces and injection mechanisms of different identity branches, which cause uncoordinated interactions and persistent over-dominance of the facial branch during the denoising process.

Methodology
The authors propose a unified framework built upon the FLUX.1-dev diffusion transformer backbone, introducing a Naive Dual-Branch (NDB) baseline and refining it with a Dynamic Balancing Scaling (DBS) fine-tuning strategy.

Naive Dual-Branch (NDB) Baseline:
The framework integrates two distinct conditioning pathways:
- Appearance Branch: Based on InstantCharacter [23], this branch captures global identity cues (hairstyle, body proportion, clothing) using fused DINOv2 and SigLIP features projected via a Q-Former and injected via cross-attention.
- Face Branch: Based on InfiniteYou [10], this branch focuses on fine-grained facial geometry and texture using a pretrained face encoder, injected via a ControlNet-style residual pathway.
  While this combination yields promising results, it exhibits instability where the face branch dominates the generation, leading to inconsistent appearance.
Dynamic Balancing Scaling (DBS):
To coordinate the two branches and mitigate the face-branch dominance, DBS introduces two complementary components:
- Adaptive Temporal Gating (ATG): A time-aware mechanism that dynamically modulates the contribution strength of the appearance and face branches throughout the denoising trajectory. The face branch serves as a stable reference anchor, while the appearance branch's contribution is progressively relaxed relative to this anchor. This prevents unstable optimization in early training and ensures balanced guidance across different timesteps.
- Region-Aware Optimization (RAO): A loss function designed to address spatial imbalance. Since facial regions are small and appearance regions are large, standard global Mean Squared Error (MSE) provides uniform supervision that fails to optimize specific regions effectively. RAO decomposes the denoising error into three regional objectives: facial ( $L_f$ ), appearance ( $L_a$ ), and global ( $L_g$ ). It employs a reference-guided Pareto-style aggregation where facial and appearance losses are activated only when they underperform relative to an exponential moving average (EMA) reference, while the global loss remains a persistent background objective. This protects region-specific identity cues without allowing a single branch to dominate the optimization trajectory.

Key Contributions

Naive Dual-Branch (NDB) Method: The proposal of explicitly modeling appearance and local facial features separately within a shared generation framework to address holistic identity consistency.
Dynamic Balancing Scaling (DBS): A fine-tuning strategy comprising Adaptive Temporal Gating and Region-Aware Optimization. This strategy alleviates the insufficient holistic consistency maintenance caused by strong signal dominance in the NDB baseline, achieving a better trade-off between facial fidelity and appearance consistency.
Pexels-100 Benchmark: The construction of the first test benchmark specifically for evaluating holistic identity consistency in personalized person image generation, utilizing full-body images and text prompts to assess performance across facial and appearance dimensions.

Experimental Results
The method was evaluated on the Pexels-100 benchmark using metrics including Face Similarity (ArcFace), Person Re-identification (ReID), DINOv2 score, and CLIP-T.

Quantitative Performance: The proposed method outperforms existing open-source baselines (e.g., InstantCharacter, InfiniteYou, UNO, Kontext) in holistic identity consistency. Specifically, it achieves a Face Score of 0.7875 and a DINOv2 score of 0.3202, surpassing most open-source models and remaining competitive with closed-source commercial systems (e.g., GPT-Image-2).
Qualitative Performance: Visual comparisons demonstrate that while some methods excel in either facial similarity or clothing consistency, the proposed method maintains a more favorable balance, preserving both facial identity and overall appearance coherence across diverse scenarios.
Ablation Studies: Experiments confirm that both ATG and RAO contribute to performance improvements. The full DBS model achieves the best holistic consistency, validating that the components are complementary rather than redundant.

Significance and Claims
The paper claims to provide a controllable basic framework for holistic identity modeling. By addressing the inherent trade-off between facial fidelity and appearance consistency, the proposed method offers a more stable and interpretable approach to personalized person generation compared to existing open-source solutions. The authors emphasize that their approach does not aim to maximize facial similarity in isolation but rather to improve the coherence of the entire identity representation. While the method relies on gating-based modulation without altering the backbone's underlying feature learning (limiting its performance on complex poses or fine-grained structures like hands), it establishes a significant step forward in unifying global and local identity constraints in diffusion-based image generation.

Technical Summary: Beyond Facial Consistency: Personalized Person Image Generation with Holistic Identity Preservation

More like this