💻 computer science

Agent-Driven Multi-View Facial Prior Learning for Identity-Preserving Pose-Guided Generation

The paper proposes ADF-Pose, an agent-driven framework featuring an Identity Prior Extractor and a Multimodal Identity Converter to effectively preserve facial identity and fine-grained details in pose-guided person image generation, particularly under large pose variations where existing methods struggle.

Original authors: Zhaoyang Liu, Luosi Yan, Mubai Li, Xu Zheng, Haotian Yang

Published 2026-08-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhaoyang Liu, Luosi Yan, Mubai Li, Xu Zheng, Haotian Yang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to take a photograph of a friend, but instead of asking them to turn their head, you are asking a computer to imagine what they would look like if they were facing a different direction. The goal is to create a new picture that looks exactly like that person, keeping their unique face, the shape of their jaw, and the texture of their skin, while placing them in a new pose. This is the challenge of pose-guided image generation. For years, computers have become quite good at moving a person's body, shifting their arms and legs to match a new stance. However, when the computer tries to turn the head, the result often feels wrong. The face might lose its specific shape, the eyes might drift apart, or the skin might look like a smooth, plastic mask. The computer struggles because it usually looks at the whole person as one big picture, missing the tiny, crucial details that make a face truly unique, especially when that face is turned away from the camera.

A team of researchers has developed a new approach to solve this problem, creating a system that treats the face with a level of care it has never received before. Instead of relying on a single, blurry guess of what the face might look like from a new angle, their system builds a detailed mental model of the person's face from multiple viewpoints. They call their method ADF-Pose. It works by first isolating the face in the original photo and then using a smart, automated assistant to imagine the missing sides of the face. If the person in the photo is looking slightly to the left, the system doesn't just guess what the right side looks like; it constructs a logical, verified version of that side, checking its work to ensure the nose, mouth, and jawline still belong to the same person. This process creates a set of "facial priors," which are essentially verified blueprints of the person's face from the front, left, and right.

Once these blueprints are ready, the system combines them with a written description of the person's facial features. It then feeds this combined information into a powerful image generator, which is a type of artificial intelligence known for creating realistic pictures from scratch. The generator uses these detailed blueprints and descriptions at every single step of the drawing process, from the initial rough sketch to the final fine-tuning of skin texture. This ensures that the identity of the person is locked in place, preventing the face from drifting or changing shape as the body moves. The researchers tested this method on large collections of fashion and street photography images, comparing their results against the best existing techniques. They found that their system was significantly better at keeping the face looking like the original person, even when the head was turned sharply to the side.

The results showed a clear improvement in how well the system preserved the person's identity. In tests using a dataset of fashion images, the new method achieved a score of 0.312 for facial similarity, a noticeable jump from the previous best score of 0.275. It also produced images with sharper details, reducing the blurriness of facial textures by about 7.4 percent compared to the leading alternative. When looking at the images side by side, the difference is striking. Older methods often produced faces that looked plausible but generic, with softened features or slightly wrong proportions. The new system, however, kept the specific contours of the jaw, the exact spacing of the eyes, and the fine lines around the mouth, even when the person was rotated to a profile view. The system managed to maintain these details without sacrificing the realism of the rest of the body, keeping the clothing and posture looking natural.

The researchers also broke down their system to understand which parts were doing the heavy lifting. They found that the step where the system constructs and verifies the missing views of the face was the most critical. Without this step, the face would lose its unique shape. The second most important part was the way the system combined the visual blueprints with the written descriptions, ensuring the computer understood not just what the face looked like, but how its features related to each other. Finally, the method of feeding this information into the image generator at every stage of the process ensured that the identity details were not lost as the image became clearer. While the system is not perfect and can still struggle if the original face is heavily blocked or too small to see clearly, it represents a significant step forward. It moves beyond simply moving pixels around to actually understanding and preserving the complex, unique geometry of a human face, allowing computers to generate new poses that feel truly like the person they are meant to represent.

Technical Summary: Agent-Driven Multi-View Facial Prior Learning for Identity-Preserving Pose-Guided Generation

Problem Statement

Pose-guided person image generation aims to synthesize a reference person in a target pose while preserving their appearance and identity. While recent diffusion-based methods have improved pose alignment and image realism through structural conditioning and attention mechanisms, they face significant limitations in preserving facial identity under large head pose variations.

Existing approaches typically encode the reference image into global appearance features or rely on implicit correspondence maps. This paradigm proves insufficient for the face region because:

Cross-pose facial identity drift: A single reference view provides limited evidence for facial regions that become visible under novel poses (e.g., a side profile when the reference is frontal). Consequently, models often infer new facial content from general data distributions rather than subject-specific observations, leading to geometry shifts, inconsistent proportions, and identity degradation.
Insufficient facial texture fidelity: Global features and standard cross-attention mechanisms often fail to maintain fine-grained facial textures (eyes, lips, jawline, hair boundaries) throughout the multi-step denoising process, resulting in blurred or over-smoothed details even when global identity similarity is maintained.

Methodology: ADF-Pose Framework

To address these challenges, the authors propose ADF-Pose, a diffusion framework that employs an agent-driven strategy to construct and verify multi-view facial priors. The framework explicitly models facial identity through complementary visual and semantic priors, consisting of two key modules: the Identity Prior Extractor (IDPE) and the Multimodal Identity Converter (MIDC).

1. Identity Prior Extractor (IDPE)

IDPE transforms a single visible face from the reference image into a verified multi-view identity prior representation. It operates through three stages:

Explicit Facial Prior Anchoring: The module localizes the face region using a face segmenter and estimates the visible facial orientation (front, left, or right) using facial landmarks (nose, eyes) to compute a continuous yaw angle.
Multi-View Prior Completion and Verification: For missing views (e.g., generating a side view from a frontal reference), a frozen multimodal agent constructs structured topological directives. These directives guide a generative model to synthesize complementary facial views based on the specific subject's topology and orientation. Crucially, IDPE employs a parallel dual-metric verification process. Generated candidates are accepted only if they satisfy both identity consistency (similarity to the reference) and geometric rationality (correctness relative to the target view). If a candidate fails, the system adjusts generation seeds/weights for up to three trials.
Multiview Semantic Decoupling Encoding: The verified visual priors are paired with view-specific textual identity descriptions (e.g., specific facial geometry, feature relations) generated by the agent. This creates a standardized identity prior set containing both visual and semantic data.

2. Multimodal Identity Converter (MIDC)

MIDC aligns the heterogeneous visual and semantic priors and integrates them into the diffusion process:

Dual Cross-Attention Fusion (DCAF): This module performs two ordered cross-attention stages:
1. Facial-Text Fusion: Aligns the visual identity embedding (from multi-view priors via a frozen CLIP encoder) with the semantic identity embedding (from textual descriptions via a frozen Sentence Transformer). This allows the model to associate local textures with semantic relations.
2. Local-Global Fusion: Integrates the fused identity representation with the current noisy visual latent ( $z_t$ ) of the denoising process.
Stage-wise Identity Injection (SII): Unlike methods that form a single aggregate condition, ADF-Pose injects the identity-conditioned features into multiple stages of the denoising U-Net. This ensures that identity constraints are maintained from coarse structural generation to fine-grained facial texture refinement.

3. Training and Inference

The framework is built upon Stable Diffusion 1.5. During training, the VAE, CLIP, Sentence Transformer, and IDPE components (segmenter, detector, generator, agent) remain frozen. Only the Pose Encoder, the denoising U-Net, and the MIDC are optimized. The training objective is the standard noise prediction loss. During inference, IDPE runs once per reference image to generate priors, which are then reused throughout the denoising trajectory.

Key Contributions

The paper highlights four primary contributions:

ADF-Pose Framework: A diffusion framework utilizing an agent-driven strategy to address cross-pose facial identity drift and texture fidelity issues.
Identity Prior Extractor (IDPE): The first agent-driven module in this domain to construct and verify complementary multi-view facial priors from a single reference image, providing explicit subject-specific evidence for unseen views.
Multimodal Identity Converter (MIDC): A mechanism that aligns visual priors with identity semantics and adapts them to the denoising latent, preserving geometry and texture throughout the generation process.
Empirical Validation: Extensive experiments demonstrating superior performance in identity consistency and texture fidelity compared to state-of-the-art methods.

Experimental Results

The authors evaluated ADF-Pose on the DeepFashion and Market-1501 datasets, comparing it against twelve state-of-the-art methods (including PIDM, PCDMs, and CFLD).

Quantitative Performance:
- On DeepFashion, ADF-Pose achieved the highest IDSim (0.312, a 13.5% improvement over the best baseline) and the lowest F-LPIPS (0.151), indicating superior facial identity preservation and texture fidelity. It also achieved the highest SSIM and lowest LPIPS for the full image.
- On Market-1501, the method achieved the best SSIM (0.3206), LPIPS (0.2219), and FID (13.094).
Qualitative Performance: Visual comparisons show that ADF-Pose better preserves facial contours, feature layouts, and local details (e.g., hair boundaries, jawlines) under large pose changes compared to baselines, which often suffer from identity drift or blurring.
Ablation Studies: Removing IDPE, DCAF, or SII consistently degraded performance, confirming that each component contributes complementarily to reducing identity drift and improving texture fidelity.

Significance and Claims

The paper claims that ADF-Pose represents a significant step forward by shifting from implicit, global appearance representations to explicit, multi-view facial priors. By leveraging an agent to reason about missing views and verify their geometric and identity consistency, the framework provides the diffusion model with robust, subject-specific evidence that is otherwise missing in single-reference settings.

The authors emphasize that their approach effectively bridges the gap between global pose alignment and local identity preservation. However, they modestly acknowledge limitations: the system's performance depends on the quality of the visible face and the accuracy of the generated complementary views. Heavy occlusion, low resolution, or landmark detection errors can degrade the priors. Additionally, the agent-driven preprocessing adds complexity to the pipeline. Future work is proposed to improve robustness under these adverse conditions and simplify the construction process.