🤖 AI

Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model

This paper proposes FPDM, a novel diffusion-based framework for Pose-Guided Person Image Synthesis that utilizes an explicit Image-Pose Fusion module and contrastive learning to align source-pose embeddings with target images, thereby significantly improving texture fidelity and generation consistency across varying poses and appearances.

Original authors: Donghwna Lee, Kirok Kim, Jisu Lee, Kyungha Min, Wooju Kim

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Donghwna Lee, Kirok Kim, Jisu Lee, Kyungha Min, Wooju Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photo of a friend wearing a specific outfit (the Source Image) and a stick-figure drawing of a different pose (the Target Pose). Your goal is to create a new photo of that same friend wearing that exact outfit, but standing in the new pose. This is called Pose-Guided Person Image Synthesis.

For a long time, computers struggled with this. They would either get the pose right but mess up the clothes, or keep the clothes right but lose the person's identity.

Here is how the authors of this paper, FPDM, solved the problem using a new "recipe" for AI image generation.

1. The Problem: The "Confused Chef"

Previous methods were like a chef who gets two separate instructions: "Make a pizza" (the outfit) and "Make it round" (the pose). The chef tries to mix these instructions together while cooking. Because the instructions are mixed up in the pot, the result is often messy. Sometimes the pizza looks like a square, or the toppings get lost.

In technical terms, older AI models tried to blend the "look" of the person and the "shape" of the pose at the very last second of creation. This led to inconsistent results: if you asked for the same pose with a slightly different source photo, the AI would get confused and change the person's identity or the texture of the clothes.

2. The Solution: The "Master Blueprint" (Fusion Embedding)

The authors propose a new method called FPDM. Instead of mixing the instructions while cooking, they create a Master Blueprint before the cooking starts.

Think of it like this:

The Old Way: You hand the chef a photo of a shirt and a sketch of a pose, and say, "Figure it out as you go."
The FPDM Way: You first take the photo of the shirt and the sketch of the pose, and you use a special translator to combine them into a single, perfect Blueprint. This blueprint says exactly: "This is the shirt, and this is how it looks when the person is in this pose."

3. How They Make the Blueprint: The "Matchmaker"

How do they make this perfect blueprint? They use a technique called Contrastive Learning, which acts like a strict Matchmaker.

The AI creates a blueprint by mixing the "Source Photo" and the "Pose Sketch."
The Matchmaker then compares this blueprint to the Real Target Photo (the ground truth).
If the blueprint doesn't match the real photo perfectly, the Matchmaker says, "No, that's wrong!" and pushes them apart.
If they look similar, the Matchmaker pulls them closer.

Crucially, the authors added a special trick: they also show the Matchmaker the Source Photo and say, "This is not the target; don't confuse them." This forces the AI to learn the difference between "what the person looks like" and "how they are posing," ensuring the blueprint captures the relationship between the two, not just a blurry mix of both.

4. The Cooking Process: The "Denoising Network"

Once the AI has this perfect Fusion Blueprint, it uses a powerful engine called a Diffusion Model.

Imagine the Diffusion Model is like a sculptor starting with a block of noisy, static-filled clay. The Blueprint acts as a guide for the sculptor.

The sculptor slowly removes the noise (the "denoising" part).
Because the Blueprint is so precise and pre-aligned, the sculptor knows exactly where every wrinkle in the shirt and every fold in the fabric should go.
The result is a high-quality image where the person's identity is preserved, the clothes look real, and the pose is exactly what was requested.

5. Does It Work? (The Results)

The authors tested this on two main things:

Fashion Photos: They used a massive dataset of clothing photos (DeepFashion). Their method was better at keeping the texture of the clothes (like stripes or patterns) consistent, even when the person turned around or changed poses.
Sign Language: They tested it on videos of people signing (RWTH-PHOENIX dataset). This is tricky because hand details are tiny and complex. Their method produced clearer hands and more accurate movements than previous AI models.

Summary Analogy

Old AI: Like trying to paint a portrait while someone is shouting instructions about the pose and the clothes at the same time. You end up with a blurry mess.
FPDM: Like having a master architect draw a perfect, detailed blueprint that combines the client's style and the building's shape before construction begins. The construction crew (the Diffusion Model) just follows the blueprint, resulting in a perfect building every time.

The paper concludes that by creating this explicit "Fusion Blueprint" first, the AI can generate consistent, high-quality images that respect both the person's identity and the new pose, solving a major headache in computer vision.

Technical Summary: Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model (FPDM)

Problem Statement
Pose-Guided Person Image Synthesis (PGPIS) aims to generate a photo-realistic image of a person in a specified target pose while preserving the identity, clothing, and texture of a source image. While recent diffusion-based approaches have improved image quality, they typically rely on implicit feature aggregation within the denoising process. In these existing methods, source appearance and target pose embeddings are injected directly into the denoising network (e.g., UNet) without explicit pre-alignment. Consequently, the interaction between appearance and pose is learned implicitly, often leading to "entanglement" where the relationship becomes dependent on the specific denoising trajectory. This results in inconsistent generation outcomes for the same identity when source appearance or target pose varies, manifesting as distorted textures, blurry contours, or identity drift.

Methodology
The authors propose FPDM (Fusion Embedding for PGPIS with Diffusion Model), a framework that explicitly aligns fused source–pose embeddings with target image embeddings prior to the diffusion process. The architecture consists of two primary stages:

Image-Pose Fusion (IPF) Module (Pre-Diffusion Stage):
- This module learns a unified representation by fusing the source image embedding and the target pose embedding.
- It employs a Combiner Network (adapted from CLIP4CIR) to integrate features from a CLIP-based Image Encoder and a CLIP-based Pose Encoder.
- Contrastive Learning Strategy: The core innovation is the Source-Enhanced Pose Fusion Approach. Unlike baseline contrastive learning which simply aligns the fused embedding with the target image, this approach explicitly treats the source image embedding as a negative sample during training.
- By constructing a batch-augmented InfoNCE loss where the fused embedding must be close to the target image but explicitly distant from the source image (under the target pose condition), the model enforces a pose-aware separation. This prevents the fused embedding from collapsing into a representation dominated solely by source appearance, ensuring it captures pose-conditioned distinctions.
Conditional Diffusion Model (Generation Stage):
- Built upon the Stable Diffusion (SD) framework, the generative model is guided by three distinct inputs:
  - Source Appearance: Extracted via a DINOv2 encoder and injected into UNet transformer blocks as key/value features.
  - Target Pose: Extracted via a CNN-based encoder (ControlNet-style) and concatenated with the noisy latent.
  - Fusion Embedding ( $e_f$ ): The output of the pre-trained IPF module, injected into ResNet blocks alongside timestep embeddings.
- During inference, the model utilizes an extended Classifier-Free Guidance (CFG) scheme to weight the contributions of the source-pose conditioning and the fusion embedding independently.

Key Contributions

Explicit Alignment via Contrastive Learning: FPDM is the first diffusion-based PGPIS framework to explicitly align fused source–pose embeddings with target image embeddings using contrastive learning, addressing the limitations of implicit conditioning.
Source-Enhanced Pose Fusion: The authors introduce a novel contrastive learning objective that explicitly distinguishes between source and target representations by treating the source embedding as a negative sample. This ensures the fusion embedding is invariant to source appearance variations while remaining sensitive to pose.
Robust Conditioning Signal: The learned fusion embedding serves as a stable conditioning signal throughout the denoising process, enabling consistent identity preservation across varying poses and source images.

Experimental Results
The method was evaluated on two benchmarks: DeepFashion (fashion image synthesis) and RWTH-PHOENIX-Weather 2014T (sign language video generation).

Quantitative Performance: On DeepFashion, FPDM achieved competitive performance against state-of-the-art methods (e.g., MCLD, PCDM, XMDPT). It demonstrated superior LPIPS scores (indicating better perceptual similarity and texture preservation) and strong FIDt (Fréchet Inception Distance on the test set), suggesting better generalization. On the Phoenix dataset, FPDM achieved the highest Hand SSIM (0.682) and competitive Hand Pose Error, critical for sign language fidelity.
Qualitative Performance: Visual comparisons showed FPDM's ability to reconstruct fine-grained details (e.g., specific clothing patterns, logos) and maintain structural consistency even when the source image provided only partial views (e.g., inferring back views from front views).
Ablation Studies:
- The Source-Enhanced Pose Fusion strategy (A3) significantly outperformed baseline fusion (A2) and source-only embeddings (A1) in retrieval accuracy (Recall@1 of 99.7% vs. 5.0% for baseline).
- The full model (B6) consistently outperformed variants without the IPF module or with different fusion strategies in terms of texture fidelity and pose consistency.
- Convergence analysis indicated that while the IPF module adds initial optimization complexity, it leads to superior final performance in perceptual quality.

Significance and Claims
The paper claims that FPDM addresses a fundamental limitation in current diffusion-based PGPIS: the lack of explicit alignment between heterogeneous conditioning signals (appearance and pose). By decoupling the alignment process from the generation process via a pre-trained fusion embedding, the method ensures that identity preservation is robust to variations in pose and source appearance.

The authors position FPDM not as a general-purpose video generation model, but as a task-specific solution for Pose-Guided Person Image Synthesis. They highlight its practical applicability in:

Virtual Try-on and Digital Avatars: Due to high-fidelity texture and identity preservation.
Sign Language Generation: Specifically for generating signer-consistent video frames, where maintaining hand structure and identity across different poses is crucial. The paper notes that while FPDM operates at the image synthesis level, its strong performance on hand-centric metrics makes it a viable core component for sign language video pipelines.

The authors acknowledge limitations, including the difficulty of reproducing fine-grained text or logos and the inherent computational cost of iterative diffusion sampling compared to single-pass GANs. They conclude that future work should focus on extending this explicit alignment strategy to temporally structured video generation tasks.