💻 computer science

DiffSwap++: 3D Latent-Controlled Diffusion for Identity-Preserving Face Swapping

DiffSwap++ is a novel 3D latent-controlled diffusion framework that enhances identity preservation and geometric consistency in face swapping by leveraging 3D facial structure to disentangle identity from pose and expression, outperforming existing methods on multiple benchmarks.

Original authors: Weston Bondurant, Arkaprava Sinha, Hieu Le, Srijan Das, Stephanie Schuckers

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weston Bondurant, Arkaprava Sinha, Hieu Le, Srijan Das, Stephanie Schuckers

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the realm of digital imaging, the ability to swap one person's face onto another's body has long been a fascination, blending art with complex mathematics. For years, the tools used to create these images relied on a type of artificial intelligence known as a generative adversarial network, or GAN. These systems work like two competing artists: one tries to create a fake image, while the other tries to spot the forgery. Over time, this competition forces the creator to become incredibly skilled. However, this process is notoriously unstable, often resulting in images that look slightly off, with blurry features or strange distortions that the human eye can easily spot. More recently, a different approach called diffusion has emerged. Instead of a competitive game, diffusion models work like a sculptor slowly refining a block of stone, gradually removing noise to reveal a clear picture. While these newer models produce sharper images, they still struggle with a specific problem: keeping the person's true identity intact when they are placed into a new pose or expression. Without a deep understanding of the three-dimensional structure of a face, the computer often confuses the person's unique features with the angle of their head or the shape of their smile, leading to results that look realistic but belong to the wrong person.

A team of researchers at the University of North Carolina at Charlotte has developed a new method called DiffSwap++ to solve this specific puzzle. Their work focuses on teaching the computer to understand the invisible three-dimensional skeleton of a face, even though the final image it creates is flat and two-dimensional. The researchers realized that to swap a face perfectly, the system needs to know not just what the face looks like from the front, but how the features are arranged in space. To achieve this, they trained their model using a special kind of digital map that captures the depth and volume of a face. During the training phase, the computer learns to project a flat photo into this three-dimensional space, analyzing the underlying structure before converting it back into a standard image. This process allows the model to separate the person's identity—their unique bone structure and skin texture—from the pose and expression, which are merely the temporary positions of that structure.

The innovation lies in how this three-dimensional knowledge is used. The researchers do not require the computer to build a full 3D model every time it swaps a face; instead, they use the 3D information only while the system is learning. Once trained, the model can perform the swap using only standard 2D images, making the process fast and practical. The system takes a source image of a person and a target image of a different person in a different pose. It then uses the lessons learned from the three-dimensional training to reconstruct the target's face, replacing the features with those of the source person while strictly maintaining the target's head angle and facial expression. This ensures that the final image looks like the source person is naturally present in the scene, rather than just having a sticker pasted onto a different face.

To test if their method actually worked, the researchers ran extensive experiments using thousands of images from public datasets containing high-quality portraits of real people. They compared their new system against several of the best existing methods, including both older GAN-based tools and newer diffusion models. The results showed a clear advantage for DiffSwap++. When the researchers measured how well the swapped faces retained the identity of the original person, their method consistently scored higher than any previous approach. In tests designed to see if the swapped faces could fool a biometric recognition system, DiffSwap++ was significantly more successful at preserving the source identity than its competitors. At the same time, it did not sacrifice the natural look of the target's pose or expression. While other methods often produced faces that looked slightly distorted or failed to transfer the identity completely, DiffSwap++ produced images that were both highly realistic and faithful to the original person.

The team also conducted a study with human volunteers to see how convincing the images looked to the naked eye. Participants were shown sets of swapped faces and asked to rate them on how well the identity, expression, and pose were preserved. The results confirmed what the computer measurements suggested: people found the images created by DiffSwap++ to be the most realistic and convincing. The faces looked natural, with no strange artifacts or unnatural blending that often give away digital manipulation. In contrast, other methods sometimes left behind tell-tale signs of forgery, such as hollow eyes or irregular textures around the mouth. By integrating three-dimensional structural awareness into the learning process, the researchers demonstrated that it is possible to achieve a level of detail and accuracy that was previously out of reach, creating face swaps that are not only visually stunning but also scientifically robust in their preservation of identity.

Technical Summary: DiffSwap++: 3D Latent-Controlled Diffusion for Identity-Preserving Face Swapping

1. Problem Statement

Face swapping involves transferring the identity of a source face onto a target face while preserving the target's pose, expression, and background. While Generative Adversarial Networks (GANs) have been widely used, they suffer from training instability and often produce results with noticeable artifacts and poor identity consistency. Recent diffusion-based approaches have improved fidelity but still struggle with fine-grained artifacts and weak identity preservation. A primary limitation identified in existing diffusion methods is the lack of explicit modeling of the underlying 3D facial structure. Without 3D awareness, models find it difficult to effectively disentangle identity from pose and expression, leading to swaps that either fail to transfer the source identity or distort the target's attributes. Furthermore, existing 3D-aware diffusion methods (e.g., DiffSwap) often limit 3D features to inference-time landmark generation rather than using them to guide the training process directly.

2. Methodology: DiffSwap++

The authors propose DiffSwap++, a novel pipeline that leverages 3D-aware latent representations to guide the diffusion generation process. The method operates entirely within the semantic latent space of a pre-trained Latent Diffusion Model (LDM) using Denoising Diffusion Implicit Models (DDIMs) for efficiency.

Core Components

3D-Aware Training, 2D Inference: A key innovation is that DiffSwap++ incorporates 3D facial information during training but operates without 3D features at inference time. This ensures the model learns robust geometric disentanglement without requiring 3D reconstruction capabilities during deployment.
3D Latent Extraction: The system utilizes a pre-trained 3D face reconstruction model (based on EG3D) to project 2D face images into 3D latent embeddings. These embeddings are hierarchically organized into three levels of granularity:
- Coarse-grained (Layers 1-4): Controls low-frequency structural geometry (head shape, pose).
- Middle-grained (Layers 5-9): Governs localized features (eye shape, mouth structure).
- Fine-grained (Layers 10-14): Represents high-frequency details (skin texture, lighting).
  These features are aggregated to form a holistic 3D latent vector ( $f_{3D}$ ) that captures the structural identity.
Conditioning Pipeline: The diffusion model is conditioned on three distinct feature sets:
1. Identity Features ( $f_{ID}$ ): Extracted from the source image using a pre-trained face recognition model (e.g., ArcFace) and projected via an MLP.
2. Landmark Features ( $f_{LM}$ ): Derived from 2D facial landmarks of the target image to preserve pose and expression.
3. 3D Latent Features ( $f_{3D}$ ): The aggregated 3D representations of both source and target images, which guide the disentanglement of identity from pose/expression.
  These features are combined via weighted aggregation and injected into the UNet via cross-attention mechanisms.

Training Objective

The training process is formulated as a conditional inpainting task and optimized using a two-step objective function:

Reconstruction Loss ( $L_{Recon}$ ): A standard denoising diffusion loss ensuring the model can reconstruct the target face from noise.
Identity Preservation Loss ( $L_{ID}$ ): Calculated using cosine similarity between the swapped face and the source identity (via ArcFace) to ensure the source identity is retained.
Perceptual Similarity Loss ( $L_{PS}$ ): Uses LPIPS to ensure the generated face blends seamlessly with the target background and maintains visual coherence.

The total loss is a weighted sum: $L_{Total} = L_{Recon} + w_{ID}L_{ID} + w_{PS}L_{PS}$ .

3. Key Contributions

The paper outlines three primary contributions:

First 3D-Guided Diffusion Framework: DiffSwap++ is the first face-swapping framework to leverage 3D facial latent features to guide a diffusion-based generation pipeline. It integrates 3D geometric cues during training while remaining fully 2D at inference.
Superior Performance and Robustness: Extensive experiments on CelebA-HQ, FFHQ, and CelebV-Text demonstrate that the method outperforms state-of-the-art baselines in identity retention and perceptual realism. A user study confirms that the generated swaps are more visually convincing than prior methods.
Novel Biometric Evaluation: The authors introduce a rigorous evaluation perspective by analyzing the trade-off between the Impostor Attack Presentation Acceptance Rate (IAPAR) and the False Reject Rate (FRR). This provides a quantitative measure of how effectively the method preserves source identity against biometric recognition systems.

4. Experimental Results

The method was evaluated on three public datasets (CelebA-HQ, FFHQ, CelebV-Text) against baselines including DiffFace, DiffSwap, REFace, HiFiFace, and SimSwap.

Identity Preservation: DiffSwap++ achieved the highest Top-1 (94.7%) and Top-5 (98.2%) identity retrieval accuracy on CelebA-HQ, and 95.1%/98.4% on FFHQ. This significantly outperforms diffusion-based baselines like DiffSwap (16.9% Top-1) and GAN-based methods like HiFiFace (86.4% Top-1).
Visual Quality: The method achieved the lowest Fréchet Information Distance (FID) of 6.42 on CelebA-HQ and 6.57 on FFHQ, indicating high photorealism.
Pose and Expression: While some methods achieved slightly better pose/expression scores by making minimal changes to the target face (and thus failing to transfer identity), DiffSwap++ maintained a balanced compromise, preserving target attributes with minimal degradation (Pose $\ell_2$ distance: 3.29; Expression: 1.00).
Biometric Analysis: In the IAPAR vs. FRR analysis, DiffSwap++ demonstrated high-fidelity swaps that successfully preserved the source identity, maintaining a high threshold for acceptance by biometric systems.
Efficiency: DiffSwap++ requires 3.35 seconds per image for inference, making it approximately 1.5x faster than DiffSwap and 11x faster than DiffFace, while utilizing 18.69 GB of VRAM.
Ablation Studies: Experiments confirmed that the "Avg. Pool" strategy for 3D feature aggregation (combining coarse, middle, and fine-grained layers) yields the best identity retrieval (79.30%) compared to using only specific layers. The study also verified that both identity and landmark features are essential for training, and that 4 DDIM steps provide the optimal balance during training.

5. Significance and Claims

The paper claims that DiffSwap++ addresses the critical limitation of 3D awareness in diffusion-based face swapping. By incorporating 3D latent features during training, the model effectively disentangles identity from pose and expression, resulting in swaps that are both photorealistic and highly effective at transferring the source identity.

The authors emphasize that their method does not require 3D features at inference, making it practical for deployment. The significance of the work lies in its ability to produce face swaps that are difficult to distinguish from authentic images (high IAPAR) while maintaining the structural integrity of the target face. The paper concludes that DiffSwap++ offers the most effective balance of high mean fidelity and consistent generation quality, validated through both quantitative metrics and human user studies. The code is made publicly available to facilitate further research in this domain.