High-Fidelity Diffusion Face Swapping with ID-Constrained Facial Conditioning

1. Problem Statement

Face swapping aims to transfer a source face's identity onto a target face while preserving the target's attributes (pose, expression, lighting). While Generative Adversarial Networks (GANs) have been the standard, they suffer from limited image quality and mode collapse. Diffusion models offer superior generative capabilities but face two specific challenges in face swapping:

• Identity-Attribute Conflict: Identity preservation and attribute alignment often work in opposition during training. Pushing the model to match the source identity can degrade the target's expression/pose, and vice versa.
• Joint Conditioning Failure: Existing diffusion-based methods often inject identity and attribute conditions simultaneously in a single-stage training scheme. This leads to sub-optimal training, where the model struggles to balance conflicting gradients, resulting in artifacts, identity leakage, or distorted faces.

2. Methodology

The authors propose a novel framework called Identity-Constrained Facial Conditioning, which decouples the injection of identity and attributes and employs a multi-stage training strategy.

A. Decoupled Facial Condition Injection

To prevent condition leakage and interference, the authors introduce a two-path injection mechanism:

• Data-Wise Decoupling: Instead of using augmented versions of the same image (which risks leaking attributes into the identity signal), they use paired images from the same identity but with different attributes.
• Two-Path Feature Extraction:
  • Identity Path ($c_{id}$): Extracted from the source face using a pre-trained ArcFace recognizer and DINOv2 (for fine-grained visual details like skin texture). These are fused via attention mechanisms.
  • Attribute Path ($c_{attr}$): Extracted from the target face using the feature encoder of SimSwap (a GAN-based method), capturing expression, pose, and spatial semantics.
• Fusion: The identity and attribute features are fused using an attention layer where the identity embedding queries the attribute embedding. A fusion factor ($\lambda_{fuse}$) controls the balance, allowing the model to prioritize identity initially.

B. Multi-Stage Training Scheme

The training process is divided into three distinct stages to resolve the identity-attribute conflict:

1. Stage 1: Identity-Oriented Tuning

   • Goal: Adapt the pre-trained text-to-image diffusion model to the face-swapping task and narrow the solution space to outputs consistent with the source identity.
   • Method: The model is fine-tuned using only identity conditions ($\lambda_{fuse} = 0$). The background is preserved via an inpainting mask, while the foreground is generated based solely on the source identity.

2. Stage 2: Identity-Constrained Attribute Tuning

   • Goal: Align the generated face with the target's attributes (expression, pose) without forgetting the identity.
   • Method: Attribute conditions are injected ($\lambda_{fuse} = 1$). To prevent "catastrophic forgetting" of the identity:
     • The output weights of the fusion module's attention layer are zero-initialized (cold start for attributes).
     • The spatial enhancement factor for identity ($\lambda_{id}$) is reduced (to 0.2) to weaken the identity constraint slightly, allowing the model to learn attribute alignment.

3. Stage 3: Post-Training Identity Refinement

   • Goal: Enhance fidelity and identity similarity without compromising sample quality.
   • Method: Unlike previous methods that add Identity (ID) loss directly to the diffusion noise-prediction loss (which relaxes the ELBO bound and degrades quality), this stage treats the entire 50-step DDIM sampling process as a cascaded end-to-end generative model.
   • Loss Function: A combined loss of GAN loss (adversarial) and ID loss is applied. To manage memory costs, gradients are calculated only on randomly sampled steps ($k$) from the DDIM trajectory.

3. Key Contributions

1. Decoupled Condition Injection: A novel strategy using paired data and two-path feature extraction to separate identity and attribute signals, preventing leakage and conflict.
2. Identity-Constrained Conditioning Scheme: A multi-stage training pipeline that prioritizes identity preservation first, then fine-tunes for attribute alignment, effectively solving the gradient conflict problem.
3. State-of-the-Art Performance: The proposed model achieves new benchmarks in both qualitative and quantitative metrics, outperforming existing GAN and diffusion-based methods.

4. Experimental Results

The model was evaluated on the FFHQ dataset against strong baselines (SimSwap, InfoSwap, E4S, DiffFace, DiffSwap, REFace).

• Quantitative Metrics:
  • FID (Fréchet Inception Distance): Achieved 3.61, significantly outperforming all competitors (next best was 4.26 for InfoSwap). This indicates superior overall sample quality and realism.
  • ID Retrieval: Achieved 97.90% (Top-1) and 99.70% (Top-5), demonstrating the highest identity preservation accuracy.
  • Pose/Expression: Remained competitive, matching the best scores of GAN-based methods (e.g., SimSwap) in expression consistency.
• Qualitative Results:
  • Generated faces exhibit natural skin textures and lighting, leveraging the pre-trained foundation model's capabilities.
  • Successfully handles out-of-distribution cases, such as stylized paintings and artistic photos, maintaining texture harmony.
• Human Evaluation:
  • In a user study, the proposed method received the highest votes for Fidelity (57.1%) and Attribute Consistency (33.2%), and the second-highest for Identity Preservation (31.6%), proving a superior balance between all factors.

5. Significance

This paper addresses a fundamental limitation in diffusion-based face swapping: the inability to jointly optimize for conflicting identity and attribute conditions. By decoupling these conditions and employing a staged training approach, the authors demonstrate that diffusion models can surpass GANs in face swapping tasks.

The significance extends beyond face swapping; the identity-constrained, multi-stage conditioning framework offers a generalizable solution for other conditional generation tasks where multiple, potentially conflicting constraints (e.g., style vs. content) must be balanced. It establishes a new state-of-the-art for high-fidelity, controllable image generation.