Technical Summary: Face Inpainting with Identity Preserving Latent Diffusion Models

Problem Statement

Face inpainting aims to reconstruct missing or occluded facial regions with visual realism and semantic coherence. While essential for applications like photo restoration and digital forensics, a critical challenge remains: preserving the subject's unique identity in the reconstructed output. Standard diffusion models, despite their generative fidelity, often lack explicit mechanisms to encode subject-specific identity, causing reconstructed regions to drift toward an "average" facial appearance. Existing identity-aware solutions typically rely on computationally expensive per-subject fine-tuning (e.g., MyStyle, Custom Diffusion) or exhibit limited robustness to diverse occlusions and poses. Furthermore, evaluating identity preservation is complicated by existing datasets (e.g., CelebA-HQ, FFHQ) that often employ random or large-area occlusions, making it difficult to distinguish between plausible facial generation and true identity recovery.

Methodology: ID-ControlNet

The authors propose ID-ControlNet, an identity-preserving face inpainting framework built upon Latent Diffusion Models (LDMs). The approach integrates facial identity cues directly into the diffusion process without requiring per-subject fine-tuning.

Architecture

• Base Model: The framework utilizes a frozen Stable Diffusion (SD1.5) latent diffusion backbone for image generation.
• ControlNet Extension: Inspired by ControlNet, the authors introduce a trainable auxiliary branch that injects external conditioning signals. Unlike standard ControlNet which uses spatial cues (edges, depth), ID-ControlNet injects identity embeddings.
• Identity Extraction: A pretrained face recognition network (ArcFace) extracts identity embeddings from the unmasked reference image.
• Latent Integration: The identity embeddings are processed by the control branch, which upsamples them from $8 \times 8$ to $64 \times 64$ to align with the U-Net decoder pathway. Zero-initialized convolution layers inject these identity-guided features into the frozen diffusion backbone, guiding the denoising trajectory toward the target identity while preserving global coherence.

Training Strategy

The model is trained using a composite objective function that combines standard denoising loss with explicit identity supervision:

1. Denoising Loss ($\mathcal{L}_{denoise}$): Standard LDM loss to predict noise in the latent space.
2. Identity Consistency Loss ($\mathcal{L}_{id}$): A cosine distance loss ensuring the embedding of the generated image matches the conditioning identity embedding.
3. Triplet Loss ($\mathcal{L}_{trip}$): A batch-level loss to maximize separability between the target identity and negative samples within the batch.
4. Efficiency: To avoid the computational cost of backpropagating through a full $T$-step denoising chain, the authors employ a single-step denoising approximation. The latent is perturbed to a small timestep, one denoising step is applied, and the resulting image is decoded to compute the embedding loss.

Key Contributions

1. ID-ControlNet Framework: A novel architecture that conditions latent diffusion on identity embeddings extracted from a pretrained face recognition network, enabling identity-preserving inpainting without per-subject fine-tuning.
2. Training Strategy: An identity consistency and triplet loss strategy that enforces alignment between generated faces and target identity representations in a self-supervised manner.
3. E-Mask Dataset: The introduction of a new benchmark dataset where occlusions are strictly restricted to the eye region. This addresses the limitation of existing datasets by focusing on the most identity-informative facial area, allowing for a more precise evaluation of identity restoration capabilities.
4. Zero-Shot Capability: The method achieves state-of-the-art identity preservation in a zero-shot setting, eliminating the need for the costly adaptation or fine-tuning required by previous diffusion-based personalization methods.

Experimental Results

The authors evaluated ID-ControlNet on CelebA-HQ, FFHQ, and the proposed E-Mask dataset, comparing it against standard diffusion baselines (SD1.5, SD2) and identity-aware methods (PVA, MyStyle, Custom Diffusion).

• Perceptual Quality: ID-ControlNet achieves perceptual quality (FID, mFID, LPIPS) comparable to standard diffusion baselines, indicating that identity conditioning does not significantly degrade visual realism.
• Identity Preservation:
  • On the CelebA-HQ-IDI-5 benchmark, ID-ControlNet matches the identity fidelity of PVA (a state-of-the-art method) while operating in a zero-shot setting, whereas PVA typically requires fine-tuning.
  • On the E-Mask dataset (eye-specific occlusions), ID-ControlNet significantly outperforms both the baseline and PVA (No-Tune) in identity similarity scores (0.78 vs. 0.63 for PVA No-Tune), demonstrating superior recovery of identity-defining features.
• Generalization: Models trained on CelebA-HQ-IDI generalize effectively to the unseen FFHQ dataset without additional fine-tuning.

Significance and Claims

The paper claims that ID-ControlNet represents a significant advancement in balancing visual realism with identity fidelity in face inpainting. By integrating identity embeddings directly into the diffusion process via a ControlNet-inspired architecture, the method overcomes the "average face" drift of standard models and the scalability limitations of fine-tuning-based approaches.

The authors emphasize that their approach provides a lightweight, efficient solution for identity-consistent generation that is particularly valuable for downstream applications such as face recognition, digital forensics, and human-computer interaction, where subtle identity distortions can critically degrade system performance or trust. The introduction of the E-Mask dataset is highlighted as a necessary step for rigorous evaluation, as it isolates the most critical facial regions for identity recognition.

The authors conclude that while their current results are promising, future work should focus on upgrading the backbone models and conducting user studies to further validate the robustness of image quality and identity preservation.