Technical Summary: DiffAttack

Problem Statement

Deep face recognition (FR) systems, while effective for authentication and surveillance, possess decision boundaries narrow enough to be vulnerable to adversarial attacks. Existing adversarial methods targeting facial biometrics face significant limitations:

• Noise-based attacks: While often imperceptible, they lack robustness against real-world variations (e.g., lighting changes) and exhibit poor transferability across different models.
• Patch-based attacks: These introduce conspicuous visual artifacts, compromising stealth and making them unsuitable for natural image sharing.
• Semantic/Makeup-based attacks: While improving perceptual quality, they often rely on rigid reference images, manual semantic cues, or large-scale makeup datasets that introduce demographic bias and limit generalization.
• GAN-based approaches: These typically require retraining for new target identities and are constrained by the generative manifold of the GAN.
• Existing Diffusion-based methods: Approaches like DiffAM or Adv-Diffusion often rely on iterative optimization, target-specific retraining, or rigid heuristic masks, leading to boundary artifacts and reduced efficiency.

The core challenge is to generate adversarial faces that are photorealistic, identity-aligned with a target, and imperceptible to human observers, while maintaining high transferability across diverse, unseen black-box FR systems.

Methodology: DiffAttack

The authors propose DiffAttack, a novel framework that performs adversarial generation via latent-space optimization using a frozen Latent Diffusion Model (specifically Stable Diffusion v2.1).

Core Architecture and Process

1. Latent Encoding: A clean source image ($x_{src}$) is encoded into a latent representation ($z_{src}$) using a frozen Variational Autoencoder (VAE) encoder.
2. Noise Injection: A fixed amount of Gaussian noise is added at a specific timestep $t$ to create a noisy latent input for the U-Net backbone.
3. LoRA-Augmented U-Net: Instead of fine-tuning the massive U-Net parameters, the framework injects Low-Rank Adaptation (LoRA) adapters specifically into the cross-attention layers (Q, K, V projections). These lightweight, trainable matrices are optimized to redirect the diffusion process toward the target identity embedding.
4. Optimization Loop:
   • The U-Net predicts the noise, allowing reconstruction of the clean latent ($x_0$).
   • The latent is decoded to generate the adversarial face in pixel space.
   • Multi-Source Feedback: The generated face is passed through an ensemble of frozen FR models (IR152, IRSE50, MobileFace) to calculate the identity distance between the adversarial face and a target identity.
   • Gradient Backpropagation: Gradients from the FR models are backpropagated solely to the LoRA weights, iteratively refining the adversarial features until the source is misidentified as the target.

Loss Function

The optimization minimizes a unified multi-objective loss ($L_{total}$):

• Ensemble Identity Loss ($L_{id}$): Pushes the generated embedding toward the target embedding using a cosine similarity hinge.
• Directional Guidance ($L_{dir}$): Ensures the latent shift follows semantic identity trajectories (source $\to$ target) rather than superficial shortcuts.
• Source Suppression ($L_{src}$): Penalizes the optimizer to prevent drifting back to the original source identity.

Global Harmonization

Unlike methods using rigid segmentation masks or localized blending, DiffAttack optimizes the full-image latent representation. This allows the diffusion model's generative priors to naturally harmonize the identity shift with the original background, lighting, and peripheral attributes, eliminating boundary halos and ensuring photorealism.

Key Contributions

1. Novel Latent-Space Evasion Framework: A unified pipeline performing optimization in the latent space using LoRA-augmented cross-attention layers. This avoids conspicuous artifacts of pixel-space noise and reduces computational overhead compared to global fine-tuning.
2. Black-Box Transferability: A multi-source optimization strategy leveraging white-box feedback from diverse surrogate FR backbones (IR152, IRSE50, MobileFace). The method injects adversarial semantics that generalize to unseen targets (e.g., FaceNet) without access to their internal parameters.
3. Global Latent Harmonization: By optimizing the adversarial signal across the entire latent space rather than using localized masks, the framework ensures the identity shift is seamlessly integrated with the surrounding context, avoiding boundary artifacts.
4. State-of-the-Art Performance: Extensive evaluation on FFHQ and CelebA-HQ datasets demonstrates superior performance over existing noise-based, makeup-based, and semantic-based methods.

Experimental Results

The framework was evaluated on FFHQ and CelebA-HQ datasets under a strict black-box setting.

• Attack Success Rate (ASR): DiffAttack achieved a new state-of-the-art average ASR of 84.86% across multiple FR models.
  • It surpassed traditional noise-based methods by over 15.28% and semantic-based approaches by approximately 5.21%.
  • Specific high success rates were observed against MobileFace (95.03%) and IRSE50 (94.24%).
  • When attacking FaceNet as the black-box target, the surrogate ensemble consisted of IR152, IRSE50, and MobileFace (excluding FaceNet). Conversely, when attacking MobileFace, the surrogate ensemble included FaceNet, IRSE50, and IR152.
• Perceptual Quality:
  • SSIM: Achieved 0.961, indicating high structural similarity to the source.
  • PSNR: 24.54 dB, showing subtle embedding of perturbations.
  • FID: 27.58, indicating the generated images remain within the natural image manifold.
• Visual Comparison: Unlike noise-based methods (which cause "ghosting") or makeup-based methods (which often look artificial), DiffAttack preserves the source's original texture, expression, and lighting. While Face++ scores are used in Figure 1 to illustrate identity alignment, the formal evaluation metrics rely on the IR152, IRSE50, MobileFace, and FaceNet models.

Significance and Claims

The paper claims that DiffAttack represents a significant advancement in biometric security evaluation by demonstrating that latent-space optimization with LoRA can generate adversarial examples that are both highly effective against black-box systems and visually indistinguishable from genuine photographs to human observers.

The authors emphasize that their work is intended for defensive security analysis and privacy protection. By revealing these vulnerabilities, the research aims to inform the development of more robust biometric verification systems capable of withstanding advanced generative AI threats. The study strictly adheres to ethical guidelines, utilizing only public research datasets and refraining from releasing specific pre-trained weights for individual impersonation.