Bypassing Copyright Protection in Diffusion-based Customization via Two-Stage Latent Feature Optimization

Technical Summary: Bypassing Copyright Protection in Diffusion-based Customization via Two-Stage Latent Feature Optimization

Problem Statement
The rapid advancement of diffusion-based generative models, particularly through customization techniques like DreamBooth and Textual Inversion, has introduced significant intellectual property risks. Malicious actors can fine-tune these models on copyrighted images to forge visual content or counterfeit identities. In response, researchers have developed adversarial defense mechanisms (e.g., AdvDM, SimAC, DisDiff) that inject targeted perturbations into input images. These perturbations aim to disrupt the model's ability to learn personalized concepts, thereby suppressing unauthorized generation. However, the authors observe that these defenses rely on a specific vulnerability: they disrupt the stable semantic mapping between input images and their latent space representations within Latent Diffusion Models (LDMs). While effective against naive attacks, these defenses introduce high-frequency noise and corrupt the latent features, creating an opportunity for a targeted bypass.

Methodology: Two-Stage Latent Feature Optimization (TS-LFO)
The paper proposes TS-LFO, an attack framework designed to restore the disrupted latent-image mapping and bypass state-of-the-art (SOTA) copyright defenses. The method operates under a white-box threat model where the attacker has access to the protected image ($x_{adv}$) and the architecture of the LDM (VAE and Unet), but not the original clean image ($x$) or the specific defense parameters.

The framework consists of two sequential optimization stages:

1. Latent Denoising Stage:
   The primary goal is to eliminate high-frequency adversarial perturbations and restore the semantic consistency between the latent code and the input image. The authors introduce a joint optimization objective:

   • Latent-Image Alignment Loss ($\mathcal{L}_{align}$): This loss minimizes the discrepancy between the protected image ($x_{adv}$) and the reconstruction of the optimized latent feature ($z$) via the VAE decoder. Surprisingly, the authors find that traditional Mean Squared Error (MSE) outperforms perceptual metrics like LPIPS or SSIM in this context, as advanced metrics may misinterpret adversarial perturbations as legitimate features.
   • Latent Diffusion Loss ($\mathcal{L}_{DM}$): This loss simulates the training dynamics of the downstream Unet. It minimizes the noise prediction error of a frozen, pre-trained Unet when processing the noisy version of the optimized latent feature. By reducing this proxy training loss, the attack ensures the latent feature is "easy to learn" for the diffusion model.
   • Timestep-Varying Weighting: To balance these two losses, the authors employ a linearly varying weight factor ($\lambda_t$) dependent on the diffusion timestep. This prevents the optimization from prioritizing one loss term over the other at different stages of the denoising trajectory.

2. Latent Reconstruction Stage:
   Following the denoising stage, the optimized latent feature may still lack high-fidelity structural details or contain residual perturbations. This stage applies pixel-level constraints to refine the latent representation:

   • The latent feature is decoded to the image space, clamped within a small range ($\epsilon_{rec}$) around the original protected image to preserve semantic fidelity, and then re-encoded. This step recovers low-frequency semantic information and ensures the final latent feature is suitable for high-quality generation without deviating significantly from the source content.

Key Contributions

• Novel Attack Framework: The paper introduces TS-LFO, a two-stage attack that specifically targets the mechanism by which copyright defenses disrupt latent-image mappings. It utilizes a combination of alignment and diffusion losses with dynamic weighting to restore these mappings.
• Mechanistic Analysis: The authors provide a theoretical and empirical analysis demonstrating that existing defenses work by corrupting the encoder's latent representation and increasing the Unet's training loss. TS-LFO is designed to reverse these specific effects.
• Comprehensive Evaluation: The study evaluates TS-LFO against multiple SOTA defenses (Anti-DreamBooth, AdvDM, SimAC, DisDiff, PhotoGuard) and compares its performance against advanced attack baselines (DiffPure, GrIDPure, IMPRESS) across diverse datasets (CelebA-HQ, LSUN-Cat, LSUN-Sheep).

Experimental Results
Extensive experiments demonstrate that TS-LFO consistently outperforms existing copyright attack methods:

• Defense Bypass: TS-LFO significantly reduces the effectiveness of SOTA defenses. For instance, in Textual Inversion experiments, the method recovers FID scores (a measure of generation quality) to near-baseline levels, whereas defended models without attack show significant degradation.
• Superiority over Baselines: Compared to DiffPure, GrIDPure, and IMPRESS, TS-LFO achieves lower FID (better quality), higher CLIP image similarity, and better identity consistency (measured by ISM) on face datasets.
• Generalization: The method proves effective across different image distributions (faces, cats, sheep) and different customization techniques (DreamBooth and Textual Inversion).
• Ablation Studies: The authors validate the necessity of each component, showing that the timestep-varying weights and the specific choice of MSE for alignment loss are critical for optimal performance. They also note that while TS-LFO requires approximately 6 minutes per image, this is acceptable compared to the 15–30 minutes required for the actual customization training.

Significance and Claims
The paper claims that TS-LFO represents a significant challenge to the robustness of current adversarial copyright protection strategies. By exploiting the specific vulnerability of latent-space disruption, the authors demonstrate that current defenses can be systematically bypassed to achieve high-fidelity unauthorized generation.

The authors position this work not as an endorsement of copyright infringement, but as a necessary stress test for the field. They argue that protection researchers often focus on traditional attacks and overlook robustness against advanced, targeted attacks like TS-LFO. Consequently, the paper calls for the development of more sophisticated defense mechanisms that account for feature vulnerabilities in latent spaces and emphasizes the need for ethical frameworks to balance copyright protection with open innovation in generative AI. The authors make their source code publicly available to facilitate further research into robust defenses.