Technical Summary: Evaluating Dataset Watermarking for Fine-tuning Traceability of Customized Diffusion Models

1. Problem Statement

The rapid adoption of fine-tuning techniques (e.g., DreamBooth, LoRA, Textual Inversion) for diffusion models has enabled the generation of content mimicking specific identities and artistic styles. However, this capability introduces significant security and ethical risks, primarily regarding copyright infringement (unauthorized use of proprietary images for training) and accountability (tracing the source of harmful or inappropriate generated content).

To address this, dataset watermarking has been proposed, where imperceptible watermarks are embedded into training images. Theoretically, these watermarks should persist through the fine-tuning process and remain detectable in the model's outputs, enabling post-hoc attribution. However, existing approaches suffer from a lack of a unified evaluation framework, making it difficult to assess their effectiveness across diverse real-world scenarios. Furthermore, the robustness of these methods against adversarial removal attacks remains largely unexplored.

2. Methodology

2.1. Unified Threat Model and Evaluation Framework

The authors formalize a universal threat model involving two parties: the Image Owner (who embeds watermarks for copyright protection and traceability) and the Image User (who may use mixed datasets or apply post-processing to remove watermarks).

Based on this model, the paper proposes a comprehensive evaluation framework comprising three dimensions:

1. Universality: The ability of watermarking methods to function across diverse fine-tuning paradigms (e.g., Textual Inversion, DreamBooth, LoRA, Full-UNet) and generation tasks (face, virtual objects, art styles).
2. Transmissibility: The capacity of the watermark signal to propagate through the fine-tuning process even when only a subset of the training data is watermarked (e.g., 20%–80% mixing ratios).
3. Robustness: The resilience of the watermark against both common image processing operations (Gaussian noise, blur, JPEG compression) and targeted adversarial removal attacks.

2.2. Comprehensive Benchmark

The authors constructed a benchmark evaluating four state-of-the-art open-source dataset watermarking methods: DIAGNOSIS, DiffusionShield, SIREN, and WatermarkDM.

• Datasets: CelebA-HQ (facial identity), Pokémon (virtual objects), and WikiArt (artistic styles).
• Metrics: Generation quality (FID, CLIP-T) and watermark detection accuracy (Acc).
• Settings: Experiments were conducted on Stable Diffusion v1.4 (SD1.4) with various fine-tuning configurations, including freezing or training the text encoder.

2.3. Proposed Removal Approach (DeAttack)

To evaluate the robustness dimension, the paper introduces DeAttack, a practical watermark removal framework.

• Mechanism: DeAttack utilizes a unified autoencoder backbone that combines multi-domain degradations (pixel-space and latent-space) with high-quality restoration.
• Process: It simulates degradation (e.g., Gaussian blur, noise injection) followed by adaptive restoration using pre-trained networks (IRNeXt for deblurring, SwinIR for denoising and JPEG artifact correction).
• Goal: To estimate a clean reconstruction that minimizes residual watermark energy while preserving perceptual image quality.

3. Key Results

3.1. Universality and Transmissibility

• Universality: DiffusionShield demonstrated the highest universality, achieving near 100% detection accuracy across all fine-tuning methods and datasets. SIREN performed poorly (~50% accuracy, equivalent to random guessing), while DIAGNOSIS and WatermarkDM showed variable performance depending on the specific fine-tuning method (e.g., performance dropped significantly when the text encoder was trained).
• Transmissibility: When using mixed datasets (e.g., 20% watermarked images), DiffusionShield maintained high detection accuracy (approaching 100%), whereas DIAGNOSIS and WatermarkDM saw significant drops in accuracy as the proportion of watermarked images decreased.

3.2. Robustness Against Common Distortions

Existing methods generally exhibited resilience against common image processing operations such as Gaussian blur, JPEG compression, and Gaussian noise. Detection accuracy remained high for DiffusionShield and WatermarkDM under these conditions.

3.3. Vulnerability to Targeted Removal

The most critical finding concerns the Robustness dimension against adversarial attacks.

• DeAttack Effectiveness: The proposed DeAttack method successfully removed dataset watermarks from all evaluated methods, reducing detection accuracy to near-random levels (e.g., ~50-55% for DiffusionShield, which was previously ~100%).
• Preservation of Quality: Crucially, DeAttack achieved this removal while maintaining the perceptual quality of the images and the performance of the fine-tuned models (FID and CLIP-T scores remained comparable to clean baselines).
• Generalizability: The effectiveness of DeAttack was validated across different diffusion architectures (SD1.4 and SDXL) and various fine-tuning strategies, indicating that current watermarking designs are not robust against principled removal attacks.

4. Significance and Contributions

The paper makes three primary contributions:

1. Principled Evaluation Foundation: It establishes a unified threat model and a three-dimensional evaluation framework (Universality, Transmissibility, Robustness) to systematically assess dataset watermarking for diffusion models.
2. Comprehensive Benchmark: It provides a rigorous benchmark revealing that while existing methods (particularly DiffusionShield) are effective in universality and transmissibility, they lack robustness against targeted removal.
3. Vulnerability Exposure: By proposing DeAttack, the paper demonstrates that current dataset watermarking techniques are fragile against advanced removal strategies. This highlights a critical gap in current designs and underscores the urgent need for developing more robust, adversary-aware watermarking strategies.

The authors conclude that while dataset watermarking shows promise for traceability, its current implementations are insufficient for real-world security scenarios where malicious actors may employ targeted removal techniques. The work serves as a call to action for the community to develop watermarking methods that can withstand such adversarial pressures.