DiffusionPrint: Learning Generative Fingerprints for Diffusion-Based Inpainting Localization

1. Problem Statement

The rise of text-guided generative AI models (e.g., Stable Diffusion, FLUX, Adobe Firefly) has introduced a new class of image forgery: fully-regenerative inpainting. Unlike traditional splicing or partial inpainting, these models operate in a compressed latent space and decode the entire image representation via a Variational Autoencoder (VAE) decoder.

Key Challenges:

• Destruction of Forensic Traces: Traditional Image Forgery Localization (IFL) methods rely on detecting discontinuities in sensor noise (PRNU) or acquisition artifacts. Because fully-regenerative models reconstruct every pixel (even those outside the inpainting mask), they destroy the original camera noise patterns, rendering existing forensic signals ineffective.
• Spectral Shift: The latent reconstruction process induces a pervasive spectral shift across the entire image, altering frequency statistics. This confounds detectors that rely on absolute frequency differences between real and fake regions, as "real" regions are shifted to resemble "fake" spectral profiles.
• Lack of Generalization: Existing models trained on splicing or partial inpainting fail when applied to fully-regenerative outputs without retraining.

2. Methodology: DiffusionPrint

The authors propose DiffusionPrint, a patch-level contrastive learning framework designed to learn a forensic signal robust to latent reconstruction artifacts. Instead of detecting noise, it learns a generative fingerprint specific to the diffusion model used.

Core Components:

1. Asymmetric Positive Pair Construction:

   • Generated Patches (Inside Mask): For images generated by a specific model $g_i$, the model performs inpainting using the same model's variant. The original generated patch and the newly inpainted patch at the same spatial location form a positive pair. This teaches the encoder to recognize the shared generative fingerprint of the model.
   • Authentic Patches (Outside Mask): For real images, the model performs inpainting using various generators. Patches outside the mask (which are semantically unchanged but spectrally reconstructed) are paired with their original counterparts. This forces the encoder to become invariant to the latent reconstruction artifacts introduced by the VAE, learning to ignore the spectral shift on authentic regions.

2. MoCo-Style Contrastive Training:

   • Architecture: Uses a DnCNN backbone (17 layers) to produce dense feature maps.
   • Momentum Encoder: A momentum-updated encoder ($f_{\bar{\theta}}$) generates stable embeddings for positive pairs and the negative queue.
   • Cross-Category Hard Negative Mining: A memory queue stores embeddings tagged by category (Real vs. Inpainted). For each anchor, the system retrieves the top-$k$ most similar embeddings from the opposite category. This focuses the learning on the hardest decision boundaries (e.g., a real patch that looks like a generated one).
   • Loss Function: Optimizes an InfoNCE loss combined with a Generator-Aware Classification Head. The classification head (trained jointly) predicts whether a patch is Real or which specific generator (SD 2.1, SDXL, Flux) created it. This auxiliary task encourages the backbone to learn distinct, model-specific fingerprints.

3. Integration:

   • DiffusionPrint acts as a "drop-in" replacement for noise-based modalities (like Noiseprint++) in fusion-based IFL frameworks. It outputs a dense forensic feature map that is fused with RGB features to localize forgery.

3. Key Contributions

• Novel Framework: Introduction of DiffusionPrint, the first patch-level contrastive framework specifically designed to handle the spectral distortions of fully-regenerative inpainting.
• Invariance Learning: A unique pairing strategy that explicitly teaches the model to be invariant to VAE reconstruction artifacts on authentic regions while sensitive to the generative fingerprint of inpainted regions.
• Cross-Category Hard Negatives: A training mechanism that mines the most confusing samples across categories to refine the decision boundary.
• Generator Awareness: The inclusion of a classification head that improves the discriminative power of the learned features by forcing the model to distinguish between different generative architectures.

4. Experimental Results

The method was evaluated on the TGIF benchmark (specifically the Fully Regenerated subset) across three fusion frameworks: TruFor, MMFusion, and a Lightweight Baseline.

• Performance Gains: DiffusionPrint consistently outperformed the state-of-the-art Noiseprint++ modality.
  • Achieved up to +28% improvement in F1 score on random mask types (unseen during training).
  • Significant gains were observed on Flux variants (e.g., Flux dev, Flux schnell), demonstrating generalization beyond U-Net-based models.
• Generalization:
  • Unseen Architectures: When trained only on SD 2.1 and SDXL (excluding Flux), the model still achieved competitive or superior performance on unseen Flux variants, proving it learns a fundamental generative signal rather than overfitting to specific model artifacts.
  • Unseen Masks: The model generalized well to random mask types not seen during fine-tuning, indicating it captures the manipulation process rather than mask-specific artifacts.
• Ablation Studies:
  • The proposed pairing strategy improved accuracy from ~75% to 86.35% over standard MoCo baselines.
  • The Generator-Aware Classification Head further boosted linear probing accuracy to 88.17%.
  • Optimal performance was found with 512 hard negatives and 64 output channels.

5. Significance and Conclusion

• Paradigm Shift: DiffusionPrint addresses the critical limitation of current forensic tools against modern generative AI. It moves away from relying on sensor noise (which is destroyed by latent decoding) to learning the intrinsic "fingerprint" of the generative process itself.
• Robustness: By learning to ignore the spectral shifts caused by the VAE decoder, the method provides a reliable signal for localization even when the entire image is reconstructed.
• Practical Impact: As a modular component, it can be easily integrated into existing forensic pipelines (like TruFor) to significantly boost their ability to detect sophisticated, text-guided forgeries, offering a crucial defense against disinformation and visual evidence fabrication.

The code is publicly available, facilitating further research into generative fingerprinting and robust image forensics.