Technical Summary: AdaptPrompt for Generalizable Deepfake Detection

1. Problem Statement

The paper addresses the critical generalization failure in deepfake detection. Current detectors trained on Generative Adversarial Network (GAN) imagery tend to overfit to specific, periodic frequency artifacts (e.g., "checkerboard" patterns from upsampling) inherent to GANs. Consequently, these models fail to detect images generated by diffusion models or commercial tools (e.g., Midjourney, DALL-E 3), often misclassifying them as "real" due to a "sink-label" effect where the "fake" class is defined solely by the training data's specific artifacts.

The authors identify two root causes for this lack of generalizability:

1. Training Data Bias: Existing large-scale corpora (like ProGAN/LSUN) are GAN-centric, teaching models to recognize GAN-specific fingerprints rather than general synthetic traces.
2. Adaptation Strategy: Standard fine-tuning or linear probing of Vision-Language Models (VLMs) like CLIP may either overfit to the training generator or discard the low-level forensic traces necessary for detection during semantic alignment.

2. Methodology

The proposed solution, AdaptPrompt, operates on two fronts: introducing a new training corpus and a parameter-efficient adaptation framework for CLIP.

2.1 The Diff-Gen Corpus

The authors introduce Diff-Gen, a balanced training dataset comprising:

• 100,000 diffusion-generated images created using Stable Diffusion v1.5.
• 100,000 real images from the LSUN dataset.
• Design: The real subset mirrors the class distribution of the synthetic data (20 LSUN categories) to prevent detectors from learning semantic biases.
• Spectral Analysis: Unlike GAN data, which exhibits sharp, periodic high-frequency peaks, Diff-Gen shows a broad, non-periodic elevation of high-frequency energy. The authors hypothesize that training on this "broadband noise" forces models to learn generalizable generation artifacts rather than specific architectural fingerprints.

2.2 AdaptPrompt Architecture

AdaptPrompt is a parameter-efficient adaptation of the CLIP model (specifically ViT-L/14). It freezes the pre-trained backbone and trains only ~0.1% of the parameters (approx. 602k) via two modules:

1. Visual Adapter: A residual bottleneck adapter attached to the output of the frozen visual encoder. It uses a down-projection, ReLU activation, and up-projection to learn compact corrections to the semantic features, specifically targeting artifact-sensitive directions.
2. Textual Prompt Tuning: Instead of fixed text prompts (e.g., "a photo of a fake"), the model learns 16 shared context vectors for the "real" and "fake" classes. These learnable vectors are concatenated with the fixed class embeddings and passed through the frozen text encoder.

2.3 Encoder Truncation

A key methodological insight involves truncating the vision encoder. The authors observe that CLIP's final transformer blocks are optimized for semantic image-text alignment, which may suppress the low-level irregularities required for forensic detection. They propose removing the final transformer block (variant v2) before the adapter, allowing the model to access penultimate features that retain more high-frequency detail.

3. Key Contributions

1. Diff-Gen Dataset: A diffusion-based training corpus that demonstrates superior generalization to unseen generator families compared to the standard ProGAN corpus.
2. AdaptPrompt Framework: A dual-modality adaptation method combining visual adapters and learnable text prompts, achieving high performance with minimal parameter updates.
3. Encoder Truncation Finding: Empirical evidence that removing the final transformer block of CLIP consistently improves forensic performance, suggesting that semantic alignment layers discard forensic traces.
4. Comprehensive Benchmarking: Evaluation across 25 test sets, including 10 GANs, 9 diffusion models, 3 commercial tools, and face-swap benchmarks.

4. Experimental Results

The evaluation covers a benchmark of 25 test sets. Key findings include:

• Generalization: Models trained on Diff-Gen significantly outperform those trained on ProGAN when tested on diffusion models and commercial tools.
  • AdaptPrompt-v2 (trained on Diff-Gen) achieves a Mean Average Precision (mAP) of 98.60% and Accuracy of 92.72% across all families.
  • In contrast, ProGAN-trained baselines (e.g., Wang et al.) drop to near-chance accuracy (~52-57%) on commercial tools.
  • The generalization is asymmetric: Diff-Gen-trained models detect GAN artifacts well, whereas GAN-trained models fail on diffusion artifacts.
• Parameter Efficiency: AdaptPrompt matches or exceeds the performance of fully fine-tuned CLIP baselines while training roughly 700 times fewer parameters.
• Impact of Truncation: Removing the final transformer block (v2) consistently yields the best results. For example, AdaptPrompt-v2 improves mAP from 97.40% (v0) to 98.60% (v2).
• Robustness: The model shows robustness to Gaussian blur but degrades under heavy JPEG compression (quality 50), consistent with the reliance on broadband high-frequency residuals.
• Limitations:
  • Face-Swap: Performance drops significantly on FaceForensics++ (face-swap) datasets, as these are partial manipulations rather than fully generated images.
  • Person-Centric Imagery: Accuracy is lower for images containing people (80.5%) compared to those without (96.5%), likely due to the high quality of synthetic faces and photometric variations in real faces mimicking smoothing cues.
• Source Attribution: The framework extends to closed-set source attribution across 22 generators, achieving 81.4% accuracy and 96.49% family-level accuracy, with confusion primarily occurring between configurations of the same generator.

5. Significance and Claims

The paper claims that Diff-Gen and AdaptPrompt collectively solve the generalization problem in deepfake detection by addressing both the data distribution and the model adaptation strategy.

• Data Perspective: Training on diffusion artifacts (Diff-Gen) generalizes "backward" to GANs and "forward" to commercial tools, whereas the reverse is not true. This suggests that diffusion synthesis artifacts are a more fundamental representation of "synthetic" imagery than GAN artifacts.
• Modeling Perspective: The success of AdaptPrompt demonstrates that VLMs like CLIP contain rich forensic signals that are often discarded by standard fine-tuning or semantic alignment layers. By using parameter-efficient adaptation and truncating the final layers, these signals can be recovered without catastrophic forgetting or overfitting.
• Practical Impact: The approach offers a computationally efficient solution (low training cost, minimal parameter updates) that achieves state-of-the-art performance across diverse generator families, making it a viable candidate for real-world deployment against evolving generative AI threats.

The authors conclude that while their current model has limitations regarding face-swap manipulations and heavy compression, the framework provides a strong foundation for future work in face-aware adaptation and compression-robust training.