SPARK-IL: Spectral Retrieval-Augmented RAG for Knowledge-driven Deepfake Detection via Incremental Learning

1. Problem Statement

The rapid advancement of generative AI (GANs, Diffusion models, Face-swapping) has made distinguishing AI-generated images from real ones increasingly difficult. Current deepfake detection methods face two critical limitations:

• Poor Generalization: Detectors trained on specific generators often fail to generalize to unseen models because they rely on generator-specific spatial artifacts (e.g., texture inconsistencies) that vary significantly across synthesis pipelines.
• Static Nature: Traditional parametric classifiers cannot adapt to new generators without retraining from scratch, leading to "catastrophic forgetting" of previously learned knowledge.

The authors observe that while pixel-level artifacts vary, frequency-domain statistics exhibit greater consistency across different generator families. Synthetic images tend to deviate from natural spectral distributions in characteristic ways that persist even when spatial cues are suppressed.

2. Methodology: SPARK-IL Framework

SPARK-IL (Spectral Retrieval-Augmented RAG for Knowledge-driven Deepfake Detection via Incremental Learning) is a framework that combines dual-path spectral analysis with retrieval-augmented inference and incremental learning.

A. Dual-Path Spectral Architecture

The model processes images through two parallel paths to capture both low-level and high-level artifacts:

1. Semantic Path: Uses a partially frozen ViT-L/14 encoder (blocks 0–21 frozen, 22–23 fine-tuned) to extract high-level semantic representations ($h_{vit}$).
2. Pixel Path: Projects raw RGB pixels directly to the same embedding dimension to capture low-level artifacts ($h_{rgb}$).

Multi-Band KAN-FFT Processing:
Both paths undergo a Multi-Band KAN-FFT block:

• FFT Transformation: Features are transformed into the frequency domain via 1D Fast Fourier Transform (FFT), and the log-magnitude spectrum is computed.
• Band Partitioning: The spectrum is split into four disjoint frequency bands (Low to High frequency).
• Kolmogorov-Arnold Networks (KAN): Each band is processed by a KAN module with a Mixture-of-Experts (MoE) mechanism. This allows the model to learn non-linear, band-specific transformations, capturing distinct artifacts (e.g., structural inconsistencies in low frequencies vs. noise patterns in high frequencies).

Fusion:
The spectral embeddings from both paths are fused using Multi-Head Cross-Attention, where the RGB spectral embedding acts as the query and the ViT spectral embedding provides keys/values. Residual connections ensure the preservation of original information.

B. Retrieval-Augmented Inference (RAG)

Instead of relying solely on a fixed parametric classifier for prediction:

• Training: The fused embeddings are stored in a Milvus vector database along with their ground-truth labels and generator IDs.
• Inference: For a test image, the system retrieves the $K$ nearest neighbors (based on cosine similarity) from the database.
• Prediction: The final label is determined by majority voting over the retrieved labels. This non-parametric approach allows the model to adapt to local query distributions without updating weights.

C. Incremental Learning

To handle new generators without catastrophic forgetting, SPARK-IL employs:

• Elastic Weight Consolidation (EWC): Penalizes changes to parameters critical for previously learned tasks.
• Knowledge Distillation: Minimizes the distance between current and previous embeddings/logits.
• Experience Replay: Maintains a buffer of representative samples from previous techniques.
• Database Expansion: New embeddings are simply added to the Milvus database, expanding the knowledge base without retraining the entire model.

3. Key Contributions

1. Dual-Path Multi-Band Spectral Architecture: A novel design processing both pixel-level and feature-level representations via parallel FFT blocks, utilizing KANs for adaptive, band-specific non-linear transformations.
2. Retrieval-Augmented Classification: A shift from fixed decision boundaries to a similarity-based approach using cosine similarity search and majority voting over spectral signatures in a vector database.
3. Incremental Adaptation: A mechanism combining EWC and continuous embedding indexing to adapt to new generators while preserving performance on previously seen techniques.
4. State-of-the-Art Performance: Achieved 94.60% mean accuracy on the UniversalFakeDetect benchmark across 19 diverse generative models.

4. Experimental Results

The method was evaluated on the UniversalFakeDetect benchmark, which includes 19 models (GANs, Diffusion, Face-swapping, etc.). The training was constrained to ProGAN data to rigorously test cross-generator generalization.

• Overall Accuracy: SPARK-IL achieved 94.60% mean accuracy, outperforming the previous state-of-the-art (REVEAL) by 0.70% and a standard fine-tuned ViT baseline by 13.22%.
• Generalization:
  • GANs: Exceeded 99% accuracy on ProGAN, BigGAN, and GauGAN.
  • Diffusion Models: Strong performance on LDM (99.40%) and DALL·E (98.90%).
  • Challenging Cases: Improved performance on difficult categories like SAN (65.20%) and DeepFakes (94.30%).
• Retrieval Impact: Increasing the number of retrieved neighbors ($K$) improved performance, stabilizing around $K=10$ to $K=20$. $K=5$ yielded 94.10%, while $K=20$ reached the peak 94.60%.
• Efficiency: With 315M parameters and 60.33 GFLOPs, SPARK-IL is computationally efficient, using fewer parameters than larger competitors like FatFormer (493M) while achieving higher accuracy.

5. Significance and Conclusion

SPARK-IL represents a paradigm shift in deepfake detection by moving away from rigid, parametric classifiers toward knowledge-driven, retrieval-based systems.

• Robustness: By leveraging frequency-domain signatures, which are more consistent across generators than spatial artifacts, the model achieves superior cross-generator generalization.
• Scalability: The incremental learning approach allows the system to evolve as new generative models emerge without the computational cost of full retraining or the risk of forgetting old data.
• Interpretability: The use of nearest-neighbor retrieval provides evidence-based predictions grounded in specific, labeled examples rather than abstract decision boundaries.

The authors conclude that while global FFT has limitations for spatially localized artifacts, the combination of spectral analysis, KANs, and retrieval-augmented inference offers a highly effective and adaptable solution for the evolving landscape of AI-generated content.