Diversity Matters: Dataset Diversification and Dual-Branch Network for Generalized AI-Generated Image Detection

1. Problem Statement

The rapid proliferation of AI-generated images (AIGIs) via Generative Adversarial Networks (GANs) and diffusion models poses significant threats to digital security, misinformation control, and biometric systems. While existing detection methods exist, they suffer from two critical limitations:

• Lack of Generalization: Models trained on specific generative paradigms (e.g., GANs) often fail to detect images from unseen models (e.g., diffusion models) and vice versa.
• Data Redundancy and Overfitting: Current datasets often contain highly redundant samples with similar artifacts. Training on these dense clusters causes detectors to memorize specific patterns rather than learning generalizable synthetic cues, leading to poor performance on unseen data.
• Feature Incompleteness: Most approaches rely solely on pixel-domain (spatial) or frequency-domain information, failing to jointly model semantic inconsistencies and structural/spectral artifacts.

2. Methodology

The authors propose a novel framework named Diversity Matters, which addresses the above issues through two core components: Dataset Diversification and a Dual-Branch Network.

A. Dataset Diversification (Feature-Space Filtering)

To combat data redundancy and overfitting, the authors introduce a four-stage deduplication pipeline using a pre-trained CLIP encoder (ViT-L/14):

1. Feature Extraction: Visual features are extracted from the input dataset using a frozen CLIP encoder.
2. Global Deduplication: Cosine similarity is computed between feature vectors. Samples exceeding a similarity threshold $T$ are discarded to remove globally similar pairs.
3. Class-Based Refinement: The remaining samples are split into real and fake classes. If the real class is larger, further deduplication is applied to balance the classes.
4. Final Balancing: The filtered real and fake samples are merged to create a diverse, representative, and balanced training set.

• Goal: To ensure the training set captures diverse characteristics across different generative models rather than reinforcing narrow decision boundaries.

B. Dual-Branch Network Architecture

The detection model employs a dual-branch architecture to capture complementary cues:

1. Pixel-Domain Branch: Uses a frozen CLIP image encoder to extract semantic and appearance-based representations ($R_p$) from the raw input image. This branch captures high-level semantic inconsistencies.
2. Frequency-Domain Branch:
   • The input image is transformed into the frequency domain using the Discrete Fourier Transform (DFT).
   • The magnitude spectrum is fed into a fine-tuned CLIP encoder to extract spectral features ($R_s$). This branch captures structural irregularities, periodic patterns, and high-frequency artifacts often invisible in the spatial domain.
3. Fusion and Classification: Class tokens from both branches are concatenated to form a fused feature representation. This is passed through a trainable detection head (three linear layers with LeakyReLU and dropout) to predict the real/fake probability.

C. Loss Function

The model is optimized using a hybrid objective:
$$L = L_{CE}(y, \hat{y}) + \lambda L_{SC}(y, f)$$

• $L_{CE}$ (Cross-Entropy): Ensures correct binary classification.
• $L_{SC}$ (Supervised Contrastive Loss): Regularizes the feature space by pulling representations of the same class together and pushing different classes apart, ensuring the two branches learn complementary and discriminative features.

3. Key Contributions

1. Insight on Generalization: Demonstrated that training on a single generative model (e.g., ProGAN) is insufficient for generalizing to unseen diffusion models, and that large volumes of unfiltered data lead to overfitting.
2. Dataset Diversification Strategy: Introduced a CLIP-based feature-space filtering mechanism to remove redundant samples, creating a sparse, diverse training set that improves cross-model generalization.
3. Dual-Branch Architecture: Proposed a novel network that jointly leverages pixel-domain (semantic) and frequency-domain (structural) features, significantly outperforming single-branch baselines.
4. Comprehensive Evaluation: Validated the approach across multiple benchmark datasets (GAN and diffusion-based) and under various perturbation conditions.

4. Experimental Results

The method was evaluated on benchmark datasets including MGD, CNNDF, GANDF, GENI, DIRE, UDF, and MNW.

• Generalization Performance:
  • The proposed method achieved an average AUC of 98.1% and AP of 98.3% across all datasets, outperforming State-of-the-Art (SoTA) methods like C2PClip, ForL, and RINE.
  • On the challenging MNW dataset (containing unseen diffusion models), the proposed method maintained an ACC of 85.1%, whereas many SoTA methods degraded to near-random guessing (e.g., C2PClip dropped to 43.4% ACC).
• Ablation Studies:
  • Data Diversity: Using filtered diverse data (19K samples) yielded higher performance (AP 98.1%) than using unfiltered data (180K samples, AP 97.5%), proving that diversity is more critical than quantity.
  • Dual-Branch: Combining pixel and frequency features ($f_p + f_s$) consistently outperformed using either feature alone across all backbone networks (ViT, ResNet, MobileNet).
• Robustness: The model demonstrated superior resilience against Gaussian blurring and JPEG compression compared to single-branch baselines.
• Backbone Efficiency: Foundation models (ViT-L/14, DINOv2) significantly outperformed traditional CNNs (ResNet, VGG), with ViT-L/14 achieving the best results.

5. Significance

The paper "Diversity Matters" makes a significant contribution to the field of AI-generated content detection by shifting the focus from simply increasing data volume to optimizing data quality and diversity.

• Practical Impact: It offers a robust solution for detecting "zero-day" generative models, addressing the urgent need to combat misinformation and fraud in real-world scenarios.
• Theoretical Insight: It establishes that feature-domain complementarity (pixel + frequency) and dataset deduplication are essential for building detectors that generalize across the rapidly evolving landscape of synthetic media.
• Efficiency: The approach demonstrates that a smaller, carefully curated dataset can outperform massive, redundant datasets, suggesting a more efficient path for training future detectors.