HiMix: Hierarchical Artifact-aware Mixup for Generalized Synthetic Image Detection

1. Problem Statement

The rapid advancement of generative models (GANs and Diffusion Models) has led to the creation of highly realistic synthetic images, posing severe challenges for Synthetic Image Detection (SID).

• Generalization Gap: Existing detectors are typically trained on limited, biased datasets. Consequently, they suffer from source overfitting, where models learn generator-specific cues rather than universal forgery patterns.
• Low-Confidence Regions: When tested on unseen generators, synthetic images often fall into "low-confidence regions" of the decision space, leading to dispersed logits and poor classification accuracy.
• Semantic Bias: Methods relying on pre-trained models (e.g., CLIP) often capture high-level semantic content (e.g., object categories) rather than intrinsic forgery artifacts, causing detectors to learn content-biased patterns instead of generalizable forgery cues.

2. Methodology: HiMix Framework

The authors propose HiMix, a unified framework designed to enhance generalization from both data distribution and representation perspectives. The architecture consists of two core modules:

A. Mixup-driven Distributional Augmentation (MDA)

This module addresses the data distribution issue by expanding the training space.

• Mechanism: It generates continuous transitional samples between real and fake images using pixel-wise Mixup:
  $$X_{mix} = \lambda X_{fake} + (1 - \lambda) X_{real}$$
  where $\lambda$ is sampled from a Beta distribution. Crucially, these mixed samples are labeled as "fake" regardless of the mixing ratio.
• Purpose:
  • Coverage: It forces the model to learn the boundary between real and fake, effectively filling the "low-confidence regions" where unseen forgeries usually reside.
  • Semantic Reduction: The pixel-wise interpolation smooths out high-level semantics (content), forcing the model to rely on low-level, generator-invariant artifacts (e.g., noise patterns, frequency inconsistencies) for detection.

B. Hierarchical Artifact-aware Representation (HAR)

This module addresses the representation issue by extracting robust forgery cues using a pre-trained CLIP backbone enhanced with specific architectural designs.

• Backbone & Adaptation: Uses a frozen CLIP ViT encoder with LoRA (Low-Rank Adaptation) applied to the QKV projections. This allows the model to adapt to forgery detection tasks with minimal parameters while preserving pre-trained knowledge.
• Hierarchical Region Pooling (HiRP):
  • Instead of relying solely on the global CLS token, HiRP partitions patch tokens into local windows at multiple scales ($s_r$).
  • It aggregates statistics via intra-window average pooling and inter-window max pooling to capture spatially consistent, subtle artifacts.
  • Learnable weights ($\beta_r$) fuse these multi-scale regional tokens.
• Cross-Layer & Cross-Granularity Fusion:
  • Cross-Layer: Aggregates features from different Transformer layers (both CLS and Region tokens) to capture artifacts emerging at different semantic depths.
  • Cross-Granularity Fusion (CGF): A lightweight MLP dynamically fuses the global structural features (CLS) and local artifact features (Region) to produce a final, generator-invariant representation.

3. Key Contributions

1. Unified Framework: Proposes HiMix, which jointly optimizes data augmentation (MDA) and feature representation (HAR) to tackle the generalization problem in SID.
2. Distributional Augmentation Strategy: Introduces a Mixup-driven approach that enriches low-confidence regions and suppresses high-level semantic dominance, enabling the model to focus on intrinsic forgery traces.
3. Hierarchical Artifact Learning: Designs a novel representation module (HAR) that integrates multi-scale, cross-layer, and cross-granularity cues to extract robust forgery patterns.
4. State-of-the-Art Performance: Demonstrates superior cross-source generalization across diverse benchmarks, significantly outperforming existing methods on unseen generators.

4. Experimental Results

The paper evaluates HiMix on multiple benchmarks, including GenImage, Ojha (Diffusion), ForenSynths (GANs), and TwinSynths-GAN.

• General Performance: On the GenImage dataset (8 diverse generators), HiMix achieves 97.6% Average Accuracy and 99.7% Average Precision, outperforming strong baselines like SAFE, CoD, and AIDE.
• Diffusion Generalization: When trained only on Stable Diffusion v1.4, HiMix generalizes effectively to other diffusion models (LDM, Glide, ADM), achieving 98.3% Acc and 99.9% AP.
• Challenging Scenarios: On the TwinSynths-GAN dataset (high-fidelity, content-aligned forgeries), HiMix achieves 90.6% AP, surpassing the previous SOTA (SFLD) by a significant margin (12.29% improvement).
• Robustness: The model maintains stable performance under JPEG compression and Gaussian blur, outperforming AIDE across all distortion levels.
• Data Efficiency: HiMix achieves 96.71% accuracy using only 1% of the training data, demonstrating high efficiency.
• Logit Distribution: Visualizations show that HiMix produces compact within-class distributions and well-separated between-class logits, even for unseen generators, whereas other methods show dispersed and overlapping logits.

5. Significance

• Solving the "Unseen Generator" Problem: HiMix directly addresses the critical limitation of current SID systems failing on new, unseen generative models by explicitly training on the decision boundary (via Mixup) and learning invariant artifacts.
• Practical Deployment: The method's robustness to image degradations (compression, blur) and high data efficiency make it suitable for real-world deployment where training data may be scarce or post-processing is common.
• Architectural Insight: The work highlights the importance of moving beyond global semantic features to hierarchical, multi-scale artifact analysis, providing a new blueprint for designing robust forensic detectors.

In conclusion, HiMix represents a significant step forward in making synthetic image detection reliable and generalizable, shifting the focus from memorizing specific generator signatures to learning fundamental, universal forgery artifacts.