Mitigating Hierarchical Shortcut Reliance in Frozen CLIP for Cross-Generator AI-Generated Image Detection

Technical Summary: Mitigating Hierarchical Shortcut Reliance in Frozen CLIP for Cross-Generator AI-Generated Image Detection

1. Problem Statement

The detection of AI-generated images (AIGI) faces a critical challenge in cross-generator generalization. While detectors often perform well on generators seen during training, their reliability degrades significantly when tested on unseen generators. Existing explanations for this failure often focus on limited training coverage, specific generator artifacts, or unstable low-level cues. However, this paper argues that a primary cause is hierarchical shortcut reliance: a phenomenon where a detector over-relies on a small set of discriminative but weakly transferable layers within a Vision Transformer (ViT), while underutilizing complementary evidence from other layers.

The authors posit that different generator families (e.g., GANs vs. Diffusion models) favor different representation levels and frequency patterns. Consequently, a detector trained to exploit specific layer-frequency correlations for one generator fails to generalize when those correlations shift in unseen generators.

2. Methodology

The proposed solution is a layer-wise debiasing framework built upon a frozen CLIP ViT-L/14 backbone. The core philosophy is to intervene at specific hierarchical levels to reduce over-reliance on source-specific shortcuts and encourage the use of complementary evidence. The framework operates through three primary intervention modules and two auxiliary training branches:

A. Hierarchical Interventions (Applied to the "Original" Branch)
The framework assigns specific debiasing tasks to different layers of the frozen backbone based on empirical observations of layer sensitivity:

1. Spatial Consistency Intervention (SCI) at Low Layers:
   • Goal: Stabilize fine-grained spatial cues which are often unstable in low layers.
   • Mechanism: Computes a global consistency score between patch tokens and the CLS token, and a local inconsistency descriptor by aggregating neighborhood patches. A lightweight residual adapter modulates patch tokens to emphasize regions with weak local or global consistency.
2. Frequency Prior Intervention (FPI) at Middle Layers:
   • Goal: Complement spatial evidence with frequency-aware cues, as middle layers often carry transferable artifact information sensitive to frequency shifts.
   • Mechanism: Restores patch tokens to a 2D grid, applies Discrete Wavelet Transform (DWT) to extract sub-band responses (LL, LH, HL, HH), and fuses these frequency responses with spatial artifact responses before writing them back to the token sequence.
3. High-Layer Semantic Debiasing Module (HSDM) at High Layers:
   • Goal: Reduce reliance on source-specific semantics and content patterns encoded in high-level features.
   • Mechanism:
     • Semantic Alignment (SAS): Uses text embeddings for "Camera" (real) and "deepfake" (forged) as anchors to regularize high-layer image features, ensuring they align with broad real/forged semantics rather than generator-specific patterns.
     • Debiasing Training (DTR): Utilizes auxiliary branches to perturb the input during training. A Shuffle branch permutes patch tokens to reduce position shortcuts, and a Mix branch replaces tokens with those from same-class donor samples to weaken sample-specific content correlations.

B. Training Augmentation and Optimization

• Forgery-Only Latent Augmentation (FLA): During training, latent space perturbations are applied only to forged samples ($y=1$) to expand the feature support of the forged class without blurring the real/forged boundary.
• Optimization Objective: The total loss ($L_{total}$) combines:
  • Classification Loss ($L_{ce}$): Applied to the Original branch (and semantic logits from SAS).
  • Feature Consistency Loss ($L_{con}$): Uses InfoNCE to ensure the Original branch representation remains consistent with the perturbed Shuffle and Mix branches.
  • Prediction Alignment Loss ($L_{align}$): Minimizes Jensen–Shannon divergence between the probability distributions of the Original branch and the auxiliary branches.

C. Inference
At inference time, only the Original branch is retained. The Shuffle and Mix branches, along with the FLA augmentation, are discarded, ensuring the framework adds no computational overhead during deployment.

3. Key Contributions

1. Conceptual Framework: Introduction of hierarchical shortcut reliance as a layer-wise perspective for analyzing cross-generator degradation in ViT-based detectors.
2. Empirical Evidence: Provision of layer-wise probing and frequency sensitivity analysis demonstrating that different generators favor different representation levels and layer-frequency patterns (e.g., GANs vs. Diffusion models showing distinct layer peaks and frequency sensitivities).
3. Proposed Framework: A diagnosis-guided, frozen-CLIP layer-wise debiasing framework that strategically assigns spatial, frequency, and semantic interventions to low, middle, and high layers, respectively.
4. Experimental Validation: Demonstration of competitive cross-generator detection performance on standard benchmarks, supported by robustness tests and ablation studies confirming the value of signal-level, layer-specific interventions.

4. Experimental Results

The framework was evaluated on two major benchmarks: UniversalFakeDetect and GenImage.

• UniversalFakeDetect: Trained on ProGAN and tested on 19 unseen generators (including StyleGAN, BigGAN, CycleGAN, and various Diffusion models).
  • Achieved an average accuracy of 96.03% and average precision of 99.45%.
  • Outperformed or matched state-of-the-art methods (e.g., UniFD, FatFormer, NPR) across the cross-model setting.
• GenImage: Trained on SDv1.4 and tested on other diffusion models (Midjourney, SDv1.5, ADM, etc.) and BigGAN.
  • Achieved an average accuracy of 92.32%.
• Robustness: Under common post-processing operations (JPEG compression, Gaussian blur, noise, downsampling), the method maintained high Average Precision (AP) across varying intensities, outperforming CNN-based, gradient-based, and frequency-domain baselines.
• Ablation Studies:
  • Removing any module (SCI, FPI, SAS, DTR, FLA) resulted in performance degradation, with the removal of FPI causing the largest drop (from 96.03% to 89.15%), highlighting the critical role of frequency priors.
  • Synergy analysis confirmed that combining SCI, FPI, and HSDM yields significantly better results than using them individually.
  • Position ablation confirmed that SCI performs best in shallow layers, FPI in middle-to-high layers, and DTR at specific intermediate layers.

5. Significance and Claims

The paper claims that cross-generator detection benefits from a signal-level analysis of hierarchical representations. By explicitly controlling how evidence is used across layers—rather than treating the backbone as a monolithic feature extractor—the proposed framework mitigates the reliance on source-specific shortcuts.

The authors maintain a modest scope regarding their claims:

• The conclusions are limited to the evaluated benchmarks (UniversalFakeDetect, GenImage) and the specific frozen CLIP ViT-L/14 setting.
• The study focuses on image-level binary classification.
• While the results support the value of layer-wise interventions, the paper acknowledges that broader backbone validation and stronger causal evidence for hierarchical shortcut reliance remain future work.
• The framework is designed to be computationally efficient at inference, fitting on a single GPU during training with no additional cost at test time.