Adaptive Forensic Feature Refinement via Intrinsic Importance Perception

1. Problem Statement

The rapid advancement of generative AI (e.g., Diffusion models, GANs) and multimodal editing tools has made Synthetic Image Detection (SID) increasingly difficult. The core challenge lies in cross-distribution generalization: detection models trained on specific generators often fail when faced with unknown generation sources or unseen post-processing techniques.

While Visual Foundation Models (VFMs) (e.g., CLIP) offer rich, pre-trained visual priors that could improve generalization, existing VFM-based SID methods suffer from two main limitations:

1. Coarse-grained Adaptation: They typically use the final layer representations or indiscriminately fuse multi-layer features. This ignores the fact that different layers encode different types of information; deep layers often compress fine-grained forgery cues into high-level semantics, while shallow layers are noisy.
2. Destructive Fine-tuning: Aggressive fine-tuning of VFMs to adapt to specific forgery tasks can damage the pre-trained structural priors that enable open-set generalization, leading to "representation drift" and catastrophic forgetting.

The paper addresses the tension between task specificity (detecting forgeries) and preserving transferability (maintaining the VFM's generalization capability).

2. Methodology: The I2P Framework

The authors propose I2P (Intrinsic Importance Perception), a two-stage framework designed to adapt VFMs for SID without destroying their pre-trained structure.

Stage 1: Critical Layer Identification (CLI)

• Motivation: Empirical analysis (using T-SNE, spectral structure, and linear separability tests) reveals that intermediate layers of VFMs contain the most discriminative features for SID. Shallow layers are too noisy, and deep layers lose fine-grained artifact details due to semantic compression.
• Mechanism:
  • Instead of manually selecting a layer, I2P uses a lightweight, learnable gating mechanism to score the discriminative contribution of each layer's representation (specifically CLS tokens).
  • A scoring function assigns a weight $\pi_\ell$ to each layer. The layer with the highest weight is identified as the Critical Layer ($\ell^*$).
  • Pruning: Once $\ell^*$ is identified, representations from deeper layers are pruned, and the model relies solely on the features from this critical intermediate layer for the final classification. This avoids information dilution from irrelevant layers.

Stage 2: Controlled Knowledge Injection (CKI)

• Motivation: To adapt the VFM to the SID task without disrupting the pre-trained "flat basin" of parameters that supports generalization.
• Mechanism:
  • Instead of updating all parameters, CKI identifies a low-sensitivity parameter subspace.
  • It approximates the Hessian matrix (curvature of the loss landscape) using the second moment of activations to estimate parameter importance.
  • Parameters with low importance scores (low sensitivity to the pre-trained structure) are selected for updating.
  • Update Rule: Only the selected low-sensitivity subset of parameters and the classification head are fine-tuned. High-sensitivity parameters remain frozen. This constrains the optimization to directions that minimize disturbance to the pre-trained visual priors.

3. Key Contributions

1. Systematic Analysis of Hierarchical Representations: The authors demonstrate that for SID, the optimal representational hierarchy is not the deepest semantic layer but rather an intermediate layer where fine-grained forgery cues are preserved before being compressed.
2. Unified Adaptation Framework (I2P): They propose a novel framework that unifies Critical Layer Identification (finding the best feature source) and Controlled Knowledge Injection (finding the safest way to update parameters).
3. Robust Generalization: By combining CLI and CKI, the method achieves high task specificity while preserving the VFM's intrinsic transferability, effectively solving the "task-specific tension."
4. Extensive Validation: The approach is validated across multiple cross-domain benchmarks, showing superior performance over both specialized detectors and other VFM-based methods.

4. Experimental Results

The paper evaluates I2P on three settings using the GenImage, ForenSynths, Chameleon, and COSPY benchmarks.

• Setting 1 (GenImage): Trained on SDv1.4, tested on 8 diverse generators (including Midjourney, BigGAN, GLIDE).
  • Result: I2P achieved an average Accuracy (ACC) of 98.22% and Average Precision (AP) of 99.72%, outperforming the second-best method (ForgeLens) by a significant margin.
• Setting 2 (ForenSynths/UniversalFake): Trained on ProGAN, tested on 18 generators (GANs and Diffusion models).
  • Result: I2P achieved 96.59% ACC, surpassing the runner-up (FerretNet) by 1.35%. It showed particular stability when transferring from GAN training to Diffusion testing, where other methods degraded significantly.
• Setting 3 (Chameleon & COSPY): Tested on real-world distributions and new 2024+ generators (e.g., FLUX.1, SD-3).
  • Result: I2P achieved the best performance on both benchmarks, demonstrating strong generalization to unseen, real-world distribution shifts.
• Ablation Studies:
  • Removing either CLI or CKI resulted in performance drops, confirming their complementary nature.
  • The method works effectively across different VFM backbones (CLIP, SigLIP, DINOv3), proving its architecture-agnostic nature.
  • Using a very small update rate ($\eta \approx 0.0005$) for the low-sensitivity parameters yielded optimal results, balancing adaptation and stability.

5. Significance

This paper makes a significant contribution to the field of AI-generated content detection by shifting the paradigm from "training specialized detectors from scratch" to "strategically adapting foundation models."

• Efficiency: It requires significantly fewer training samples (only 1,600 images per set in experiments) compared to specialized methods that need massive datasets.
• Stability: It addresses the critical issue of "catastrophic forgetting" in VFMs, ensuring that the model remains robust against unseen generators.
• Insight: It provides a theoretical and empirical basis for understanding where (intermediate layers) and how (low-sensitivity subspaces) to adapt foundation models for forensic tasks, offering a blueprint for future cross-distribution detection systems.