TAP into the Patch Tokens: Leveraging Vision Foundation Model Features for AI-Generated Image Detection

1. Problem Statement

The rapid advancement of AI image generation (e.g., Stable Diffusion, Midjourney, Flux) has created a significant challenge in distinguishing between real and synthetic images. Existing detection methods face two primary hurdles:

• Generalization: Detectors trained on specific generators often fail when encountering unseen models (out-of-distribution generalization).
• Feature Utilization: Most state-of-the-art (SOTA) detectors rely on Vision Transformers (ViTs) pretrained on large datasets (like CLIP). However, these methods typically discard the patch tokens (which encode local visual information) and rely solely on the cls token (a global representation). This is suboptimal for AI-generated image (AIGI) detection because generative artifacts and inpainting manipulations often manifest as localized inconsistencies rather than global patterns.
• Outdated Backbones: Many current methods still use the original CLIP-ViT, ignoring newer Vision Foundation Models (VFMs) that offer architectural improvements, larger scales, and diverse pretraining objectives.

2. Methodology

The authors propose a two-stage framework that combines modern VFMs with a redesigned classifier head.

A. Semantic Feature Extraction

Instead of using the original CLIP encoder, the method utilizes frozen Vision Foundation Models (VFMs) as the backbone. The encoder processes the input image $x$ to produce a feature set $\mathbf{F}$ containing:

• The global cls token.
• The sequence of patch tokens ($\mathbf{f}_1, \dots, \mathbf{f}_{N-1}$), which encode local spatial details.

B. Tunable Attention Pooling (TAP)

To effectively utilize the full output sequence, the authors introduce Tunable Attention Pooling (TAP) to replace the standard cls-token-only approach.

1. Learnable Query: A learnable query vector $\mathbf{q}$ is initialized.
2. Multi-Head Cross-Attention: The query attends to the normalized feature set $\mathbf{F}$ (keys and values) via a Multi-Head Attention (MHA) mechanism. This allows the model to selectively focus on spatial regions containing generative artifacts while preserving global semantics.
   $$\mathbf{z} = \text{MHA}(\mathbf{q}, \text{LN}(\mathbf{F}))$$
3. Refinement: The pooled representation $\mathbf{z}$ is passed through a LayerNorm and a two-layer MLP with a residual connection to produce a refined vector $\mathbf{z}'$.
4. Classification: The refined vector is projected to a global-pooled token ($\mathbf{g}_{pl}$). This $\mathbf{g}_{pl}$ is concatenated with the original cls token and fed into a linear classifier for binary classification (Real vs. Generated/Inpainted).

Key Design Choice: The VFM encoder remains frozen; only the TAP layer and the final linear classifier are trainable. This minimizes parameters while maximizing feature utilization.

3. Key Contributions

1. Comprehensive VFM Benchmark: The authors conducted the first extensive benchmark of modern VFMs (including DINOv2/3, SIGLIP2, SAM2, and Perception Encoder) for AIGI detection. They evaluated various model scales, pretraining objectives (contrastive, self-supervised), and input resolutions.
2. Tunable Attention Pooling (TAP): They proposed a lightweight architectural change that aggregates both cls and patch tokens. This aligns the feature space better with the AIGI task by capturing local artifacts without discarding global context.
3. New State-of-the-Art (SOTA): The combination of the Perception Encoder (PE-Core) and TAP established new SOTA performance on multiple challenging benchmarks, significantly outperforming methods that rely solely on CLIP or cls tokens.

4. Experimental Results

The method was evaluated on three major datasets: GenImage, Chameleon, and OpenSDI (which includes both fully generated and inpainted images).

• Benchmarking VFMs:

  • Perception Encoder (PE-Core) outperformed all other families, including CLIP, DINOv2, and SIGLIP2.
  • PE-Core-ViT-G (Large) achieved the highest baseline performance, surpassing the original CLIP by a wide margin.
  • Resolution Impact: For fixed-resolution models, higher resolutions generally improved performance. However, for variable-resolution models (like DINOv2), simply increasing resolution did not guarantee better results and sometimes degraded performance.

• Impact of TAP:

  • Integrating TAP yielded substantial gains across all tested models.
  • GenImage: PE-Core-ViT-G + TAP achieved 97.16% mean accuracy, beating the previous SOTA (OMAT) by ~2.5% and AIDE by ~10%.
  • Chameleon: The method achieved 95.64% accuracy, surpassing the previous best (OMAT) by >29%.
  • OpenSDI (Inpainting): TAP proved particularly effective for detecting inpainted images (where artifacts are localized). PE-Core-ViT-G + TAP improved mean accuracy by ~3% over the cls-only baseline and outperformed DualSight by >4% in mean F1 score.

• Generalization: The approach demonstrated strong zero-shot generalization to unseen generators (SD2.1, SD3, SDXL, Flux, Midjourney) when trained only on SD1.5.

5. Significance

• Paradigm Shift: The paper challenges the long-standing reliance on the cls token in AIGI detection, demonstrating that patch tokens contain critical forensic evidence that was previously ignored.
• Efficiency: The TAP mechanism adds negligible computational overhead (only a few trainable parameters) while delivering massive performance boosts, making it a highly efficient upgrade for existing detection pipelines.
• Foundation Model Adoption: It establishes that newer, larger VFMs (specifically Perception Encoder) are superior to CLIP for forensic tasks, urging the community to update their feature extraction backbones.
• Robustness: The method sets a new standard for detecting both full-image synthesis and subtle inpainting manipulations, addressing a critical need for content authenticity in the era of generative AI.