Technical Summary: ImageAttributionBench

Problem Statement

The rapid evolution of generative AI, particularly through auto-regressive and diffusion models, has enabled the creation of highly realistic synthetic images. While beneficial for industries like entertainment and medicine, this progress exacerbates challenges regarding misinformation, privacy violations, and fraud. Consequently, image attribution (identifying the specific generative model used to create an image) has become a critical task.

However, existing research is hindered by the limitations of current datasets. The paper identifies three primary constraints in existing benchmarks:

1. Limited Model Diversity: Most datasets include fewer than 15 generative models, failing to reflect the complexity of real-world scenarios.
2. Outdated Generation Methods: Existing datasets rely heavily on older GAN architectures and early diffusion models, lacking contemporary paradigms such as Diffusion Transformers (DiT) and Auto-Regressive (AR) models.
3. Insufficient Semantic Diversity: Limited semantic categories often lead to an entanglement between model-specific fingerprints and semantic features, compromising the generalization capabilities of attribution models.

Methodology: ImageAttributionBench Construction

To address these gaps, the authors introduce ImageAttributionBench, a comprehensive dataset and benchmark designed to evaluate the robustness and generalizability of attribution methods. The construction pipeline involves three key stages:

1. Real Data Collection: The authors curated 20,000 real-world images from diverse sources (CelebA-HQ, FFHQ, AFHQ, LSUN, ImageNet-1k, COCO) spanning 10 semantic classes (e.g., human faces, animal faces, bedrooms, churches). To ensure ethical integrity, they applied demographic balancing (using FairFace) and Personally Identifiable Information (PII) filtering.
2. Caption Generation: Using tailored prompts and the QwenVL-chat vision-language model, the authors generated accurate, descriptive captions for the real images. This step ensures semantic fidelity between the source image and the generated prompt.
3. Image Synthesis: The generated captions were used to synthesize images using 31 state-of-the-art generative models. This suite includes:
   • 14 Commercial Models: Including DALL·E 3, Midjourney (V5.2/V6.0), GPT-4o/1/1.5, Gemini variants, and others.
   • 17 Open-Source Models: Spanning Latent Diffusion Models (LDMs like Stable Diffusion variants), Diffusion Transformers (DiTs like FLUX.1, Hunyuan-DiT), and Auto-Regressive (AR) models (e.g., Infinity-2B, Janus-Pro).
   • The final dataset comprises approximately 640,000 images.

Experimental Setup and Evaluation

The authors evaluated a wide range of state-of-the-art (SOTA) attribution methods (categorized into spatial/frequency features, data augmentation, image patches, and disentangled representation learning) under two distinct settings:

1. Standard Balanced-Split Task: A standard setup where training and testing data are balanced across all models and semantic classes. This includes testing on clean images and images subjected to six levels of degradation (downsampling, JPEG compression, Gaussian blur).
2. Semantic-Split Task: A rigorous evaluation designed to test generalization. Models are trained on a single semantic class (e.g., "Cat") and tested on all other semantic classes (e.g., "Human Face," "Bedroom"). This setup isolates the model's ability to learn generator-specific fingerprints independent of semantic content.

Key Results

The experiments reveal significant limitations in current attribution technologies:

• Performance on Standard Task: On clean images with full semantic information, most methods achieve high accuracy (average 90.4%). However, performance drops sharply (to ~47.3%) when images are degraded, indicating a lack of robustness against common real-world distortions.
• Performance on Semantic-Split Task: When tested on semantically disjoint data, performance collapses. Even the best-performing methods on the standard task (e.g., FSD, Omni-DFA) see accuracy drop below 40%, representing a decline of over 50 percentage points.
• Correlation with Semantic Similarity: The authors observed a strong positive correlation between attribution accuracy and the semantic similarity between training and testing sets. Methods perform better when the test semantic class is similar to the training class (e.g., "Dog" vs. "Cat"), confirming that current models rely heavily on semantic cues rather than pure generator fingerprints.
• Method Comparison: Data augmentation-based methods (e.g., DNA-Det, RepMix) and patch-based methods (e.g., PatchForensics) showed relatively better generalization compared to others, though they still suffered significant performance drops in the semantic-split setting.

Significance and Claims

The paper claims that ImageAttributionBench serves as a rigorous benchmark that exposes the fragility of current attribution methods. Its primary contributions are:

1. A New Standard: It provides a dataset with unprecedented scale (640k images), model diversity (31 SOTA models including DiT and AR), and semantic balance.
2. Revealing Generalization Gaps: By demonstrating that current methods fail when semantic content changes or images are degraded, the benchmark highlights the urgent need for techniques that can disentangle generator fingerprints from semantic features.
3. Future Direction: The authors posit that the field must move beyond relying on semantic correlations to develop attribution models that are robust, semantically generalizable, and applicable in real-world scenarios where images are often degraded and semantically diverse.

The work concludes that while existing methods work well in controlled, semantically rich environments, they are currently insufficient for the complex, dynamic landscape of real-world AI-generated content. ImageAttributionBench is intended to facilitate the development of more robust solutions.