Technical Summary: GenSyn10

Problem Statement

The rapid advancement of generative AI has outpaced the development of reliable detection mechanisms. A central challenge in AI-generated image detection is the out-of-distribution (OOD) generator shift problem: detectors trained on specific generators (e.g., Stable Diffusion v1.4) often fail to generalize to outputs from unseen, architecturally distinct generators. Existing benchmarks, such as CIFAKE, rely on outdated single-generator datasets or lack standardized generation protocols, preventing controlled evaluation of cross-generator generalization. Furthermore, current benchmarks often fail to include images from the latest generation of open-source models, leaving a gap in understanding how detectors perform against modern architectural paradigms like Rectified Flow Transformers, Mixture-of-Experts (MoE), and Multimodal Diffusion Transformers.

Methodology

Dataset Construction: GenSyn10

The authors introduce GenSyn10, a synthetic image dataset aligned with the CIFAR-10 structure (10 classes, 32×32 resolution, 60,000 images total).

• Generators: Images are generated using three state-of-the-art, open-source models representing distinct architectural families:
  1. FLUX.2-dev: Utilizes a Rectified Flow Transformer.
  2. HunyuanImage-3.0: Employs an autoregressive Mixture-of-Experts (MoE) architecture.
  3. Qwen-Image-2512: Combines a Multimodal Diffusion Transformer (MMDiT) with a frozen Qwen2.5-VL vision-language model.
• Prompt Engineering: A combinatorial, template-based prompt engine generates over 100,000 unique prompt combinations per class. Prompts are structured across six slots (Instance, Scene, View, Lighting, Action, Opener) and include "Clean" and "Hard" difficulty tiers (introducing occlusions, motion blur, etc.).
• Standardization: All images are generated at 512×512 resolution and downsampled to 32×32 using Lanczos resampling to ensure consistency.
• OOD Protocol: To evaluate generalization, Stable Diffusion 3.5 Large is used as a held-out generator. It is excluded from the training set but used to generate a separate test set for OOD evaluation, ensuring that performance drops are attributable to architectural shifts rather than prompt or resolution variations.

Benchmark Protocol

The paper evaluates 17 classification architectures across seven model families (including VGG, ResNet, DenseNet, EfficientNet, ViT, Swin, and ConvNeXt) using a four-stage evaluation protocol:

1. Baseline (S1): Evaluation of CIFAR-10-initialized models on the CIFAR-10 test set.
2. Zero-Shot Transfer (S2): Evaluation of the same models on the GenSyn10 test set without fine-tuning.
3. Retention (S3): Fine-tuning on GenSyn10 followed by evaluation on the CIFAR-10 test set to measure catastrophic forgetting.
4. Adaptation (S4): Fine-tuning on GenSyn10 followed by evaluation on the GenSyn10 test set.

Additionally, a binary real-vs-synthetic classification task is performed. Models are trained to distinguish real CIFAR-10 images from GenSyn10 synthetic images (In-Distribution) and then tested on synthetic images from the held-out Stable Diffusion 3.5 Large (Out-of-Distribution).

Key Contributions

1. Multi-Generator Benchmark: GenSyn10 is the first CIFAR-10-equivalent dataset to include images from FLUX.2-dev, HunyuanImage-3.0, and Qwen-Image-2512, covering distinct generative families absent from prior benchmarks.
2. Combinatorial Prompt Grammar: A standardized, template-based system ensuring diverse, reproducible generation with controllable difficulty, eliminating prompt-content confounds.
3. Comprehensive Classifier Benchmark: A systematic evaluation of 17 architectures under a four-stage protocol, establishing performance baselines for current-generation synthetic images.
4. Systematic OOD Diagnosis: A controlled experimental setup using a held-out generator to quantify the vulnerability of detectors to unseen architectures, demonstrating that architectural inductive bias is a primary determinant of OOD resilience.

Results

Zero-Shot and Fine-Tuning Performance

• Zero-Shot Transfer: CIFAR-10-trained models demonstrated high semantic transferability to GenSyn10. The best model (Swin-Base) achieved 96.86% zero-shot accuracy, while the weakest (MNASNet-1.0) still reached 74.32%.
• Fine-Tuning Adaptation: After fine-tuning on GenSyn10, nearly all models exceeded 98.5% accuracy, with ConvNeXt-Base achieving 99.88%.
• Retention: Transformer-based models (ViT, Swin) exhibited superior retention of original CIFAR-10 knowledge (e.g., ViT-Base/16 retained 95.12% accuracy) compared to classical CNNs, which suffered significant forgetting (e.g., ResNet-50 dropped to 52.02%).

Binary Real-vs-Synthetic Classification

• In-Distribution (ID): Models achieved near-saturated performance on the GenSyn10 test set, with Swin-Base reaching 99.94% accuracy.
• Out-of-Distribution (OOD): When tested on the held-out Stable Diffusion 3.5 Large, all models experienced accuracy drops ranging from 3.80 to 18.15 percentage points.
  • ViT-Base/16 demonstrated the strongest OOD robustness, with the smallest accuracy drop (−3.80%) and the highest OOD accuracy (96.08%).
  • ConvNeXtV2-Base suffered the largest drop (−18.15%).
• Analysis: The performance degradation was primarily driven by a drop in synthetic recall (misclassifying unseen synthetic images as real), while specificity remained high. High AUC scores (near 1.0) persisted even with accuracy drops, suggesting that while thresholded performance degrades, the models retain some ability to rank synthetic vs. real samples.

Significance and Claims

The paper positions GenSyn10 as a controlled benchmark to address the "OOD generator shift" problem, which is identified as a central open challenge in the field. The authors claim that:

• Architectural Bias Matters: The results suggest that architectural inductive bias (specifically global self-attention in transformers) is a more significant factor in OOD resilience than model capacity alone.
• Generalization Limits: High performance on known generators does not guarantee robustness against unseen architectures, highlighting a persistent vulnerability in current detection methods.
• Standardization: By providing a standardized, multi-generator dataset with a held-out evaluation protocol, GenSyn10 enables principled research into domain adaptation and cross-generator generalization, moving beyond single-generator benchmarks that may overfit to specific artifact signatures.

The authors note limitations, including the exclusion of certain paradigms (e.g., GANs, consistency models) and the reliance on manual inspection for generation errors, but emphasize the dataset's extensibility for future updates as generative models evolve.