IncreFA: Breaking the Static Wall of Generative Model Attribution

1. Problem Statement

The rapid evolution of AI generative models (diffusion, adversarial, and autoregressive) has rendered traditional image attribution methods obsolete. Existing approaches face two critical limitations:

• Static Assumption: Most methods operate under a "closed-set" assumption, trained on a fixed set of known generators. They fail when new models are released, requiring retraining from scratch.
• Catastrophic Forgetting & Negative Transfer: Conventional Class-Incremental Learning (CIL) methods struggle here because generative models are not independent classes; they form hierarchical lineages (e.g., Stable Diffusion 1.4 $\to$ 1.5 $\to$ 2.0). New versions inherit statistical features from predecessors, causing feature drift and negative transfer when standard replay or regularization techniques are applied.
• Open-Set Blindness: Real-world scenarios require distinguishing between known generators, real images, and unseen (unknown) generators. Most attribution pipelines lack robust mechanisms to detect these unseen instances without high false alarm rates.

Goal: To develop an attribution framework that continuously learns new generative models over time, retains knowledge of previous ones, respects the hierarchical relationships between models, and robustly detects unseen generators.

2. Methodology: IncreFA Framework

The authors propose IncreFA, a framework that redefines attribution as a structured incremental learning problem. It operates on a frozen pretrained visual encoder (e.g., CLIP ViT-B/16) and introduces two core mechanisms:

A. Hierarchical Constraints (Structural Priors)

Instead of treating models as independent classes, IncreFA explicitly models the architectural ancestry of generative models.

• Latent Space Decomposition: The latent space is decomposed into orthogonal subspaces for different families (e.g., Diffusion, GAN, AR) and further into specific models within those families.
• Learnable Orthogonal Priors: The method learns two types of anchors:
  1. Fine-Grained Anchors ($\hat{p}_{k,j}$): Unit-norm prototypes for specific models.
  2. Coarse-Level Anchors ($\hat{P}_k$): Prototypes for entire families.
• Loss Functions:
  • Orthogonality Loss ($L_1$): Enforces angular separation between all model prototypes to prevent confusion between specific models.
  • Regularization Loss ($L_2$): Ensures models within the same family share a common subspace (capturing family invariants) while remaining orthogonal to other families.
  • Result: This disentangles family-level invariants from model-specific idiosyncrasies, preventing negative transfer during incremental updates.

B. Latent Memory Bank (Continual Adaptation)

To handle the "forgetting" problem and simulate unseen data without storing raw images:

• Compact Latent Replay: Instead of storing raw images (which is memory-intensive and risks privacy), the system stores compact encoder features (latents). These are re-encoded through the current projection head to prevent projection drift.
• Random Mixing for Unseen Simulation: To improve open-set awareness, the system generates pseudo-unseen samples by linearly interpolating latent vectors from different classes ($z_u = \beta z_1 + (1-\beta)z_2$).
  • These mixed samples lie in low-density regions near decision boundaries, acting as "hard negatives."
• Open-Set Calibration: A threshold-based classifier is trained to penalize excessive confidence on these pseudo-unseen samples. During inference, if the maximum softmax probability is below a calibrated threshold $\tau$, the image is flagged as "Unseen."

C. Overall Objective

The total loss function combines:

1. Standard classification loss ($L_{cls}$).
2. Hierarchical orthogonality and regularization losses ($L_1, L_2$).
3. Open-set calibration loss ($L_u$).
4. Replay loss from the latent memory bank ($L_{replay}$).

3. Key Contributions

1. Incremental Attribution Formulation: The first framework to treat image attribution as a structured incremental learning problem, explicitly addressing the hierarchical nature of generative models.
2. IABench Benchmark: A new benchmark covering 28 generative models released between 2022 and 2025 (including GANs, Diffusions, and ARs), with temporally ordered splits to simulate real-world model evolution.
3. Novel Architectural Design: The integration of Hierarchical Constraints (to handle lineage) and Latent Memory Banks (to handle memory efficiency and open-set simulation) creates a robust, data-efficient system.
4. State-of-the-Art Performance: Demonstrated superior performance in both continual learning (retaining old knowledge) and open-set detection (identifying unknown models).

4. Experimental Results

Experiments were conducted under two protocols: EP1 (Incremental) and EP2 (Static/Closed-Set).

• Incremental Performance (EP1):

  • IncreFA achieved 78.80% average attribution accuracy in the final task (T7), significantly outperforming the second-best method (MOS at 75.68%).
  • It maintained 98.93% accuracy in detecting unseen models (Unseen Accuracy), whereas other methods dropped significantly or lacked open-set capabilities.
  • Baselines like L2P and DualPrompt suffered severe forgetting (accuracy dropped to ~35-40%) due to their inability to handle overlapping distributions.

• Static Performance (EP2):

  • Even without incremental constraints, IncreFA achieved 95.93% average attribution accuracy and 99.97% authenticity detection (Real vs. Fake), outperforming specialized attribution models like DNA-Net and RepMix.

• Ablation Studies:

  • Removing the hierarchical constraints ($L_1, L_2$) led to a significant drop in performance, confirming the necessity of modeling family structures.
  • The Latent Memory Bank was crucial for stabilizing performance across tasks.
  • The random mixing strategy effectively improved open-set detection rates.

5. Significance and Impact

• Paradigm Shift: Moves image attribution from a static, one-time classification task to a dynamic, evolving process that mirrors the lifecycle of generative AI.
• Practical Applicability: By using a Latent Memory Bank, IncreFA is highly data-efficient and privacy-preserving (no raw image storage), making it viable for real-world deployment where storage and legal constraints exist.
• Trust and Accountability: Provides a robust mechanism for identifying the source of AI-generated content, even as new models emerge, which is critical for combating copyright infringement, deepfakes, and misinformation.
• Future Direction: Highlights the need for attribution systems to evolve structurally alongside the models they trace, rather than merely reacting to them.

In summary, IncreFA breaks the "static wall" of current attribution methods by leveraging the inherent hierarchy of generative models and employing continual learning strategies tailored for open-world scenarios, setting a new standard for forensic analysis in the era of generative AI.