🤖 AI

AIFIND: Artifact-Aware Interpreting Fine-Grained Alignment for Incremental Face Forgery Detection

The paper proposes AIFIND, an incremental face forgery detection framework that mitigates catastrophic forgetting and feature drift by leveraging artifact-driven semantic anchors and adaptive decision harmonization to enforce fine-grained feature alignment across evolving forgery types.

Original authors: Hao Wang, Beichen Zhang, Yanpei Gong, Shaoyi Fang, Zhaobo Qi, Yuanrong Xu, Xinyan Liu, Weigang Zhang

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hao Wang, Beichen Zhang, Yanpei Gong, Shaoyi Fang, Zhaobo Qi, Yuanrong Xu, Xinyan Liu, Weigang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a high-security airport. Your job is to spot fake passports.

The Problem:
Every day, new types of forgeries appear. Yesterday, forgers were bad at copying the paper texture. Today, they are perfect at copying the paper but terrible at the ink color. Tomorrow, they might be perfect at the ink but fake the photo.

If you try to learn these new tricks by memorizing every single fake passport you've ever seen, you will eventually run out of memory space. If you try to forget the old ones to make room for the new ones, you'll start forgetting how to spot the old fakes. This is called "Catastrophic Forgetting."

Most current security systems try to solve this by keeping a "Replay Box"—a small pile of old fake passports they keep looking at to remember them. But this is slow, takes up space, and raises privacy issues.

The Solution: AIFIND (The "Universal Truth" Guard)

The authors of this paper propose a new system called AIFIND. Instead of memorizing specific fake passports, they teach the guard to recognize universal "flaws" that exist in any fake, no matter how it was made.

Here is how it works, using simple analogies:

1. The "Semantic Anchors" (The Rulebook)

Imagine that instead of showing the guard a pile of old fake passports, you give them a Rulebook of Flaws.

Rule 1: "Real eyes blink naturally; fake eyes often look blurry."
Rule 2: "Real skin has consistent lighting; fake skin often has weird shadows."
Rule 3: "Real jawlines connect smoothly; fake jawlines often look jagged."

In the paper, these are called Semantic Anchors. They are stable, unchanging truths about what makes a face look "real" or "fake." No matter how the forgers change their tricks, these fundamental flaws (blurry eyes, bad lighting) usually remain.

2. The "Artifact-Driven Generator" (The Translator)

The system has a smart assistant (an AI) that looks at a new, suspicious photo. It doesn't just say "Fake." It acts like a translator, converting the messy visual mess into the language of the Rulebook.

Visual: "Hey, the eyes look a bit smudged."
Translation: "This matches Rule 1: Blurry Eyes."

This is the Artifact-Driven Semantic Prior Generator. It turns messy pixels into clear, stable concepts.

3. The "Probe" (The Magnifying Glass)

Once the system knows which "Rule" applies (e.g., Blurry Eyes), it uses a magnifying glass called Artifact-Probe Attention.

Instead of looking at the whole face vaguely, the magnifying glass zooms in specifically on the eyes to check for that blur.
This forces the computer to focus on the specific detail that matters, rather than getting distracted by the background or the person's hair.

4. The "Harmonizer" (The Memory Keeper)

As the guard learns new tricks (e.g., a new type of fake jawline), they might accidentally forget the old rules (e.g., how to spot fake eyes).

The Adaptive Decision Harmonizer is like a wise mentor. It says, "Hey, you are learning a new rule about jawlines, but don't forget the rule about eyes. Keep the angle between these two rules consistent."
It ensures that the guard's brain stays organized. It doesn't let the new information push the old information out of place. It keeps the "geometry" of their knowledge stable.

Why is this better?

No Replay Box Needed: You don't need to store thousands of old fake photos. You just need the Rulebook (the Semantic Anchors). This saves space and protects privacy.
Stable Learning: Because the "Rules" (like "blurry eyes are bad") never change, the system doesn't get confused or forgetful when new forgeries appear. It just checks the new photo against the same old, reliable rules.
Better Focus: It stops the AI from guessing based on background noise and forces it to look at the actual "suspicious" parts of the face.

The Result

In their tests, this new system (AIFIND) was much better at spotting new types of fakes without forgetting how to spot old ones, compared to other systems that rely on memorizing old examples. It's like upgrading from a guard who memorizes faces to a guard who understands the physics of forgery.

1. Problem Statement

The rapid advancement of generative AI has led to increasingly realistic face forgeries, posing significant security risks. While Incremental Face Forgery Detection (IFFD) aims to continuously train models on new forgery types without forgetting previous knowledge, existing methods face critical limitations:

Reliance on Data Replay: Most current approaches (e.g., DFIL, SUR-LID) store subsets of past data to mitigate catastrophic forgetting. This raises privacy concerns and storage constraints.
Coarse Supervision: Existing methods often rely on binary labels (Real vs. Fake), failing to leverage fine-grained artifact cues (e.g., blurry eyes, unnatural jawlines).
Feature Drift: Without stable constraints, the feature space drifts as new tasks are introduced, causing the model to overwrite previous representations and leading to catastrophic forgetting.
Limitations of VLMs: While Vision-Language Models (VLMs) like CLIP show promise, they are typically used for static scenarios or coarse global labels, neglecting the specific, region-level artifact semantics required for forgery detection.

2. Methodology: AIFIND Framework

The authors propose AIFIND, a data-replay-free framework that uses Semantic Anchors derived from low-level artifacts to stabilize the incremental learning process. The framework consists of four core components:

A. Artifact-Driven Semantic Prior Generator (ASPG)

Goal: To instantiate invariant semantic anchors from volatile visual cues.
Process:
1. Low-level Analysis: The system defines 5 forgery dimensions (Blur, Color, Structure, Texture, Boundary) and maps them to 6 facial regions (Eyes, Nose, Cheeks, Mouth, Jawline, Boundary) using MediaPipe.
2. Metric Calculation: It computes specific indicators (e.g., Laplacian variance for blur, SSIM for structure, CIELAB luminance deviation for color) to quantify anomalies.
3. Semantic Anchoring: An LLM generates contrastive text pairs describing real vs. fake states for each dimension/region. A CLIP-based similarity search selects the optimal textual descriptions as Semantic Anchors.
4. Dynamic Selection: For a given image, the system selects the top- $N$ most severe anomalies (for fake images) or pristine details (for real images) to retrieve corresponding textual anchors.

B. Semantic-Guided Incremental Detector (SGID)

This module injects the selected semantic anchors into the visual learning process.

Artifact-Probe Attention (APA): A cross-modal attention mechanism within the Vision Transformer (ViT). Visual tokens act as queries to probe the textual embeddings of the matched semantic anchors. A learnable gating mechanism dynamically modulates the injection of these semantic priors to prevent overriding intrinsic visual cues.
Dual Supervision:
1. Binary Classification: Predicts global authenticity (Real/Fake).
2. Multi-label Classification: Predicts the presence of specific artifact dimensions (e.g., "Blurry Eyes," "Inconsistent Lighting"). This forces the model to align visual features with stable semantic definitions, preventing feature drift.

C. Adaptive Decision Harmonizer (ADH)

To prevent decision boundary drift across tasks without data replay:

Geometric Consistency: ADH aligns the weights of the current task's classifiers with historical knowledge.
Spherical Alignment: Instead of Euclidean interpolation, it performs alignment on a hypersphere. It computes a "Global Semantic Reference" based on the angular similarity of previous task weights.
Geodesic Update: The current decision boundary is rotated toward the global reference along a geodesic path. An adaptive coefficient balances plasticity (learning new tasks) and stability (preserving old knowledge) based on semantic similarity.

D. Training Strategy

Warm-up: The first $n$ epochs use fixed anchors for stability.
Dynamic Matching: Subsequent epochs adaptively retrieve anchors based on cosine similarity between visual features and text embeddings.
Loss Function: Combines Binary Cross-Entropy (authenticity), Multi-label BCE (artifact types), and Knowledge Distillation (feature consistency with the previous task).

3. Key Contributions

Data-Replay-Free Paradigm: A novel framework that eliminates the need for storing past samples by using invariant semantic anchors to constrain the feature space.
Fine-Grained Artifact Alignment: The introduction of Artifact-Probe Attention, which explicitly anchors volatile visual forgery cues to immutable semantic descriptions, effectively preventing feature drift.
Geometric Decision Harmonization: A spherical alignment strategy (ADH) that preserves the angular relationships of classifiers across tasks, ensuring geometric consistency and mitigating catastrophic forgetting.
Comprehensive Validation: Extensive experiments demonstrating state-of-the-art performance across multiple incremental protocols and unseen domains.

4. Experimental Results

The method was evaluated on standard IFFD protocols using datasets like DFDCP, Celeb-DF-v2, FaceForensics++, and newer diffusion-based datasets (SDv21).

Performance: AIFIND achieved superior AUC scores compared to state-of-the-art replay-based methods (e.g., SUR-LID, DFIL) and general incremental learning methods (e.g., Coda-Prompt, CL-LoRA).
- Example (Protocol 1 Avg): AIFIND (0.9686) vs. SUR-LID (0.9315) vs. DFIL (0.8628).
Generalization: In cross-dataset evaluations on unseen benchmarks (DFD, UniFace, SDv15, FAVC), AIFIND significantly outperformed baselines, proving its ability to learn transferable forensic features rather than dataset-specific artifacts.
Robustness: Under various image perturbations (blur, noise, compression), AIFIND maintained higher AUC scores, indicating robustness to image distortions.
Ablation Studies:
- Removing ADH caused significant decision boundary drift.
- Removing Dual Supervision ( $L_{ind}$ ) impaired the model's ability to disentangle specific artifacts.
- High-level injection of APA (into deeper transformer layers) yielded the best results, confirming that semantic anchors align best with high-level semantic features.

5. Significance

Privacy & Efficiency: By removing the need for data replay buffers, AIFIND addresses critical privacy concerns and reduces storage overhead, making it more deployable in real-world scenarios.
Interpretability: The use of semantic anchors provides a "white-box" mechanism where the model's decisions can be traced back to specific, human-understandable artifact descriptions (e.g., "unnatural jawline"), enhancing trust.
Scalability: The framework offers a robust solution for the open-world problem of face forgery detection, where new manipulation techniques emerge continuously, ensuring the detector remains effective without retraining from scratch or storing massive datasets.

In conclusion, AIFIND represents a paradigm shift in incremental forgery detection, moving from data-centric replay strategies to semantics-centric alignment, effectively solving the catastrophic forgetting problem while enhancing model interpretability and generalization.