💻 computer science

VIGIL: Part-Grounded Structured Reasoning for Generalizable Deepfake Detection

VIGIL is a part-grounded structured reasoning framework that improves generalizable deepfake detection by decoupling facial part planning from evidence examination through a stage-gated injection mechanism and progressive training, achieving superior performance on the newly constructed OmniFake benchmark.

Original authors: Xinghan Li, Junhao Xu, Jingjing Chen

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinghan Li, Junhao Xu, Jingjing Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Is this photo of a person real, or is it a perfect forgery created by AI?

For a long time, computers tried to solve this by looking at the whole picture at once and giving a simple "Yes" or "No" answer. But as AI gets better at faking photos, these simple detectors get confused. They can't explain why they think it's fake, and they often get tricked by high-quality fakes.

Recently, scientists tried using "Super-Intelligent AI" (called Multimodal Large Language Models) to act as detectives. These AIs can write explanations like, "The skin looks too smooth." But there was a big problem: The AI was lying to itself. It would guess the photo was fake, then invent a reason why, without actually checking the evidence. It was like a lawyer making up facts to win a case instead of finding the truth.

Enter VIGIL (Visual Intelligence for Generalized Inspection and Localization). Think of VIGIL not as a single detective, but as a forensic team that follows a strict, step-by-step procedure used by real human experts.

Here is how VIGIL works, using a simple analogy:

1. The "Plan-Then-Examine" Strategy

Imagine you are inspecting a used car to see if it's been in an accident.

Old AI: Looks at the car, says "It looks shiny," and immediately concludes, "It's a good car!" (or "It's a bad car!") without looking under the hood.
VIGIL:
- Step 1: The Plan (The Walkaround): First, VIGIL looks at the whole car and says, "Hmm, the paint looks perfect, but the tires look a bit weird. I'm going to focus my inspection on the tires and the bumper." It decides what to look at based on its own eyes, not on a cheat sheet.
- Step 2: The Examination (The Microscope): Now, VIGIL zooms in only on the tires and bumper. But here is the magic: It doesn't just use its eyes. It pulls out a special forensic toolkit (frequency analysis and pixel-level sensors) that normal humans and regular AI can't see. It checks the tires for microscopic scratches that prove they were replaced.
- Step 3: The Verdict: Only after gathering this hard evidence does it make a final decision. If the tires show signs of tampering, it changes its mind: "Okay, I thought it was a good car, but the evidence says it's been in a crash."

2. The "Cheat Sheet" Problem (Stage-Gated Injection)

In previous AI methods, the "forensic toolkit" was handed to the AI before it started thinking. This was bad because the AI would just look at the toolkit's data and say, "Oh, the toolkit says 'Tires are bad,' so I'll plan to look at the tires." It wasn't thinking; it was just following orders.

VIGIL uses a "Stage-Gated" system.

The Gate: The forensic data is locked in a vault.
The Rule: The AI can only open the vault after it has made its own plan.
The Result: The AI plans what to look at based on its own intuition. Then, it opens the vault to get the hard proof. This ensures the AI isn't cheating; it's actually investigating.

3. The Training: From Student to Expert

To teach VIGIL how to be a good detective, the researchers used a three-step training camp:

Classroom (Supervised Learning): They showed VIGIL thousands of examples where a human expert wrote down exactly what to look for and why. VIGIL learned the format.
Hard Mode (Self-Training): They gave VIGIL the "impossible" cases that confused the experts. VIGIL tried to solve them, and if it got it right, it kept the solution. If it failed, it tried again. This taught it how to handle tricky situations.
The Reward System (Reinforcement Learning): This is the most important part. They didn't just reward VIGIL for getting the right answer. They rewarded it for logic.
- Did you check the parts you said you would? (Part-Aware Reward)
- Did you make up evidence to support your conclusion? (Consistency Reward)
- If VIGIL tried to lie or hallucinate, it got a "bad grade." If it stuck to the facts, it got a "gold star."

4. The Ultimate Test: OmniFake

To prove VIGIL is the best, the researchers built a giant test called OmniFake.

Imagine a video game with 5 levels of difficulty.
Level 1: Easy fakes (old AI models).
Level 5: "In-the-Wild" fakes (created by the newest, most powerful AI models, posted on social media, compressed, and blurry).
VIGIL was trained only on Level 1.
The Result: When tested on Level 5 (the hardest stuff), VIGIL didn't just survive; it crushed the competition. It could spot fakes that even the best "expert" detectors missed.

Summary

VIGIL is a new way to catch AI fakes. Instead of guessing and making up reasons, it acts like a disciplined forensic team:

Plans what to look at.
Gathers hard, scientific evidence only for those specific parts.
Decides based on that evidence, even if it means changing its mind.

It's the difference between a detective who guesses "It's fake because it feels weird" and a detective who says, "It's fake because the microscopic texture of the left ear doesn't match the rest of the face."

1. Problem Statement

The rapid advancement of generative AI has exacerbated the risk of deepfakes, challenging public trust and media integrity. While traditional deepfake detectors (CNN-based or frequency-domain) achieve strong performance on in-distribution data, they suffer from two critical limitations:

Lack of Interpretability: They operate as black boxes, providing binary classifications without explaining why an image is fake or where the forgery lies.
Poor Generalizability: They struggle to detect fakes generated by unseen models or in "in-the-wild" scenarios (e.g., social media compression).

Recent attempts to use Multimodal Large Language Models (MLLMs) for interpretable detection have introduced new issues:

Hallucination vs. Observation: MLLMs often conflate their own generated text with visual evidence, making it impossible to distinguish faithful observations from hallucinations.
Loss of High-Frequency Details: General-purpose MLLM encoders suppress high-frequency forensic signals (crucial for detecting artifacts) in favor of semantic understanding.
Entangled Reasoning: Current methods generate claims and evidence simultaneously in a single step. The model acts as both the claimant and the evidence provider, lacking independent verification.

2. Methodology: The VIGIL Framework

VIGIL (Visual Inspection and Grounded Inference Logic) proposes a part-centric structured forensic framework inspired by human expert workflows. It decouples claim generation from evidence sourcing through a "Plan-Then-Examine" pipeline.

A. Architecture & Pipeline

The reasoning process is divided into five distinct stages:

Global Evidence: A high-level summary of the image.
Planning: The model identifies which specific facial parts (e.g., eyes, mouth, eyebrows) warrant inspection based on global visual cues.
Part-Level Evidence: The model examines the selected parts using independently sourced forensic signals.
Conclusion: Synthesis of part-level findings into a final verdict.
Answer: The final binary classification (Real/Fake).

B. Key Technical Components

Dual-Path Forensic Encoders: To overcome the MLLM's inability to see fine-grained artifacts, VIGIL uses two specialized encoders:
- Spectral Branch: Applies FFT-based high-pass filtering to extract frequency-domain anomaly features.
- Pixel-Level Branch: Uses a pre-trained DINOv3 to extract fine-grained visual features sensitive to pixel inconsistencies.
Part-Level Evidence Aggregation: The image is segmented into $K=8$ anatomical regions (e.g., eyes, nose, mouth) using a face parsing map. Features from the dual encoders are aggregated per region to create specific evidence embeddings ( $e_k$ ).
Context-Aware Dynamic Signal Injection (Stage-Gated):
- A global evidence summary is injected before reasoning begins.
- Crucially, part-level evidence is injected only during the Examination stage, not the Planning stage. This prevents the model from biasing its planning based on external signals, ensuring the selection of regions is driven by the model's own visual perception.
Semantic Binding: Part tokens (e.g., <P_MOUTH>) are bound to their specific visual evidence embeddings, ensuring the model reasons over concrete signals rather than abstract text triggers.

C. Progressive Three-Stage Training Paradigm

To transform the architecture into a robust reasoner, VIGIL employs a three-stage training process:

Stage 1 (SFT): Supervised Fine-Tuning on high-quality, structured annotations generated by an automated pipeline (Signal extraction $\to$ Multi-MLLM visual description $\to$ Expert LLM synthesis).
Stage 2 (Hard-Sample Self-Training): Uses rejection sampling on difficult samples (where Stage 1 failed) to generate correct reasoning trajectories, expanding the model's coverage of hard cases.
Stage 3 (Part-Aware Reinforcement Learning): Uses Group Relative Policy Optimization (GRPO) with four specific rewards:
- Accuracy ( $R_{acc}$ ): Correct classification.
- Format ( $R_{fmt}$ ): Adherence to the structured output.
- Part-Aware ( $R_{part}$ ): Ensures the model plans to inspect parts that actually exist and analyzes exactly what it planned (Plan-Evidence consistency).
- Consistency ( $R_{cons}$ ): An independent judge LLM verifies if the evidence provided alone supports the final conclusion, preventing the model from fabricating evidence to fit a pre-determined answer.

3. Key Contributions

VIGIL Framework: A novel part-centric architecture that decouples claims from evidence, enabling verifiable, region-grounded explanations.
Stage-Gated Injection: A mechanism that delivers forensic evidence only during the examination phase, preserving autonomous planning and preventing signal bias.
Progressive Training: A three-stage paradigm incorporating hard-sample self-training and forensic-specific reinforcement learning rewards to enforce anatomical validity and evidence-conclusion coherence.
OmniFake Benchmark: A new, hierarchical 5-Level benchmark designed to rigorously evaluate generalizability:
- Level 1: In-Distribution (same generators as training).
- Level 2: Cross-Architecture (unseen models within known paradigms).
- Level 3: Cross-Model (unseen generation principles, e.g., Flow Matching, Autoregressive).
- Level 4: Cross-Task (localized manipulations like inpainting/restoration).
- Level 5: In-the-Wild (social media data with unknown degradations).

4. Experimental Results

VIGIL was evaluated on OmniFake and the HydraFake dataset, outperforming both expert detectors and concurrent MLLM-based methods.

Overall Performance: On OmniFake, VIGIL achieved 93.1% overall accuracy, surpassing the previous best expert detector (DDA) by 4.3% and the best MLLM method (Veritas) by 5.5%.
Generalizability: The performance gap widened significantly on harder levels.
- On Level 5 (In-the-Wild), VIGIL maintained 86.0% accuracy, whereas expert detectors and other MLLMs dropped significantly.
- On Level 4 (Cross-Task), VIGIL achieved 89.5% accuracy, outperforming Veritas by 10.5%.
Reasoning Quality: Qualitative analysis and ELO ratings showed VIGIL significantly outperforms other models in reasoning coherence. Notably, VIGIL demonstrated "Reasoning Reversion," where it could correct an initial "Real" hypothesis after examining part-level evidence, a capability absent in holistic models.
Robustness: VIGIL showed superior robustness to JPEG compression and Gaussian blur without explicit training on these augmentations.

5. Significance

Paradigm Shift: VIGIL moves deepfake detection from "black-box classification" to "verifiable forensic reasoning." By grounding claims in independently sourced, region-specific evidence, it addresses the hallucination problem inherent in current MLLM approaches.
Generalization: The hierarchical OmniFake benchmark and VIGIL's performance demonstrate that decoupling planning from evidence injection allows models to generalize to entirely unseen generation paradigms and real-world conditions.
Broader Impact: The principle of decoupling claim generation from evidence grounding offers a generalizable paradigm for any visual forensic task requiring explainable and trustworthy AI decisions.