🤖 AI

Defake-o3: From Speculative Rationales to Verifiable Evidence for Explainable AIGI Detection

The paper introduces Defake-o3, an explainable AI-generated image detector that replaces speculative rationales with verifiable visual evidence by combining iterative visual search with a human-trained Evidence Verifier, supported by the new GroundFake dataset and FakeFrontier benchmark.

Original authors: Bowen Deng, Jiahui Zhan, Yikun Ji, Haozhen Yan, Jianfu Zhang

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bowen Deng, Jiahui Zhan, Yikun Ji, Haozhen Yan, Jianfu Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the last few years, computers have learned to create images that look indistinguishable from photographs taken with a camera. These synthetic pictures, often called AI-generated images, can show people, landscapes, and objects with such realism that the human eye struggles to tell them apart from reality. While this technology offers new tools for artists and designers, it also creates a significant problem: if we cannot easily spot a fake, we cannot trust what we see. This has led researchers to build detectors, or automated systems, designed to spot the subtle clues that reveal an image was made by a machine. For a long time, these detectors worked like a black box: they would look at a picture and simply say "real" or "fake," offering no explanation for why. This lack of transparency made it hard for people to trust the results, especially when the stakes were high. More recently, scientists have tried to give these systems a voice, asking them to explain their reasoning in plain language. However, these explanations often turn out to be guesses or vague observations that cannot be checked, leaving the user with a verdict but no proof.

A team of researchers at Shanghai Jiao Tong University has developed a new system called Defake-o3 that changes how these detectors work. Instead of guessing or offering vague descriptions, this system is designed to find concrete, visual proof that can be seen and verified by a human. The core idea is to move from speculative reasoning to verifiable evidence. Imagine a person trying to find a flaw in a painting; they might look at the whole picture first, but if they want to be sure, they need to get closer to the canvas to inspect the brushstrokes. Defake-o3 does exactly this. It uses an interactive process where it first looks at the entire image, then decides to zoom in on specific areas that look suspicious. It repeats this process, getting closer and closer to fine details, until it finds something that clearly does not belong.

The system is built on two main parts. The first part is the ability to search visually. When the system sees an image, it does not just scan it once from start to finish. Instead, it acts like a careful investigator, choosing to zoom in on specific spots, such as the text on a sign, the texture of a hand, or the wheels of a car. By magnifying these areas, it can spot tiny errors that are invisible in the full picture, such as letters that are jumbled together or objects that melt into the background. The second part is a built-in check that ensures the system is not making things up. The researchers trained a separate "verifier" using thousands of examples where humans had already confirmed whether a specific detail was a real flaw or a mistake. This verifier acts as a judge, rewarding the system when it finds evidence that is grounded in the actual image and penalizing it when it makes up flaws that do not exist. This training ensures that the system only reports what it can actually see.

To teach this system how to work, the researchers created a new dataset called GroundFake. This collection contains 16,000 images, split evenly between real photos and AI-generated ones. What makes this dataset special is that it includes detailed notes on exactly where the flaws are and what they look like, verified by human experts. The researchers also built a new testing ground called FakeFrontier, which includes images from ten different, very recent AI generators that the system had never seen before. This allowed them to test if the system could handle the latest technology, not just the old examples it was trained on.

The results show that Defake-o3 is significantly better at both finding fakes and explaining why. On the training dataset, it correctly identified fake images with high accuracy, but more importantly, the evidence it provided was much more precise. When the system said an image was fake, it could point to a specific location, like a distorted logo on a hat or nonsensical text on a sign, and describe exactly what was wrong. In tests on the new, unseen generators, the system continued to perform well, correctly identifying fakes that other methods missed. When researchers asked other advanced AI models to judge the quality of the evidence provided by Defake-o3, they found it to be far more convincing and grounded than the explanations offered by previous systems. Those older systems often gave vague answers, such as saying an image looked "too smooth" without pointing to where, or they hallucinated flaws that were not there. Defake-o3, by contrast, provided a small number of strong, specific pieces of evidence that a human could easily verify by looking at the image themselves.

The study also explored what happens when the system is not guided by the verifier or when it is not allowed to zoom in. Without the ability to zoom, the system struggled to find the subtle flaws hidden in the details. Without the verifier, it tended to make up reasons for its decisions, even when it was wrong. This confirmed that both the interactive search and the strict checking of evidence are necessary for the system to work. The researchers found that the best performance came from balancing the system's own observations with the guidance of the verifier, ensuring that it did not rely too heavily on simple matching of words or patterns, but instead focused on the actual visual truth.

This work represents a shift in how we approach the problem of AI-generated images. It moves away from the idea that a computer must simply give a score or a guess, and toward a model where the computer acts as a transparent partner, showing its work step by step. By combining the ability to look closer with a strict requirement for proof, the system offers a level of reliability that was previously missing. The researchers conclude that for AI-generated image detection to be truly useful and trustworthy, it must be able to provide evidence that can be seen and understood, not just a verdict that cannot be checked. Defake-o3 demonstrates that this is possible, offering a path forward where technology helps us see the truth more clearly, one zoomed-in detail at a time.

Technical Summary: Defake-o3

1. Problem Statement

The rapid advancement of AI image generation (AIGI) models has produced synthetic images that are increasingly photorealistic, posing significant risks to public trust through the spread of misinformation. While traditional detection methods based on Convolutional Neural Networks (CNNs) and Vision Transformers achieve high accuracy, they function as "black boxes," providing only authenticity scores without revealing the visual evidence behind their decisions.

Recent efforts to utilize Multimodal Large Language Models (MLLMs) for explainable detection have introduced natural language rationales. However, the paper identifies critical limitations in these existing approaches:

Speculative Rationales: Current MLLM detectors often generate vague, generic, or hallucinated explanations rather than evidence grounded in specific visual artifacts.
Lack of Verifiability: Explanations frequently fail to provide evidence that can be visually verified by humans.
Insufficient Granularity: Standard MLLMs typically perform single-pass, coarse-grained inspections. They struggle to detect subtle, localized flaws left by modern generators due to input resolution constraints and a lack of iterative exploration.

Consequently, existing explanations are often ungrounded, poorly localized, and difficult to verify, limiting their trustworthiness in high-stakes settings.

2. Methodology: Defake-o3

Defake-o3 is an explainable AIGI detector designed to transition from speculative rationales to verifiable evidence. It integrates two core components: interactive visual search and verifier-guided evidence alignment.

2.1 Interactive Visual Search

Defake-o3 adopts an iterative "thinking-with-images" mechanism. Instead of a single-pass inspection, the model utilizes a zoom-in tool to dynamically crop and magnify suspicious regions. This allows the model to inspect fine-grained details and uncover subtle artifacts that are missed in full-image encoding. The process involves multi-turn interactions where the model decides whether to zoom into a specific region (predicting a 2D bounding box) or to finalize its verdict.

2.2 Verifier-Guided Evidence Alignment

To ensure the generated evidence is valid and not hallucinated, the system employs an Evidence Verifier trained on human verification annotations. This verifier acts as a reward model during Reinforcement Learning (RL). It evaluates whether a claimed piece of evidence is:

Visually Grounded: The text description accurately matches the content within the bounding box.
Artifact-Specific: The described flaw is specific to AI generation rather than a generic attribute of real photographs.

2.3 Training Pipeline

The training process consists of two stages:

Supervised Fine-Tuning (SFT): The base MLLM is fine-tuned on the GroundFake dataset (described below) using human-filtered, trajectory-corrected traces. This stage teaches the model to use the zoom-in tool and generate structured outputs (verdict, global evidence, and localized key evidence).
Reinforcement Learning (GRPO): The model is optimized using Group Relative Policy Optimization (GRPO). The reward function ( $R_{total}$ $R_{t o t a l}$ ) combines:
- Classification Reward ( $R_{cls}$ ): Accuracy of the final verdict.
- Evidence Reward ( $R_{evi}$ ): A hybrid metric combining rule-based matching (IoU and BLEU against reference annotations) and a model-based score from the Evidence Verifier. The verifier assigns validity scores to evidence items, penalizing baseless claims and rewarding grounded, specific artifacts.

3. Key Contributions

3.1 The Defake-o3 Framework

The authors propose a novel architecture that combines iterative visual exploration with a verifier-guided reward mechanism. This approach enables the model to produce structured verdicts supported by global context and localized, verifiable key evidence, effectively moving beyond vague rationales.

3.2 GroundFake Dataset

To support grounded explainable detection, the authors constructed GroundFake, a debiased training dataset containing 16,000 images (8k real, 8k fake). Key features include:

Bias Control: Standardized file formats, aspect ratios, semantic categories, and aesthetic qualities to prevent the model from learning source-specific shortcuts.
Human-Verified Annotations: Localized bounding-box evidence is manually verified by human annotators based on strict criteria of visual grounding and artifact specificity.
Corrected Trajectories: Reasoning transcripts are rewritten to ensure consistency with the verified evidence, removing hallucinated claims.
Dual Supervision: The dataset provides trajectories for SFT and valid/invalid evidence labels for training the Evidence Verifier.

3.3 FakeFrontier Benchmark

The authors introduce FakeFrontier, an out-of-distribution (OoD) benchmark containing 4,000 images (2k real, 2k fake) generated by 10 recent models (including Nano Banana and Seedream 4.5). Unlike GroundFake, it lacks human evidence annotations, serving as a testbed for generalization. The authors also propose an MLLM-based protocol to evaluate evidence quality and persuasiveness without reference boxes, using multiple judge models to assess visual grounding and the ability to convince a judge of a "fake" verdict.

4. Experimental Results

4.1 Performance on GroundFake

On the GroundFake test set, Defake-o3 achieves state-of-the-art performance:

Accuracy/F1: Matches the best results (0.992) among all compared methods.
Evidence Quality: Achieves the highest scores in lexical overlap (BLEU-1/2, ROUGE-L) and spatial overlap (IoU) with reference annotations, indicating more precise and grounded evidence generation compared to baselines like FakeVLM, LEGION, and FakeShield.
Ablation: The inclusion of the Evidence Verifier and interactive search significantly improves IoU and classification accuracy compared to variants without RL or without the zoom-in mechanism.

4.2 Generalization on FakeFrontier and OoD Benchmarks

FakeFrontier: Defake-o3 achieves the highest classification accuracy (0.9232) and F1 score (0.9246). In the MLLM-based evaluation, it ranks first in Quality Score (QS), Hit@Img (proportion of images where at least one evidence item convinces the judge), and Hit@Evi (proportion of evidence items that are convincing). This demonstrates that its evidence is not only accurate but also persuasive and specific to AI artifacts.
External OoD Benchmarks: Defake-o3 outperforms all baselines (including black-box detectors like CNNSpot and NPR) on AIGI-Now, EvalGEN, and MNW, achieving accuracies of 0.9180, 0.9872, and 0.8871 respectively.

4.3 Qualitative Analysis

Qualitative examples show that Defake-o3 successfully identifies specific artifacts (e.g., nonsensical text, distorted objects, invalid timestamps) through iterative zooming, whereas baseline methods often provide vague, global, or hallucinated explanations.

5. Significance and Claims

The paper claims that Defake-o3 represents a significant step forward in explainable AIGI detection by addressing the gap between high detection accuracy and the reliability of explanations. Its primary significance lies in:

Shifting from Speculation to Verification: It moves the field away from generating plausible-sounding but unverified rationales toward producing evidence that is visually grounded and artifact-specific.
Enhancing Trustworthiness: By providing localized, verifiable evidence, the system makes its decisions interpretable and auditable by humans, which is crucial for deployment in high-stakes scenarios.
Robustness to Distribution Shifts: The combination of interactive search and verifier-guided alignment improves robustness against new, unseen generators without sacrificing classification strength.

The authors conclude that their work establishes a new standard for training and evaluating explainable detectors, emphasizing that verifiable evidence is a necessary component for trustworthy AI-generated image detection.