ForgeryGPT: A Multimodal LLM for Interpretable Image Forgery Detection and Localization

1. Problem Statement

The rapid advancement of image editing tools, particularly diffusion-based generative models, has made photo manipulation increasingly sophisticated and widespread. This poses severe risks to societal trust, judicial integrity, and information security.

• Limitations of Existing IFDL Methods: Traditional Image Forgery Detection and Localization (IFDL) approaches primarily rely on learning low-level, semantic-agnostic cues (e.g., noise patterns, texture inconsistencies). They often fail to capture high-level forensic semantics and lack the ability to provide interpretable reasoning. Most output only a confidence score or a binary label, requiring manual thresholding and offering no explanation for why an image is forged.
• Limitations of General MLLMs: While Multimodal Large Language Models (MLLMs) like GPT-4o excel at visual reasoning, they lack specific forensic expertise. They are often insensitive to subtle tampering artifacts and cannot perform precise pixel-level localization without specialized adaptation.

2. Methodology: ForgeryGPT Framework

ForgeryGPT is a novel framework designed to bridge the gap between high-level language reasoning and low-level forensic analysis. It integrates an Image Encoder, a Mask-Aware Forgery Extractor, and a Large Language Model (LLM).

A. Core Architecture Components

1. Image Encoder: Based on the CLIP Vision Transformer (ViT), it extracts foundational visual tokens and aligns them with the LLM's embedding space using a two-layer MLP.
2. Mask-Aware Forgery Extractor: This is the core innovation, comprising two sub-modules:
   • Forgery Localization Expert (FL-Expert): Built upon a frozen CLIP architecture, it is enhanced with:
     • Object-agnostic Forgery Prompt: Learnable text prompts that adapt dynamically to capture abstract forgery concepts across diverse object categories, rather than relying on fixed object-specific descriptions.
     • Vocabulary-enhanced Vision Encoder: A trainable visual vocabulary network (sharing ViT architecture) that enriches CLIP's representations with fine-grained, domain-specific visual cues for tampering.
     • Multi-scale Fusion: Aggregates features from multiple transformer layers to capture forgery evidence at different granularities.
     • Output: Generates a precise forgery localization mask.
   • Mask Encoder: Converts the generated forgery mask into "mask tokens" aligned with the LLM's feature space. These tokens are interleaved with image and text tokens, allowing the LLM to reason about the specific location and nature of the forgery.
3. Large Language Model (LLM): A customized LLM (e.g., Vicuna-7B) that processes the interleaved tokens (Image, Mask, Text) to perform detection, localization, and generate natural language explanations.

B. Three-Stage Training Paradigm

To adapt the model to the intricacies of IFDL, the authors propose a progressive training strategy:

1. Stage 1: Image-Text Alignment Pre-training: Aligns global image representations with text embeddings using the CC3M dataset. Only the image-level MLP projector is trained.
2. Stage 2: Mask-Text Alignment Pre-training: A critical stage where the Mask-Aware Forgery Extractor is trained. The authors constructed a specialized dataset of 76,013 forged image-caption pairs (covering splicing, copy-move, and removal) synthesized via advanced inpainting and repainting techniques. This stage aligns region-level mask features with text embeddings, enabling the model to understand forgery evidence semantically.
3. Stage 3: Task-Specific Instruction Tuning: The model is fine-tuned on 48,000 multi-turn dialogue instances derived from benchmark datasets (CASIAv2, Fantastic-Reality). This stage refines the model's ability to follow instructions, identify forged objects, localize regions, classify forgery types, and provide reasoning.

3. Key Contributions

1. First MLLM for IFDL: Pioneers the application of Multimodal LLMs to the IFDL domain, enabling threshold-free detection and multi-turn interactive dialogue.
2. Mask-Aware Forgery Extractor: Introduces a novel module that extracts precise forgery masks and encodes them into the LLM's feature space, facilitating pixel-level understanding of tampering artifacts.
3. FL-Expert Design: Develops a CLIP-based module featuring Object-agnostic Prompts (for generalization across object types) and a Vocabulary-enhanced Vision Encoder (for fine-grained feature enrichment).
4. Comprehensive Data & Training: Constructs two high-quality, large-scale datasets (Mask-Text Alignment and Instruction Tuning) and a three-stage training pipeline that progressively builds vision-language alignment and forensic reasoning capabilities.
5. Interpretability: Moves beyond binary classification to provide detailed, human-interpretable explanations, identifying forged objects, types, and regions with reasoning.

4. Experimental Results

Extensive experiments were conducted on eight benchmark datasets (CASIA1.0+, Columbia, NIST16, IMD2020, DSO-1, Korus, AutoSplice, OpenForensics).

• Localization Performance: ForgeryGPT achieved state-of-the-art (SOTA) performance in pixel-level localization.
  • It attained the highest F1-score and AUC on most datasets.
  • Notably, it reached an AUC of 91.4% on the real-world IMD2020 dataset, outperforming specialized CNN-based methods (e.g., HiFi-Net, DiffForensics) and other MLLM-based approaches (FAKESHIELD, SIDA).
• Detection Performance: It achieved the highest average accuracy (80.4%) and F1-score (79.1%) across datasets, demonstrating robustness without manual threshold tuning.
• Explainability:
  • Quantitative: Outperformed GPT-4o and LLaVA in explanation quality metrics (ROUGE-L, BLEU-4, SPICE, CSS).
  • Human Evaluation: In a study with 100 forged images, 81% of images initially misclassified as "real" by humans were correctly identified as "fake" after reading ForgeryGPT's explanations. The model also increased user confidence in correct judgments by 65%.
• Robustness: The model maintained high performance under various image degradations (resizing, Gaussian blur, noise, JPEG compression), significantly outperforming baselines under JPEG compression.
• Generalization: Demonstrated strong zero-shot transferability to industrial anomaly detection tasks (MVTec-AD, VisA), proving its capability extends beyond forgery detection.

5. Significance

ForgeryGPT represents a paradigm shift in digital forensics by integrating high-order forensic knowledge with generative AI.

• Trust and Accountability: By providing interpretable, step-by-step reasoning, it addresses the "black box" nature of current detection systems, fostering user trust.
• Adaptability: Its ability to engage in dialogue allows for flexible querying (e.g., "Where is the forgery?" or "What type of manipulation is this?"), making it a practical tool for non-experts and experts alike.
• Future Direction: The work highlights the potential of combining specialized forensic modules with the reasoning power of LLMs, paving the way for more robust, explainable, and user-aligned AI systems for combating misinformation.