Generating Attribution Reports for Manipulated Facial Images: A Dataset and Baseline

1. Problem Statement

Existing facial forgery detection methods primarily focus on binary classification (real vs. fake) or pixel-level localization (generating a binary mask). While these approaches identify that an image is manipulated and where the manipulation occurred, they lack interpretability.

• Limitation: Binary masks treat all manipulated pixels equally, failing to differentiate between subtle artifacts and significant alterations. They do not explain why a region is considered forged (e.g., inconsistent skin texture, blurry edges, or anatomical asymmetry).
• Gap: As generative models (Diffusion, GANs) produce increasingly realistic forgeries, human reviewers need descriptive evidence to trust detection results. There is a lack of large-scale datasets and frameworks that jointly perform forgery localization and natural language explanation generation.

2. Methodology

The authors propose a novel task called Forgery Attribution Report Generation and a corresponding framework named ForgeryTalker.

A. The MMTT Dataset (Multi-Modal Tamper Tracing)

To support this task, the authors constructed the MMTT dataset, the first large-scale benchmark for this specific problem.

• Scale: 152,217 samples derived from 100,000 source images (CelebAMask-HQ and FFHQ).
• Manipulation Paradigms: The dataset covers four types of forgery:
  1. Face Swapping: Using GAN-based E4S to swap faces.
  2. Face Editing: Using StyleCLIP and HFGI for semantic attribute changes.
  3. Image Inpainting: Using transformer-based (MAT) and diffusion-based (SDXL) models for localized repairs.
• Annotations:
  • Ground Truth Masks: Programmatically derived from the forgery process to ensure pixel-perfect alignment.
  • Textual Descriptions: Human-authored reports (avg. 27.4 words) describing specific visual inconsistencies (e.g., "unnatural skin texture," "asymmetrical eyebrows") within the manipulated regions. The corpus contains over 4 million words.
  • Quality Control: A rigorous pipeline involving 30 annotators, minimum observation times, and strict constraints to prevent false positives.

B. ForgeryTalker Framework

ForgeryTalker is a unified, end-to-end multimodal framework based on InstructBLIP, designed to generate both a localization mask and a textual report simultaneously.

• Architecture:

  • Shared Encoder: A Vision Transformer (ViT) combined with a Q-Former to extract forgery-aware multimodal features.
  • Forgery Prompter Network (FPN): A specialized module (using ViT + Convolutional branches with Coordinate Convolution) that predicts salient manipulated regions (e.g., "left eye," "nose"). These predictions act as region prompts to guide the downstream decoders.
  • Dual Decoders:
    1. Mask Decoder: Based on SAM's Two-way Transformer, it generates the binary forgery mask.
    2. Large Language Model (LLM): A T5-based decoder that generates the attribution report, conditioned on the image features and the region prompts from the FPN.

• Training Strategy (Two Stages):

  1. Forgery-aware Pretraining: Jointly optimizes the encoder and decoders using four losses:
     • Masked Language Modeling (Lmlm): Aligns visual and linguistic tokens.
     • Language Modeling (Llm): Standard text generation.
     • Forgery Localization (Lseg): Pixel-wise cross-entropy for mask prediction.
     • Cross-modal Alignment (Lcon): Contrastive learning to align image and text embeddings.
  2. Attribution Report Generation: The FPN is trained first to generate accurate region prompts. Then, the FPN is frozen, and the Q-Former and decoders are fine-tuned to improve localization and report coherence.

3. Key Contributions

1. New Task Definition: Introduced Forgery Attribution Report Generation, shifting the paradigm from simple "verdicts" to interpretable "evidence" (answering Where and Why).
2. MMTT Dataset: Released the first large-scale dataset (152k samples) with high-precision process-derived masks and rich, human-written textual explanations.
3. ForgeryTalker Baseline: Proposed a unified framework that effectively fuses visual and semantic features via a shared encoder and dual decoders, utilizing a Forgery Prompter Network to enhance cross-modal reasoning.
4. Comprehensive Benchmarking: Established a new baseline for explainable multimedia forensics, demonstrating that localization and explanation tasks are complementary.

4. Experimental Results

The authors evaluated ForgeryTalker against state-of-the-art models (e.g., InstructBLIP, LISA, SCA, FakeShield) on the MMTT dataset and zero-shot on DQ_F++ and SynthScars.

• Report Generation:
  • Achieved a CIDEr score of 59.3, significantly outperforming the next best baseline (InstructBLIP at 51.7).
  • Led in all BLEU and ROUGE-L metrics.
  • In zero-shot tests on DQ_F++, it achieved a dominant CIDEr of 113.3, proving strong generalization.
• Forgery Localization:
  • Achieved the highest IoU (73.67) and Precision (91.43) among all compared models.
  • While some models had higher Recall, they suffered from lower Precision (over-segmentation). ForgeryTalker provided the most precise boundaries.
• Ablation Studies:
  • FPN Importance: Removing the Forgery Prompter Network caused a massive drop in CIDEr (from 59.3 to 51.7), confirming that region prompts are critical for coherent report generation.
  • Pretraining: The forgery-aware pretraining stage improved IoU by ~7.8% and CIDEr by ~4.9% compared to training from scratch.
  • Data Diversity: Each manipulation type (Swapping, Editing, Inpainting) contributed uniquely; removing any type degraded performance significantly, highlighting the need for diverse training data.

5. Significance

• Enhanced Trustworthiness: By providing natural language explanations grounded in visual evidence, the system helps human reviewers understand why an image is fake, increasing trust in automated detection systems.
• Interpretability: Moves the field beyond "black box" detection to "explainable forensics," addressing the critical gap in verifying subtle deepfake artifacts.
• Foundation for Future Research: The release of the MMTT dataset and the ForgeryTalker code establishes a standardized benchmark for developing multimodal forensic tools, encouraging research into the intersection of computer vision and natural language processing for security applications.

In summary, this paper addresses the interpretability crisis in deepfake detection by introducing a dataset and a model that not only locate manipulations but also articulate the specific visual anomalies that constitute the forgery.