Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

1. Problem Statement

The rapid advancement of audio-visual deepfake generation (e.g., FaceSwap, Wav2Lip, FSGAN) has rendered perceptual detection unreliable. While multimodal detection (combining audio and video) has shown promise, current approaches largely treat the problem as a binary classification task (Real vs. Fake).

• The Core Issue: Binary-only optimization encourages models to learn "shortcuts" or dataset-specific artifacts (e.g., compression patterns, background noise) rather than the underlying generative traces (forensic fingerprints) that distinguish specific manipulation methods.
• The Consequence: Models trained this way often fail to generalize to unseen generators or datasets because they have not learned the causal features of forgery.
• The Hypothesis: If a detector is forced to answer the harder question of "Which generator produced this?" (attribution), it will inherently learn a more robust, forensically meaningful representation that also solves the easier question of "Is this fake?"

2. Methodology: The AMDD Framework

The authors propose the Attribution-Guided Multimodal Deepfake Detection (AMDD) framework. The core innovation is treating generator attribution not as a separate end-task, but as a structured regularizer within a shared embedding space.

A. Architecture

The framework utilizes a dual-stream encoder with balanced capacity to prevent the audio signal from being underutilized (a common flaw in prior work):

• Visual Encoder: A ResNet50 backbone with a Temporal Attention module (Multi-Head Attention) to capture frame-level artifacts and temporal inconsistencies.
• Audio Encoder: A pretrained ResNet18 adapted for mel spectrograms. The authors deliberately chose a deep architecture (11.4M params) to match the visual encoder's capacity (25.6M params), closing the "encoder capacity gap" found in previous multimodal detectors.
• Cross-Modal Attention: A bidirectional attention module aligns visual and audio embeddings, allowing the model to identify correlated artifacts introduced by a specific generator across both modalities.
• Heads: The fused embedding is passed to two heads:
  1. Detection Head: Binary classification (Real/Fake).
  2. Attribution Head: Multi-class classification identifying the specific generator (or Real).

B. Loss Functions

The model is jointly optimized using five loss components:

1. Detection Loss ($L_{det}$): Focal loss for binary classification to handle class imbalance.
2. Attribution Loss ($L_{attr}$): Cross-entropy loss over generator classes. This acts as the primary regularizer, forcing the embedding space to separate samples by their source generator.
3. Cross-Modal Contrastive Loss ($L_{cont}$): Aligns visual and audio projections of the same video sample.
4. Cross-Modal Forensic Fingerprint Consistency ($L_{fp}$): Key Novelty. This loss enforces alignment between the visual and audio representations of samples from the same generator. It exploits the physical coupling between speech production and facial articulation; synthetic pipelines often disrupt this coherence, while authentic recordings maintain it.
5. Centroid Regularization ($L_{cen}$): Pulls embeddings toward their generator-specific centroids (updated via Exponential Moving Average) to stabilize the geometry of the embedding space.

3. Key Contributions

1. Attribution-Guided Learning: The paper demonstrates that using attribution supervision as a regularizer shapes the representation geometry toward forensically meaningful features, even when binary detection accuracy is already high.
2. Cross-Modal Forensic Fingerprint Consistency (CMFFC): A novel loss function that explicitly enforces alignment between audio and visual artifacts for the same generator, leveraging the physical constraints of human speech and articulation.
3. Balanced Encoder Design: The authors address the "audio underutilization" problem in multimodal detection by pairing a deep ResNet50 visual encoder with a comparably deep ResNet18 audio encoder, rather than using shallow CNNs for audio.
4. Comprehensive Analysis: The study provides a rigorous breakdown of where generalization succeeds (real video detection) and fails (fake detection on unseen generators), offering a mechanistic explanation based on domain adaptation theory.

4. Experimental Results

The framework was evaluated primarily on FakeAVCeleb (containing FaceSwap, Wav2Lip, and FSGAN) and cross-dataset tested on DeepfakeTIMIT, DFDM, and LAV-DF.

• In-Domain Performance (FakeAVCeleb):

  • Balanced Accuracy: 99.7%
  • AUC: 99.8%
  • Attribution Accuracy: 95.9%
  • Per-Generator Detection: Near-perfect (100%) for Wav2Lip and FSGAN; 98.2% for FaceSwap.

• Ablation Studies:

  • Removing the attribution loss caused attribution accuracy to collapse to 11.0% (below random chance for 4 classes), while binary detection remained high (~98.3%).
  • Interpretation: This confirms that without attribution guidance, the model learns shallow shortcuts (dataset artifacts) rather than genuine forensic fingerprints. The attribution task forces the model to learn the "hard" features necessary for robust detection.

• Cross-Dataset Generalization:

  • Real Video Detection: Generalizes robustly (>92% accuracy) across unseen datasets, suggesting the model learns natural temporal statistics and audio-visual coherence.
  • Fake Video Detection: Performance drops significantly on unseen generators (e.g., dropping to ~50% on DFDM). This confirms that fingerprint-based methods struggle with domain shift when generators use fundamentally different blending pipelines.

• Visualization:

  • t-SNE: Shows that the full AMDD model creates distinct, compact clusters for each generator, whereas binary-only models collapse all fakes into a single unstructured region.
  • GradCAM: The model correctly focuses on facial boundaries (mouth, eyes, chin) where blending artifacts occur, rather than background noise.

5. Significance and Conclusion

• Paradigm Shift: The paper argues that attribution is a stronger geometric constraint than binary detection. By forcing the model to distinguish how a video was forged, it naturally learns to distinguish if it was forged.
• Forensic Interpretability: The resulting embeddings are not just "fake" or "real" but are organized by the specific manipulation method, providing actionable intelligence for forensic investigators.
• Honest Limitations: The authors transparently acknowledge that while real-video detection generalizes well, detecting fakes from unseen generators remains an open challenge. They attribute this to the fundamental nature of fingerprint-based forensics rather than a failure of their specific architecture.
• Practical Impact: AMDD achieves state-of-the-art performance without relying on massive pre-trained audio-visual models, making it a lightweight yet highly effective solution for deepfake detection and attribution.