Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities

1. Problem Statement

The rapid evolution of generative AI has shifted deepfake attacks from single-modality manipulations (e.g., fake images) to complex, coordinated multimodal deceptions (e.g., synchronized fake video, audio, and text). Current forensic detection methods suffer from a critical "modality-binding" bottleneck:

• Overfitting to Surface Artifacts: Existing models rely heavily on modality-specific physical fingerprints (e.g., spatial blending in images or frequency anomalies in audio).
• Lack of Generalization: When confronted with unseen "dark modalities" (e.g., infrared, depth streams, or entirely new sensor types not seen during training), these models suffer catastrophic performance degradation because they cannot decouple the forgery essence from the specific physical representation.
• Semantic Masking: Even when using pre-trained multimodal models (like ImageBind) to align modalities semantically, the macro-level semantic concepts often overwhelm the subtle, low-level statistical biases left by generative algorithms, preventing true forensic alignment.

2. Methodology: The MAF Framework

The authors propose a paradigm shift from modality-binding feature fusion to modality generalization. They introduce the Modality-Agnostic Forgery (MAF) framework, which aims to extract a shared, latent forgery knowledge ($E$) that transcends specific media types, while stripping away modality-specific style noise ($S$).

Core Concepts

• Feature Disentanglement: The framework hypothesizes that any forged signal $x$ consists of a modality-specific style $S$ and a modality-invariant essence $E$. The goal is to learn a universal detector $F$ that relies solely on $E$.
• Two Evaluation Paradigms:
  1. Weak MAF (Semantic Generalization): The test modality can be mapped into a unified semantic space using existing pre-trained perceptors (e.g., ImageBind). The challenge is to filter out the "forgery gap" caused by varying generative mechanisms within this shared space.
  2. Strong MAF ("Dark Modality" Challenge): The test modality is completely isolated—physically and semantically disconnected from training data. No pre-trained semantic alignment exists. The framework must use an independent, self-supervised encoder to extract features and feed them into the frozen universal detector.

Technical Architecture

• Perceptors ($\Phi$): Modality-specific encoders map raw inputs to feature vectors. In Weak MAF, these are frozen pre-trained models; in Strong MAF, they are initialized via self-supervised learning (e.g., contrastive learning).
• Universal Detector ($F$): A lightweight, shared Multi-Layer Perceptron (MLP) that processes features from all modalities.
• Optimization Strategy:
  • Loss Function: Combines standard classification loss ($L_{cls}$) with a Domain Generalization (DG) regularization term ($L_{DG}$).
  • Mechanism: The DG term treats different physical modalities as distinct "domains." It forces the model to minimize the mutual information between the feature representation and the modality style ($I(Z; S)$) while maximizing the information regarding the forgery label ($I(Z; Y)$). This effectively pushes the feature distributions of different modalities to converge into a unified "forensic space."

3. Key Contributions

1. Paradigm Shift: Redefines multimodal forensics from "feature fusion" to "modality generalization," proposing the existence of Shared Latent Forgery Knowledge.
2. MAF Framework & Benchmarks:
   • Introduces the first Modality-Agnostic Forgery (MAF) detection framework capable of source-free domain generalization.
   • Defines rigorous Weak MAF and Strong MAF evaluation settings to test generalization against semantically correlated and completely isolated modalities.
   • Releases DeepModal-Bench, the first comprehensive benchmark for this task, integrating diverse datasets (LAV-DF, FakeAVCeleb, ASVspoof5, Celeb-DF++) and state-of-the-art baselines.
3. Empirical Validation of Universal Traces: Provides evidence that generative algorithms leave invariant statistical biases across all media types, which can be captured without retraining on the target modality.

4. Experimental Results

The authors evaluated the MAF framework against 13 baseline algorithms (including Multimodal Learning methods like OGM and Domain Generalization methods like IRM, Mixup, and URM) across three datasets.

• Performance Superiority:
  • Weak MAF: Domain Generalization (DG) methods within the MAF framework consistently outperformed Multimodal Learning (MML) fusion methods by a significant margin (e.g., +5-10% AUC). This proves that extracting invariant features is superior to simple fusion.
  • Strong MAF: Even under extreme isolation (no semantic alignment), the MAF framework maintained robust performance, significantly outperforming random guessing and demonstrating the existence of universal forgery traces.
• Key Findings:
  • Semantic Masking: Pre-trained semantic aligners (like LanguageBind) often underperform compared to ImageBind because they prioritize high-level concepts, inadvertently suppressing low-level forensic artifacts.
  • Dimensionality Collapse: Principal Component Analysis (PCA) revealed that the effective intrinsic dimensionality of features drops drastically (from >100 in semantic space to 2–50 in forensic space). This confirms that the framework successfully strips away high-rank style noise, leaving a compact, low-dimensional manifold of universal forgery knowledge.
  • Neural Co-activation: Analysis of the detector's neurons showed a core set of 16 neurons consistently activated across all modalities, representing the universal forgery essence, while peripheral neurons handled modality-specific styles.

5. Significance and Impact

• Proactive Defense: The MAF framework moves forensic detection from a reactive "arms race" (collecting data for every new fake medium) to a proactive defense. It enables a "train once, generalize to all" capability.
• Dark Modality Resilience: It offers a solution for detecting forgeries in restricted or emerging domains (e.g., medical imaging, infrared sensors) where labeled training data is unavailable or impossible to obtain.
• Theoretical Insight: The work establishes that deepfake artifacts are not merely transient surface anomalies but are immutable logical flaws inherited from the generative mechanism itself. By targeting these fundamental algorithmic biases, the framework future-proofs multimedia security against the asymmetric influx of novel AI threats.

In summary, this paper provides a foundational shift in deepfake detection, proving that universal, modality-agnostic forensic knowledge exists and can be extracted through rigorous feature disentanglement and domain generalization strategies.