Technical Summary: EMO-BOOST

Problem Statement

The rapid advancement of generative AI has intensified the challenge of deepfake detection. Current forensic methods face a critical limitation: the inability to generalize to unseen manipulation techniques. Most existing deepfake detectors rely on low-level spatial, temporal, or spectral features extracted from raw data. While effective against specific known forgeries, these models struggle when encountering new generation methods not present in the training set. Furthermore, high-level semantic cues, particularly affective information (emotion), remain underexplored in this domain. The authors hypothesize that deepfake generation techniques may struggle to convincingly replicate natural human emotional expressivity, and that inconsistencies in these high-level signals could provide robust cues for detecting unseen manipulations.

Methodology

The paper proposes Emo-Boost, a multimodal framework designed to enhance existing low-level deepfake detectors by integrating high-level emotion-based signals. The system consists of two primary components:

1. EmoForensics (The Emotion-Based Detector)

EmoForensics is a novel deepfake detection network that models emotion representations from audio and visual streams. It operates on the hypothesis that deepfakes disrupt both intra-modal (within a single modality) and inter-modal (between audio and video) temporal consistency in emotional expression.

• Feature Extraction: The framework utilizes frozen, pre-trained emotion encoders to extract frame-level embeddings without retraining on deepfake data, thereby avoiding overfitting to manipulation-induced distribution shifts.
  • Visual: Uses POSTER [51] for facial emotion recognition.
  • Audio: Uses Emotion2vec [29] for speech emotion recognition.
• Temporal Modeling:
  • Intra-Modal Consistency: Frame-level embeddings are processed by modality-specific Temporal Transformers (based on MMA-DFER [9]) to capture temporal dependencies within the visual and audio streams independently.
  • Inter-Modal Consistency: The model enforces consistency between the visual and audio emotion representations.
• Fusion and Classification: The modality-level representations ($h_v$ and $h_a$) are fused via element-wise addition to form a joint representation ($f_e$).
• Learning Objective: The network is trained using a combination of:
  • Binary Cross-Entropy (BCE): For standard real/fake classification.
  • Contrastive Loss: To encourage embeddings from real audio-visual pairs to be close in the embedding space while pushing embeddings from manipulated pairs (e.g., real video + fake audio) further apart.

2. Emo-Boost (The Integration Framework)

Emo-Boost acts as a fusion mechanism that augments an off-the-shelf, low-level focused deepfake detector (denoted as $D$, specifically SIMBA [19] in experiments) with the high-level signals from EmoForensics.

• Architecture: Both the pre-trained low-level detector $D$ and the trained EmoForensics network are kept frozen during the Emo-Boost training phase.
• Fusion Strategy: The emotion representation from EmoForensics is projected to match the dimensionality of the detector's features. The two feature sets are then combined using element-wise multiplication ($f_{fused} = \tilde{f}_e \odot f_D$).
• Training: Only the projection head and the final classification head are trainable. This design ensures that Emo-Boost leverages the complementary nature of the two signals without requiring joint retraining of the base detectors.

Key Contributions

1. Emo-Boost Framework: A multimodal detection system that integrates an emotion-based detector with off-the-shelf RGB and acoustic detectors via simple late-stage fusion.
2. EmoForensics Network: A specialized detector modeling two complementary dimensions: (i) intra-modal temporal consistency of emotion representations, and (ii) inter-modal temporal consistency between audio and visual streams.
3. Generalization Improvement: The framework demonstrates that high-level semantic cues can support low-level approaches in generalizing to unseen manipulation types, specifically improving cross-manipulation performance.

Experimental Results

The authors evaluated their approach on two datasets: FakeAVCeleb (21k videos, 500 identities, 4 manipulation types) and DeepSpeak v2 (16.5k videos, 280 identities, 10 manipulation types).

In-Domain Evaluation

On standard 70-30 train-test splits where manipulation types overlap between training and testing:

• Emo-Boosted SIMBA remains competitive with the state-of-the-art (SOTA) SIMBA baseline.
• On FakeAVCeleb, Emo-Boosted SIMBA achieved an AUC of 99.89% (vs. 99.90% for SIMBA).
• On DeepSpeak v2, it achieved 99.60% AUC (vs. 99.79% for SIMBA).
• Observation: The addition of emotion signals did not degrade performance in scenarios where low-level detectors already performed near-perfectly.

Cross-Manipulation (Leave-One-Out) Evaluation

This setting tests generalization to manipulation types not seen during training.

• FakeAVCeleb: Emo-Boosted SIMBA achieved a new SOTA average AUC of 95.30%, representing a 2.1% improvement over the standalone SIMBA baseline (93.17%).
  • Significant gains were observed on specific splits, such as Faceswap (84.47% vs. 80.87%) and RealTime Voice Cloning (100.0% vs. 89.97%).
  • The authors attribute this to the complementary nature of the signals: where the low-level detector fails to capture an artifact, the emotion-based detector succeeds.
• DeepSpeak v2: The average AUC remained competitive (95.26% vs. 95.30% for SIMBA) but did not show significant improvement.
• Stability Analysis: An ablation study (Figure 4) revealed that EmoForensics exhibits significantly lower performance variability across different manipulation splits compared to SIMBA (Area under the curve of differences: 12.50 vs. 32.98), suggesting that emotion-based representations offer more stable cues for generalization.

Ablation Studies

• Components: Removing temporal transformers or contrastive loss resulted in performance drops (approx. 2% AUC), confirming the importance of modeling both intra-modal and inter-modal temporal consistency.
• Fusion Strategy: Element-wise multiplication ($\odot$) consistently outperformed element-wise addition ($\oplus$) and concatenation ($|$) in the cross-manipulation setting.

Significance and Claims

The paper claims that Emo-Boost successfully addresses the generalization gap in deepfake detection by leveraging high-level semantic cues (emotion) to complement low-level artifact detection.

• Complementarity: The core finding is that low-level detectors and emotion-based detectors capture different signals. Combining them allows the system to detect manipulations that one modality alone might miss.
• Robustness: The framework demonstrates that frozen, pre-trained emotion encoders can be effectively repurposed for forensic tasks without retraining on deepfake data, mitigating distribution shift issues.
• Modest Limitations: The authors acknowledge that EmoForensics performs weaker as a standalone detector compared to low-level methods. Furthermore, the lack of improvement on DeepSpeak v2 is attributed to the dataset's scripted nature, which limits natural emotional expressivity, thereby reducing the efficacy of emotion-based cues.
• Future Direction: The work suggests that other high-level semantic cues (e.g., mimicry, movement) could be similarly combined with pixel-level detectors to further enhance generalization.