Technical Summary: Ensemble Deep Learning Approaches for AI-Altered Video Detection

Problem Statement

The rapid proliferation of artificial intelligence tools has led to an exponential increase in AI-generated and manipulated videos (deepfakes), creating a significant challenge in distinguishing authentic content from synthetic media. While existing detection methods often rely on single models, they frequently struggle to generalize across diverse deepfake generation techniques and datasets. The authors identify a critical gap: individual models tend to overfit to specific training data, leading to performance degradation when tested on "in-the-wild" or unseen manipulation types. Furthermore, the lack of robust, multimodal detection systems leaves social media platforms vulnerable to misinformation campaigns and identity manipulation.

Methodology

The proposed solution is a multimodal deepfake detection pipeline that integrates an ensemble of four specialized models to analyze both audio and visual components of a video.

System Architecture

The pipeline processes input videos through the following stages:

1. Preprocessing: The video is separated into audio and visual streams. Faces are extracted from video frames using MTCNN (Multi-task Cascaded Convolutional Neural Networks).
2. Visual Analysis: Three Convolutional Neural Network (CNN) architectures analyze the extracted face frames:
   • EfficientNet-B1: Utilizes compound scaling and Mobile Inverted Bottleneck Convolutions (MBConv) to detect spatial features. It is preprocessed to 240x240 pixels with ImageNet normalization.
   • XceptionNet: Employs depthwise separable convolutions to encode manipulation artifacts into linear feature vectors. It processes frames resized to 299x299 pixels.
   • MesoNet (Meso4): Focuses on mesoscopic facial inconsistencies. It was re-implemented in a legacy environment to maintain fidelity to its original 2018 architecture, processing 256x256 pixel crops.
3. Audio Analysis:
   • AASIST (Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks): Analyzes raw audio waveforms (resampled to 16 kHz mono) to detect synthetic speech patterns and voice conversion artifacts using spectro-temporal graph attention layers.
4. Ensemble Fusion: The system aggregates the probability scores from the four models using several strategies:
   • Mean Fusion: Simple averaging of scores (used as a baseline).
   • Stacking: A meta-model learns to combine the outputs of the base models (AASIST, EfficientNet, MesoNet, XceptionNet) to optimize the final prediction.
   • Voting Strategies: Includes majority voting and weighted voting, where models cast binary labels.
   • Weighted Average: Assigns specific importance weights to each model.
   • Fallback Mechanism: If the stacking meta-model is unavailable, the system defaults to mean fusion to ensure pipeline reliability.

Datasets

• Training: Visual models were trained on AIGVDBench (general AI-generated videos) and FaceForensics++ (specific face-swapping techniques). The audio model (AASIST) was trained on the ASVspoof 2019 Logical Access dataset.
• Testing: The ensemble was evaluated on FakeAVCeleb, a dataset containing permutations of real/fake audio and video (e.g., Real Video/Fake Audio, Fake Video/Real Audio, etc.), allowing for the assessment of multimodal generalization.

Key Contributions

1. Multimodal Ensemble System: The authors present a unified detection framework combining audio (AASIST) and visual (EfficientNet, XceptionNet, MesoNet) analysis, comparing five distinct fusion strategies.
2. Generalization Analysis: The study demonstrates that while audio anti-spoofing models (AASIST) generalize poorly to in-the-wild multimodal data, video-based models remain comparatively robust.
3. Modality Dependency: The research analyzes how ensemble performance varies across manipulation types, revealing that the system relies heavily on visual modalities, with the audio modality contributing minimally to the final decision in many scenarios.
4. Fusion Strategy Comparison: The paper evaluates the trade-offs between simple baselines (mean fusion) and complex learning-based approaches (stacking), showing that voting-based methods can reduce noise but may lose confidence information.

Results

• Individual Model Performance:
  • AASIST: Achieved high accuracy (99.16%) on its specific benchmark (ASVspoof) but dropped to near-random performance (~50%) on the FakeAVCeleb dataset when real audio was present, indicating a significant domain gap.
  • Visual Models: XceptionNet and MesoNet showed better generalization, achieving accuracies between 70% and 80% on the diverse FakeAVCeleb test sets. EfficientNet struggled more, often hovering around 50% accuracy on unseen data.
• Ensemble Performance:
  • The ensemble improved overall consistency compared to individual models.
  • Weighted Voting and Majority Voting yielded the highest accuracy (72.8% and 71.2% respectively) on the full FakeAVCeleb test set, outperforming mean fusion (61.6%) and stacking (71.0%).
  • On a balanced test set containing only Fake Video/Fake Audio (FvFa) and Real Video/Real Audio (RvRa), the ensemble achieved 84.8% accuracy.
  • However, when tested on samples with Real Video and Fake Audio (RvFa), the ensemble accuracy dropped to 51.4%, confirming that the system's performance is driven primarily by visual artifacts rather than audio cues.
• Overall Accuracy: The system achieved an average accuracy of approximately 70% across diverse datasets, highlighting the persistent challenge of generalization.

Significance and Claims

The paper posits that while ensemble methods improve robustness and consistency across different deepfake types, they do not fully solve the generalization problem. The authors claim that:

• Multimodal integration is necessary but insufficient: Combining audio and visual data provides more information than single-modality approaches, but current audio models fail to generalize to real-world scenarios, limiting the system's effectiveness.
• Visual dominance: The ensemble is currently driven by visual models; the audio modality often acts as a neutral fallback rather than a decisive factor.
• The Generalization Gap: The central open challenge remains the ability to detect unseen manipulations. The authors note that models often learn dataset-specific artifacts rather than universal deepfake features.
• Modest Improvement: The paper modestly concludes that while ensembles offer a more reliable approach than single models, the improvement over the strongest individual component (XceptionNet/MesoNet) is relatively small, suggesting that future work must focus on improving individual model generalization and domain adaptation (e.g., via Domain Adaptive Batch Normalization) rather than solely relying on fusion strategies.