Hybrid CNN-ViT-BiLSTM Framework for Robust Deepfake Video Detection

Technical Summary: Hybrid CNN-ViT-BiLSTM Framework for Robust Deepfake Video Detection

Problem Statement
The rapid advancement of Generative Adversarial Networks (GANs) and deep learning has enabled the creation of highly convincing deepfake videos, posing significant threats to digital media security, identity integrity, and social trust. While existing detection methods utilizing Convolutional Neural Networks (CNNs), transformers, and temporal sequence analysis have shown promise, they often suffer from critical limitations. Many approaches rely heavily on static spatial features, overlooking the temporal inconsistencies crucial for detecting manipulations across entire video sequences. Furthermore, models trained on single datasets often fail to generalize to unseen manipulation techniques or operational environments due to domain shifts, compression artifacts, and variations in video quality. Current ensemble methods frequently lack deep feature-level fusion or fail to integrate spatial and temporal reasoning within a unified, modular pipeline, resulting in suboptimal robustness in real-world scenarios.

Methodology
To address these challenges, the authors propose a modular, end-to-end ensemble framework that unifies complementary spatial and temporal representations. The architecture operates through the following stages:

1. Data Preprocessing and Sampling:

   • The system utilizes face-cropped videos from the DeepFake Detection (DFD) and FaceForensics++ (FF++) datasets.
   • Frames are sampled at fixed intervals (every 20th frame) to ensure temporal coverage while maintaining efficiency.
   • Up to 50 face crops are extracted per video using the RetinaFace detector, resized to specific dimensions (224×224 for ResNet-50 and ViT; 299×299 for XceptionNet), and normalized.

2. Multi-Branch Spatial Feature Extraction:
   The framework employs three distinct backbone networks to capture diverse spatial characteristics:

   • ResNet-50: Extracts hierarchical spatial features, outputting a 2048-dimensional vector via average pooling.
   • XceptionNet: Utilizes depthwise separable convolutions to detect fine-grained local texture anomalies and high-frequency artifacts, outputting a 2048-dimensional vector.
   • Vision Transformer (ViT): Models global contextual dependencies and long-range spatial relationships using self-attention mechanisms, outputting a 768-dimensional vector from the [CLS] token.

3. Feature-Level Fusion:
   The outputs from the three backbones are concatenated and L2-normalized to form a unified 4864-dimensional feature vector for each face crop. This strategy integrates local CNN details with global transformer context prior to temporal modeling.

4. Temporal Sequence Modeling:

   • The fused spatial features are grouped into fixed-length sequences of 5 consecutive face crops.
   • A Bidirectional Long Short-Term Memory (BiLSTM) network processes these sequences to capture both forward and backward temporal dependencies, such as motion irregularities, blinking patterns, and inter-frame inconsistencies.
   • The BiLSTM outputs a 512-dimensional temporal embedding representing the video segment.

5. Classification:
   The temporal embeddings are aggregated (via average pooling) to form a video-level representation, which is passed through a fully connected layer with a sigmoid activation function to produce a binary classification (Real vs. Fake). The model is trained using the Adam optimizer and binary cross-entropy loss.

Key Contributions

• Novel Modular Architecture: The paper introduces a unified framework that integrates ResNet-50, XceptionNet, and ViT for comprehensive spatial analysis, coupled with BiLSTM for temporal dependency learning.
• Feature-Level Fusion Strategy: Unlike prior ensemble methods that rely on late or static fusion, this approach performs feature-level fusion before temporal modeling, enabling richer spatio-temporal representations.
• Explicit Temporal Reasoning: The inclusion of BiLSTM significantly enhances robustness against compression artifacts and diverse forgery techniques by modeling motion and temporal inconsistencies.
• Cross-Dataset Generalization: The framework is jointly evaluated on DFD and FF++ datasets, demonstrating improved generalization capabilities across different manipulation techniques and data sources.
• Scalability: The modular design allows for the easy integration of additional backbone models or temporal modules in future extensions.

Experimental Results
The proposed method was evaluated on a combined test set of DFD and FF++ datasets using an NVIDIA A100 GPU. The results indicate superior performance compared to individual backbones and existing state-of-the-art (SOTA) methods:

• Accuracy: 91.07%
• AUC (Area Under the Curve): 0.971
• Macro F1-Score: 0.91
• Average Precision: 0.9646

Ablation studies confirmed that each component contributes meaningfully to the final performance. The fusion of ResNet-50 and XceptionNet improved AUC to 0.95, while the addition of BiLSTM raised it to 0.97. The final integration of ViT achieved the peak AUC of 0.971. The model outperformed individual baselines (e.g., XceptionNet at 88% AUC, ViT at 90% AUC) and matched or exceeded several recent hybrid and ensemble SOTA models.

Significance and Claims
The authors claim that this work establishes a new benchmark for practical deepfake detection by achieving robust performance across different data sources and forgery techniques. The significance of the framework lies in its ability to simultaneously exploit spatial, semantic, and temporal cues, addressing the limitations of methods that focus solely on static features or lack deep integration of diverse architectures. The paper posits that the proposed spatial-temporal ensemble is particularly effective in cross-dataset scenarios, offering a more reliable solution for real-world forensic applications. While acknowledging computational complexity as a limitation for real-time edge deployment, the authors emphasize the framework's effectiveness for scalable forensic analysis and its potential as a foundation for future research in explainability and multimodal detection.