Interpretable-Aware Privacy Preserving Framework for Deepfake Detection using Federated ResNet and Explainable AI Techniques

Technical Summary: Interpretable-Aware Privacy Preserving Framework for Deepfake Detection using Federated ResNet and Explainable AI Techniques

Problem Statement
The rapid advancement of generative artificial intelligence has facilitated the creation of highly realistic deepfake images and videos, posing significant threats to digital security, personal privacy, and societal trust. While deep learning (DL) has improved detection capabilities, traditional approaches rely on centralized architectures that require aggregating large volumes of sensitive image data on a single server. This centralization creates vulnerabilities regarding data privacy, ownership, and regulatory compliance. Furthermore, existing detection systems often lack transparency, operating as "black boxes" that provide classification outputs without explaining the reasoning behind their decisions. The challenge lies in developing a detection system that simultaneously ensures data privacy, handles heterogeneous data distributions across devices, and provides interpretable insights into its predictions.

Methodology
The proposed framework addresses these challenges by integrating Federated Learning (FL), a lightweight Convolutional Neural Network (CNN), and Explainable AI (XAI) techniques.

1. Federated Learning Architecture: Instead of centralizing raw data, the system employs a distributed training approach where models are trained locally on client devices. Only model updates (weights) are transmitted to a central server for aggregation, ensuring that sensitive image data never leaves the local environment.
2. Model Architecture (ResNet-18): The core detection model utilizes ResNet-18. This architecture was selected for its balance between computational efficiency and feature extraction capability. Residual connections allow the model to train steadily and capture subtle visual artifacts (e.g., unnatural blending, texture inconsistencies) necessary for deepfake detection without the excessive computational overhead of deeper networks.
3. Optimization Strategy (FedProx): To address the issue of non-identically distributed (Non-IID) data across clients, which can cause model divergence and instability, the framework incorporates the FedProx algorithm. FedProx modifies the local loss function by adding a proximal term that constrains local model updates to remain close to the global model parameters. This regularization enhances convergence stability in heterogeneous environments.
4. Explainable Decision Generation: To improve transparency, the framework includes a forensic analysis module. During inference, the system generates predictions accompanied by visual explanations. This involves:
   • Edge Detection: Computing gradient magnitudes to highlight structural inconsistencies and sharp transitions often found in manipulated regions.
   • Residual Noise Analysis: Subtracting a smoothed (Gaussian-blurred) version of the image from the original to isolate high-frequency noise patterns and synthetic artifacts invisible to the human eye.
   • Visualization: Combining edge and noise features into an explanation map overlaid on the original image to visually justify the classification.

Key Contributions

• Privacy-Preserving Framework: The study introduces a unified system that enables collaborative deepfake detection without requiring the sharing of raw image data, thereby addressing critical privacy and regulatory concerns.
• Stable Federated Training: By integrating FedProx with ResNet-18, the work demonstrates a method to maintain training stability and model convergence despite the challenges of Non-IID data distributions common in real-world federated settings.
• Interpretability: The framework moves beyond simple classification by incorporating forensic feature analysis (edge and residual maps), providing users with visual evidence of why an image is classified as real or fake.
• Efficient Architecture: The selection of ResNet-18 ensures the system remains computationally feasible for distributed environments while effectively capturing deepfake-specific features.

Experimental Results
The framework was evaluated using the FaceForensics++ dataset, a standard benchmark containing real and manipulated videos across four manipulation types: DeepFakes, Face2Face, FaceSwap, and Neural Textures. The experiments were conducted on an NVIDIA GeForce 4 GB GPU with a learning rate of 0.001 and a batch size of 48.

The quantitative results indicate high performance:

• Accuracy: 97.20%
• Precision: 96.30%
• Recall: 96.72%
• F1-Score: 96.90%

The confusion matrix analysis revealed that the model correctly classified the majority of samples, with errors concentrated in only a few specific classes. The precision analysis (visualized via radar plots) showed consistent performance across different manipulation types, indicating the model is not biased toward specific classes. The training process required approximately 45 minutes, with an average inference time of 12 seconds per unseen image.

Significance and Claims
The authors claim that this study demonstrates a viable path toward secure, privacy-preserving, and interpretable deepfake detection. By unifying Federated Learning, CNNs, and explainability, the framework enhances transparency and user trust, which are essential for real-world deployment. The work asserts that it successfully balances the need for high detection accuracy with the constraints of data privacy and computational efficiency, making it suitable for environments where data cannot be centrally shared. The integration of FedProx is highlighted as a critical factor in ensuring the robustness of the system against data heterogeneity, while the forensic visualization component addresses the growing need for transparency in automated decision-making systems.