💻 computer science

Cross-Dataset Generalization in Deepfake Detection: A Controlled Study of Spatial, Frequency, and Video-Level Baselines

This controlled study evaluates the cross-dataset generalization of five ImageNet-pretrained RGB backbones and two lightweight probes for deepfake detection, revealing that source-domain coverage and evaluation protocols are as critical as backbone selection in addressing performance gaps across FaceForensics++, Celeb-DF, and DFD datasets.

Original authors: Samarth Kukreja

Published 2026-10-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samarth Kukreja

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern digital landscape, a new kind of visual trickery has emerged, one where artificial intelligence can swap faces or mimic expressions so convincingly that the human eye often cannot tell the difference from reality. These fabricated videos, known as deepfakes, pose a significant challenge for anyone trying to verify what is real and what is fake, whether in news reporting, legal evidence, or social media. For years, researchers have built computer programs designed to spot these fakes, but a persistent problem has remained: a program that performs perfectly on one set of test videos often fails miserably when shown a different set. This gap between doing well in a controlled lab and working in the messy real world is the central mystery this study seeks to solve. The researchers wanted to know if the failure was due to the specific way the computer was built, or if the problem lay deeper, in the very nature of the data and how the tests were conducted.

To investigate this, a researcher named Samarth Kukreja set up a highly controlled experiment to strip away the variables that usually confuse such comparisons. Instead of building a brand-new, complex detector from scratch, the study used a standard, reliable method for processing video: taking frames from a video, finding the faces, and cropping them out to be examined individually. Five different types of computer vision models, which act as the "eyes" of the system, were then tested using this exact same pipeline. These models ranged from older, established designs to newer, more efficient architectures, and they were all trained on the same data using the same rules. The goal was to see if one specific type of "eye" was inherently better at spotting fakes, or if the results were more dependent on the dataset being used. The experiments involved training on one collection of videos and then testing on others, including a completely separate dataset that the models had never seen before, to simulate the real-world scenario of encountering a new type of forgery.

The results revealed a nuanced picture that challenges the common assumption that a single, superior technology exists to solve the problem. When the models were tested on the same type of data they were trained on, one modern design called ConvNeXt-Tiny showed the highest scores, followed closely by a standard model known as ResNet-50. However, when the researchers applied strict statistical checks to see if this difference was truly meaningful, the gap between these two models disappeared; they performed essentially the same. More importantly, when the models were asked to identify fakes in a completely different dataset—a scenario that mimics real-world deployment—every single model saw its performance drop significantly. The drop was so severe in one specific direction, where a model trained on one dataset tried to detect fakes in another, that the system performed barely better than random guessing. This suggests that the choice of the computer model matters far less than the alignment between the training data and the testing data.

The study also explored two simpler, low-cost ideas to see if they could boost performance. One approach looked only at the frequency patterns in the images, similar to how a radio picks up specific signals, while the other tried to combine the scores of individual frames to make a decision about the whole video. Neither of these add-ons changed the fundamental outcome. The frequency-based model showed some ability to transfer to new data, but it still lagged behind the main models. The method of combining scores did not consistently improve the results. Furthermore, when the researchers tested how these systems handled common image problems like blurring, noise, or compression, the detectors became much less reliable. This indicates that the current systems are fragile and easily confused by the kinds of image degradation that happen naturally when videos are shared online.

Ultimately, the paper concludes that the search for a single, perfect deepfake detector is likely a complex challenge. The performance of these systems is not just about which neural network architecture is chosen; it is heavily influenced by the specific videos used for training and testing, and how strictly the data is separated to prevent the computer from memorizing specific faces. The study found that while some models appear slightly better on paper, the differences are often not statistically significant, and all models struggle when faced with data they were not trained on. The most critical takeaway is that the reliability of a deepfake detector depends more on the quality and diversity of the data it learns from and the rigor of the testing process than on the specific brand of computer vision model used. Until these systems can handle the vast variety of real-world conditions without failing, the gap between laboratory success and practical utility will remain wide.

Technical Summary: Cross-Dataset Generalization in Deepfake Detection

Problem Statement
Deepfake detectors often achieve high accuracy on in-distribution benchmarks but fail to generalize when tested on videos from different datasets, manipulated by different methods, or degraded by common image corruptions. While previous surveys have identified the gap between in-distribution scores and cross-dataset reliability, this study focuses on a specific, controlled question: when preprocessing, evaluation protocols, and training recipes are fixed, how much does the choice of image backbone alone influence generalization performance under dataset shift? The study addresses the practical reality that moderation and forensic systems rarely encounter the same data distribution they were trained on.

Methodology
The research employs a controlled, single-seed experimental design using a fixed face-crop pipeline to isolate the impact of the backbone architecture.

Pipeline: Videos are sampled (every 10th frame), faces are detected and cropped using MTCNN (confidence $\ge$ 0.95, side $\ge$ 60px, 20% padding), and crops are resized to 224 $\times$ 224. Face crops from a single source video are kept within the same data split to prevent leakage.
Backbones: Five ImageNet-pretrained RGB backbones are evaluated with a consistent fine-tuning recipe (binary cross-entropy, label smoothing, AdamW, cosine annealing, progressive unfreezing):
1. EfficientNet-B4
2. Xception
3. ResNet-50
4. ConvNeXt-Tiny
5. Vision Transformer (ViT)-Base
Experimental Settings: Four distinct evaluation protocols are used:
1. Exp1 (In-domain): Source-video-disjoint split on FaceForensics++ (FF++) (Train/Val/Test: 720/140/140 videos).
2. Exp2 (Cross-dataset): Train on FF++, Test on Celeb-DF (v2).
3. Exp3 (Reverse transfer): Train on Celeb-DF, Test on FF++.
4. Exp4 (Held-out domain): Train on FF++ + Celeb-DF, Test on DeepFake Detection (DFD).
Evaluation Metrics: Video-level metrics are computed by aggregating face-crop scores (averaging probabilities) per source video. Key metrics include Accuracy, AUC, F1 score, Equal Error Rate (EER), and 95% bootstrap confidence intervals. Statistical significance is assessed via paired McNemar tests and bootstrap resampling with Holm–Bonferroni adjustment.
Probes: Two low-cost exploratory baselines are added: a frequency-only ResNet-18 (using DCT log-magnitude inputs) and a score-level temporal aggregation baseline (logistic regression on summary statistics of crop scores).

Key Results

Backbone Performance: ConvNeXt-Tiny achieved the highest point estimates for AUC in Exp1 (0.957), Exp2 (0.806), and Exp4 (0.915). ResNet-50 followed closely (0.924, 0.799, 0.812 respectively). However, paired statistical tests indicated that the difference between ConvNeXt-Tiny and ResNet-50 was not statistically significant in Exp1 and Exp2 after Holm adjustment. ViT-Base did not lead under the common training recipe.
Generalization Gap: All backbones exhibited performance degradation when moving from in-distribution to cross-dataset settings.
- FF++ $\to$ Celeb-DF (Exp2): Performance remained useful but dropped significantly (AUC range 0.57–0.81).
- Celeb-DF $\to$ FF++ (Exp3): This reverse transfer was the weakest, with most backbones performing near chance level (AUC range 0.50–0.59). ResNet-50 was the only model with a confidence interval slightly above 0.5.
- Mixed $\to$ DFD (Exp4): Models trained on a mix of FF++ and Celeb-DF maintained reasonable discrimination on the held-out DFD dataset (AUC range 0.71–0.91).
Probes and Robustness:
- The frequency-only ResNet-18 showed some transfer ability in Exp2 (AUC 0.752) despite lower in-domain performance, but it did not surpass the strongest RGB models.
- The score-level temporal probe did not consistently improve performance over mean aggregation and, in some cases, reduced AUC.
- Corruption: Additive noise was the most damaging corruption, reducing AUC by up to 0.41 in Exp1. JPEG compression, blur, and downscaling also degraded performance, though the ordering of impact varied by experiment.

Significance and Claims
The paper positions itself as a controlled backbone study rather than a proposal for a new detector architecture. Its primary claims are modest and focused on evaluation rigor:

Evaluation Protocol Matters: Source-domain coverage, leakage controls (specifically source-video disjoint splits), and video-level evaluation are at least as critical to observed generalization as the choice of backbone.
No Universal Winner: While ConvNeXt-Tiny showed the best point estimates, the study does not claim it is a universally superior architecture, especially given the single-seed training constraint and the lack of statistically significant difference against ResNet-50 in key transfer scenarios.
In-Distribution Overestimation: High in-domain scores (e.g., on FF++) overstate deployment performance; cross-dataset results reveal a substantial reliability gap.
Diagnostic Value: The study serves as a diagnostic of the baseline pipeline, highlighting that performance drops are consistent across architectures when facing dataset shifts, suggesting that architectural changes alone are insufficient to solve the generalization problem without addressing data distribution and evaluation protocols.

Limitations Noted by the Author
The study acknowledges that results are based on a single training seed, preventing claims of run-to-run stability. The temporal probe was exploratory and confounded by changes in detector checkpoints. The corruption study did not include adversarial attacks or platform-specific re-encoding. Future work is suggested to include multiple seeds, specialized frequency or temporal branches, and broader dataset pairs (e.g., DFDC).

Technical Summary: Cross-Dataset Generalization in Deepfake Detection

More like this