💻 computer science

Trustworthy Deepfake Detection Through Explainable AI: Evaluating the Consistency of Visual Explanations Across Deepfake Datasets

This study introduces the Explanation Consistency Score (ECS) to demonstrate that deepfake detectors with high predictive accuracy often rely on unstable, dataset-specific reasoning that fails to generalize, arguing that forensic models must be evaluated on explanation consistency in addition to traditional performance metrics.

Original authors: Adedayo Ayomide ADENIRAN, Adetayo Olaniyi ADENIRAN, Abiodun OJO, Thomas Oluwaseun ONIH

Published 2026-09-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adedayo Ayomide ADENIRAN, Adetayo Olaniyi ADENIRAN, Abiodun OJO, Thomas Oluwaseun ONIH

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, the line between a real photograph and a computer-generated image has become increasingly difficult to draw. Advanced software can now create videos of people saying things they never said or doing things they never did, with a level of realism that tricks the human eye. To combat this, scientists have built computer programs designed to act as digital lie detectors. These programs analyze images and videos to spot tiny, invisible flaws that reveal a forgery. For years, the success of these programs was measured simply by how often they got the right answer. If a program could correctly identify a fake video nine times out of ten, it was considered a success. However, a correct answer does not always mean the program is thinking correctly. Just as a student might guess the right answer on a test by memorizing a pattern rather than understanding the lesson, a computer program might spot a fake for the wrong reasons. This creates a dangerous problem for fields like law enforcement and journalism, where knowing why a decision was made is just as important as the decision itself.

A team of researchers set out to investigate whether these high-performing digital lie detectors were actually trustworthy or if they were just lucky guessers. They focused on two specific computer models, known as XceptionNet and EfficientNet-B0, which are currently among the best at spotting deepfakes. The researchers wanted to see if these models were looking at the right parts of a face when they made their judgments. To do this, they used a technique that highlights the specific areas of an image that the computer is paying attention to, creating a visual map of its thought process. They compared two different ways of generating these maps to see if the computer was consistent. If the computer is truly reliable, both methods should point to the same features, such as the mouth or the eyes, where deepfakes often leave subtle traces. If the methods point to different places, it suggests the computer's reasoning is unstable.

The study began by training these models on a large collection of known real and fake videos. When tested on this familiar material, the models performed exceptionally well, correctly identifying fake videos more than 94 percent of the time. In this controlled environment, the visual maps were also highly consistent. Both methods of checking the computer's attention agreed on where to look, focusing on the facial features where manipulation usually occurs. This gave the impression of a robust and reliable system. However, the researchers then put the models to a much harder test. They asked the computers to analyze a completely different set of videos, featuring high-quality fakes of famous people that the models had never seen before. This shift in the type of data is known as a domain shift, and it is designed to simulate the messy, unpredictable reality of the internet.

The results of this second test revealed a startling disconnect. While the models still managed to identify many of the new fakes correctly, their internal reasoning fell apart. The visual maps, which had been clear and focused before, became scattered and confused. Instead of concentrating on the facial features, the computer's attention drifted to irrelevant areas like the hair, the background, or the edges of the frame. When the researchers measured the agreement between the two methods of checking the computer's attention, the score dropped by nearly half. This means that even when the computer got the right answer, it was no longer looking at the same evidence to get there. One moment it might be focusing on the mouth, and the next on the background, with no logical consistency. This phenomenon, which the researchers call attribution instability, shows that the models were likely relying on shortcuts specific to the first set of videos, such as specific compression patterns, rather than learning the universal signs of a fake face.

The study concludes that high accuracy alone is not enough to trust a deepfake detector. A system can be right for the wrong reasons, and in sensitive situations like court cases or news verification, that is a critical failure. The researchers argue that we need a new standard for evaluating these tools, one that checks not just if they are right, but if their reasoning is stable and consistent across different types of data. They propose a new metric, called the Explanation Consistency Score, to measure this stability. By ensuring that a detector's visual evidence remains focused and reliable even when faced with new and challenging fakes, we can build systems that are not only accurate but also truly trustworthy. The findings suggest that the current generation of deepfake detectors, while impressive, may be more fragile than we thought, and that true reliability requires looking under the hood to see how the machine thinks, not just what it says.

Technical Summary: Trustworthy Deepfake Detection Through Explainable AI

Problem Statement
The proliferation of high-fidelity deepfakes poses significant risks to digital integrity, particularly in forensic, legal, and journalistic contexts where evidence-based reasoning is paramount. While state-of-the-art deepfake detectors based on Convolutional Neural Networks (CNNs) achieve high classification accuracy on benchmark datasets, they often operate as "black boxes," providing binary outputs without transparent reasoning. A critical gap exists in understanding whether high predictive performance correlates with reliable internal reasoning. Specifically, detectors may achieve accuracy by exploiting dataset-specific shortcuts (e.g., compression artifacts or resolution proxies) rather than universal authenticity features. This phenomenon, related to the "Rashomon effect," suggests that models with similar accuracy can rely on divergent internal logic. Furthermore, existing Explainable AI (XAI) evaluations are largely qualitative and fail to quantify the stability of explanations when models face domain shifts, such as transitioning from training data to unseen, high-quality synthetic content.

Methodology
The study proposes a quantitative framework to evaluate the reliability of deepfake detectors by measuring the consistency of their visual explanations. The methodology involves the following components:

Datasets: Two benchmark datasets were utilized. FaceForensics++ (FF++) served as the training and intra-dataset testing ground, containing videos manipulated via DeepFakes, Face2Face, FaceSwap, and NeuralTextures. Celeb-DF (v2) was used as an external, out-of-distribution benchmark to simulate domain shift, featuring high-resolution, high-quality synthetic videos with fewer visual artifacts.
Models: Two state-of-the-art architectures, XceptionNet and EfficientNet-B0, were trained on FF++ and evaluated on both datasets.
Explainability Techniques: For every prediction, two gradient-based attribution methods were generated independently: Grad-CAM and Grad-CAM++. These methods produce heatmaps highlighting image regions influencing the classification decision.
Trust Metric (Explanation Consistency Score - ECS): A novel quantitative metric was introduced to measure the agreement between the Grad-CAM and Grad-CAM++ heatmaps for the same input image. The ECS is computed per image and aggregated at the dataset level. It combines two dimensions:
1. Spatial Overlap: Measured via Intersection over Union (IoU) of binarized heatmaps (thresholded at the top 15% of pixels).
2. Structural Fidelity: Measured via Structural Similarity Index (SSIM) to assess the preservation of texture and structural relationships in the heatmaps.
  The final ECS is a weighted average ( $w=0.5$ ) of the normalized IoU and SSIM scores.

Key Results
The experiments revealed a significant divergence between predictive accuracy and explanation stability, particularly under domain shift:

Intra-Dataset Performance (FF++): Both models achieved high accuracy (XceptionNet: 94.42%, EfficientNet-B0: 96.20%) and high AUC (~0.98–0.99). Crucially, they also demonstrated high explanation consistency, with ECS scores of approximately 0.85 for XceptionNet and 0.89 for EfficientNet-B0. Heatmaps focused semantically on facial regions known to contain artifacts (eyes, mouth, boundaries).
Cross-Dataset Performance (Celeb-DF): While accuracy degraded as expected (XceptionNet: 80.11%, EfficientNet-B0: 82.50%), the explanation consistency collapsed dramatically.
- XceptionNet: ECS dropped to 0.43 (Mean IoU: ~0.39, Mean SSIM: ~0.46).
- EfficientNet-B0: ECS dropped to 0.47 (Mean IoU: ~0.43, Mean SSIM: ~0.51).
Attribution Instability: Under domain shift, the heatmaps became diffuse, spreading to non-discriminative regions like hairlines, cheeks, and backgrounds. The two explanation methods ceased to agree on where the evidence lay, even for correctly classified samples.

Key Contributions

Definition of Explanation Consistency Score (ECS): The paper introduces a quantitative metric to measure the stability of visual explanations by comparing the spatial and structural agreement between two attribution methods (Grad-CAM and Grad-CAM++) on the same input.
Empirical Evidence of Attribution Instability: The study demonstrates that high classification accuracy does not guarantee reliable reasoning. It quantifies a sharp decline in explanation consistency when models are exposed to unseen, high-quality data distributions, even when they maintain moderate predictive success.
Conceptual Distinction: The paper distinguishes between the classical Rashomon effect (divergent models with similar accuracy) and attribution instability (divergent explanations from a single fixed model under domain shift), framing the latter as the explanation-level counterpart to the former.
Forensic Evaluation Standard: The authors argue that for forensic applications, detectors must be evaluated not only by AUC or accuracy but also by the consistency of their visual evidence.

Significance and Claims
The paper claims that relying solely on predictive metrics like accuracy or AUC is insufficient for establishing trust in deepfake detection systems, especially in sensitive domains like law enforcement and judicial proceedings. The primary finding is that predictive success and explanation reliability are complementary, not interchangeable.

The study concludes that a detector which produces correct classifications but alters its visual evidence (attribution maps) when facing new data cannot be assumed to be trustworthy. The observed "attribution instability" suggests that models may be relying on fragile, dataset-specific cues rather than robust, universal authenticity features. Therefore, the authors posit that future forensic AI deployments must incorporate consistency metrics like ECS to ensure that the reasoning behind a detection is stable, reproducible, and evidentiary, rather than a byproduct of overfitting to specific dataset artifacts. The paper explicitly avoids claiming that the instability proves reliance on shortcuts, but rather states that the instability is consistent with such reliance and is sufficient on its own to undermine the evidentiary value of the system.

Technical Summary: Trustworthy Deepfake Detection Through Explainable AI

More like this