🤖 machine learning

Interpretable facial dynamics as behavioral and perceptual traces of deepfakes

This study demonstrates that interpretable bio-behavioral features of facial dynamics, particularly those related to emotional expression, serve as measurable behavioral fingerprints for deepfake detection and reveal that while computational models and human perception converge on emotive content, they rely on fundamentally different detection strategies.

Original authors: Timothy Joseph Murphy, Jennifer Cook, Hélio Clemente José Cuve

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Timothy Joseph Murphy, Jennifer Cook, Hélio Clemente José Cuve

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a party, and someone tells you a story. You can tell if they are lying not just by what they say, but by how they say it—the slight tremble in their voice, the way they blink too fast, or how their hands move a little too stiffly.

This paper is about catching "Deepfakes" (fake videos made by AI) using that same logic. Instead of looking for pixelated glitches or weird lighting (which is like looking for a typo in a letter), the researchers looked at the "body language" of the face itself.

Here is the story of their discovery, broken down simply:

1. The Problem: The "Black Box" Detectives

Most current AI detectors are like super-smart but silent detectives. They can spot a fake video with high accuracy, but they can't explain why. They just say, "This is fake," without telling you which muscle moved wrong or which blink was unnatural. This is a problem because if we want to trust these tools in court or newsrooms, we need to understand their reasoning.

2. The Solution: Listening to the Face's "Rhythm"

The researchers decided to stop looking at the image and start listening to the movement. They treated facial muscles like a musical instrument.

The Analogy: Imagine a drummer. A real human drummer has a natural, slightly imperfect rhythm. An AI trying to mimic that drummer might get the notes right, but the timing feels robotic or "off."
The Method: They used a tool called NMF (Non-negative Matrix Factorization) to break down complex facial movements into three simple "musical themes" or patterns:
1. The Smile: Cheeks and lips moving together.
2. The Frown: Chin and lips tightening.
3. The Surprise: Eyebrows and eyelids lifting.

They then measured the rhythm, smoothness, and predictability of these movements over time.

3. The Big Discovery: Emotions are the "Tell"

Here is the most interesting part: The AI is much worse at faking emotions than it is at faking neutral faces.

The "Neutral" Trap: When a person is just talking normally (no big emotions), the AI can mimic the face movements quite well. It's like a robot copying a calm conversation; it's hard to spot.
The "Emotion" Slip-up: When a person is laughing, angry, or sad, their face involves a complex, coordinated dance of many muscles. The AI struggles to keep this dance in perfect rhythm.
- The Metaphor: Think of a real human emotion like a jazz improvisation. It's fluid, coordinated, and has a specific "flow." The AI's version is like a metronome trying to play jazz. It hits the right notes (the muscles move), but the flow is stiff and predictable. The researchers found that the AI's "jazz" was so out of sync that it gave away the fake.

4. The Human vs. Machine Showdown

The researchers also asked real humans to watch these videos and guess which were fake. They compared the humans' guesses to the computer's guesses.

The Result: Both humans and the computer got better at spotting fakes when the person was showing emotion.
The Twist: Even though they agreed on the answer ("That's fake!"), they were using different clues to get there.
- The Computer was looking for tiny, mathematical irregularities in the speed of the jaw or eyebrows (like checking the rhythm with a stopwatch).
- The Humans were picking up on a general "feeling" of unnaturalness, perhaps noticing that the movement felt "clunky" or "unpredictable" in a different way.

The Takeaway: Humans and machines are like two different detectives solving the same crime. They might catch the same criminal, but they found different pieces of evidence. This means we should use both together for the best results.

Why Does This Matter?

Transparency: Instead of a "black box" saying "Fake," we now know why: "This video is fake because the rhythm of the eyebrow movement doesn't match the natural flow of a human smile."
Better Detection: We know that if a video shows strong emotions, it's easier to catch the AI. If a video is just a person talking calmly, it's much harder.
Human-Machine Teamwork: Since humans and computers look for different things, combining them creates a super-team that is much harder to fool than either one alone.

In a nutshell: Deepfakes are getting better at looking real, but they are still terrible at feeling real. By listening to the rhythm of our facial muscles, we can hear the AI's "robotic heartbeat" and catch it in the act.

1. Problem Statement

Current deepfake detection research is dominated by deep learning (DL) approaches (e.g., CNNs, RNNs) that achieve high benchmark accuracy but suffer from three critical limitations:

Lack of Interpretability: They function as "black boxes," offering little insight into why a video is flagged as fake, which is problematic for forensic and legal applications.
Human-Machine Disconnect: There is a lack of empirical evaluation regarding the correspondence between model predictions and human perceptual judgments.
Computational Cost: High resource demands limit practical deployment.

Furthermore, while bio-behavioral approaches exist, they often rely on high-dimensional latent representations that reintroduce opacity. There is a specific gap in understanding how facial dynamics (specifically the temporal coordination of facial muscles) differ between real and manipulated videos, and how emotional expression influences these differences.

2. Methodology

The study adopts a person-generic, interpretable approach focusing on the temporal structure of facial movements rather than spatial artifacts or identity-specific features.

Dataset and Preprocessing

Source: Google DeepFakeDetection (DFD) dataset (subset of FaceForensics++), containing face-swapped deepfakes generated via the Faceswap autoencoder.
Sample: 464 videos (232 real/fake pairs), truncated to 241 frames (10 seconds).
Feature Extraction: Facial Action Unit (AU) intensities were extracted using OpenFace 2.0 for 17 AUs. Data was smoothed (rolling mean) and normalized.

Dimensionality Reduction & Feature Engineering

Non-Negative Matrix Factorization (NMF): Applied to the AU time series to identify low-dimensional, additive patterns of coordinated facial movement. A 3-component model was selected as optimal:
- Component 1: Lower-face (cheek raising, lip corners).
- Component 2: Lower-face (chin raising, lip depression).
- Component 3: Upper-face (brow raising, lid retraction).
Representative Proxy Selection: Instead of using the abstract NMF activation matrices, the authors selected the single AU with the highest loading for each component (AU12, AU17, AU01) as a proxy for that movement pattern.
Temporal Feature Extraction: Using the CMFTS (Complexity Measures and Features for Time Series) package, 111 metrics were extracted from the three representative AUs. These metrics quantified kinematic properties (smoothness, rhythm, velocity, acceleration, autocorrelation) rather than movement magnitude.
Feature Selection: The Boruta algorithm identified 8 statistically important features (e.g., rhythm of chin velocity, smoothness of brow acceleration) that distinguish real from fake.

Classification & Evaluation

Models: Four interpretable classifiers were trained: Logistic Regression, C5.0 Boosted Trees, Random Forest (RF), and Support Vector Machine (SVM).
Validation: 5-fold, 3-repeat cross-validation on the training set ( $n=370$ ); evaluation on a held-out test set ( $n=94$ ).
Emotion Stratification: The test set was split into Emotive ( $n=58$ ) and Non-Emotive ( $n=36$ ) subsets to test detection signal strength.
Valence Analysis: A separate RF model trained only on real videos predicted emotional valence (positive, neutral, negative) to test if deepfakes degrade emotional signals.

Human-Machine Comparison

Experiment: 89 human participants judged the authenticity of Point Light Displays (PLDs) of the same videos (isolating temporal dynamics, removing texture).
Analysis: Compared human judgments with model predictions to assess convergence (agreement) and divergence (underlying strategies).

3. Key Results

A. Detectability via Temporal Irregularities

Performance: Non-linear models (specifically Random Forest) achieved modest but significant above-chance performance (ROC-AUC = 0.694, Accuracy = 66.0%, MCC = 0.322).
Mechanism: Detection was driven by higher-order temporal irregularities (deviations in the predictability and smoothness of movement velocity/acceleration) rather than static artifacts.
Interpretability: The 8 selected features provided a clear "behavioral fingerprint" linked to observable muscle actions (e.g., chin and brow movement rhythms).

B. The Emotion-Dependent Effect

Differential Accuracy: Detection was significantly more accurate for Emotive videos (72.4% accuracy, $p < .001$ ) compared to Non-Emotive videos (55.6% accuracy, $p = .309$ , chance level).
Valence Degradation: A valence classification analysis revealed that deepfakes systematically degrade emotional signals.
- Real videos: High accuracy in classifying valence (59.6%).
- Fake videos: Significantly lower accuracy (51.1%), with a sharp drop in sensitivity for negative and positive expressions.
- Conclusion: Face-swapping preserves neutral dynamics better than complex, multi-muscle emotional coordination, making emotive deepfakes easier to detect.

C. Human-Machine Convergence and Divergence

Output Alignment: For emotive videos, human and model judgments converged significantly (81.2% agreement). For non-emotive videos, they did not.
Strategic Divergence: Despite agreeing on the final label for emotive videos, the underlying strategies differed:
- Human accuracy did not correlate with the model's most diagnostic features.
- Humans were fooled by different stimuli than the model.
- Feature analysis showed humans relied on entropy and long-range dependencies, while the model relied on short-term autocorrelation of velocity.
Implication: Humans and machines offer complementary, not redundant, detection routes.

4. Key Contributions

Interpretable Detection Framework: Demonstrated that deepfakes can be detected using low-dimensional, bio-behavioral features (temporal kinematics of AUs) without relying on opaque deep learning embeddings.
Emotion as a Diagnostic Signal: Provided empirical evidence that face-swapping selectively degrades the spatiotemporal coordination of emotional expressions, making emotive content a stronger diagnostic signal for detection.
Human-Machine Complementarity: Established that while humans and AI may agree on what is fake in emotional contexts, they rely on different perceptual cues. This supports the development of human-in-the-loop systems that leverage complementary strengths.
Methodological Pipeline: Validated a pipeline using NMF for pattern discovery and traditional machine learning for classification, offering a transparent alternative to "black box" DL detectors.

5. Significance

Forensic & Legal Utility: The findings offer a transparent, explainable basis for deepfake detection, crucial for legal admissibility where "why" a decision was made is as important as the decision itself.
Understanding Generative Limits: The study reveals that current autoencoder-based face-swapping struggles to replicate the fine-grained, high-coordination temporal dynamics required for authentic emotional expression.
Future AI Systems: The divergence between human and machine strategies suggests that future detection systems should not aim to perfectly mimic human perception but rather integrate both to create robust, multi-modal detection workflows.
Limitations & Future Work: The study is limited to Faceswap autoencoders and a specific dataset. Future work should explore other generation methods (e.g., diffusion models) and refine emotion annotations.

In summary, the paper argues that deepfakes leave a measurable behavioral fingerprint in the temporal dynamics of facial movement, particularly during emotional expression, and that combining interpretable computational models with human perception offers a more robust path forward than either approach alone.