💻 computer science

Listening Deepfake Detection: A New Perspective Beyond Speaking-Centric Forgery Analysis

This paper introduces Listening Deepfake Detection (LDD) as a new research direction beyond traditional speaking-centric forgery analysis, presenting the ListenForge dataset and the MANet model to effectively identify subtle motion inconsistencies in synthesized listening reactions guided by speaker audio.

Original authors: Miao Liu, Fangda Wei, Jing Wang, Xinyuan Qian

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Miao Liu, Fangda Wei, Jing Wang, Xinyuan Qian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a lively dinner party. Usually, when we talk about "fake" videos (deepfakes), we worry about the person speaking. We ask, "Is that person really saying those words? Do their lips match the voice?" It's like checking if a puppeteer is pulling the strings correctly on a talking marionette.

But this paper points out a huge blind spot: What about the person listening?

The "Nodding" Problem

In a real conversation, the person listening isn't just a statue. They smile when you tell a joke, frown when you share bad news, and nod along to show they understand.

The authors argue that scammers are now using AI to fake these listening reactions too. Imagine a video call where the scammer is talking, but the person on the other end (the victim) is actually an AI. This AI is programmed to look perfectly engaged, nodding and smiling at the right moments to make the scammer seem more trustworthy.

The Analogy:
Think of a deepfake speaker as a bad actor trying to memorize a script. They might stumble on a line or blink weirdly.
Think of a deepfake listener as a bad audience member. They might smile at a sad story or look confused when the speaker is making a clear point.

The paper says: "Hey, the 'bad audience member' is actually much easier to spot than the 'bad actor' because the technology to fake a natural reaction is still in its baby steps."

The New Tool: ListenForge (The "Fake Reaction" Library)

To catch these fakes, you need practice data. But until now, no one had collected a library of "fake listening" videos.

The authors built ListenForge, which is like a training gym for lie detectors. They took real videos of people talking and used five different AI tools to generate fake "listening" faces. They then fed these fake reactions into their system to teach it what "fake listening" looks like.

The Detective: MANet (The "Motion & Meaning" Detective)

The authors created a new AI detective called MANet. Here is how it works, using a simple metaphor:

Imagine you are watching a magic show.

The Motion-Aware Module (The "Eye for Detail"):
This part of the detective watches the listener's face like a hawk. It knows that real humans blink, nod, and shift their weight in tiny, natural ways. If the AI-generated listener nods too perfectly, or if their smile doesn't quite reach their eyes, this module spots the "glitch" in the matrix. It's looking for the uncanny valley in the movement.
The Audio-Guided Module (The "Context Checker"):
This is the clever part. The detective doesn't just watch the listener; it listens to the speaker.
- Scenario: The speaker tells a hilarious joke.
- Real Human: Laughs and smiles.
- Fake AI: Might look bored or smile at the wrong time.
This module acts like a translator. It takes the meaning of the speaker's words (the audio) and checks if the listener's face matches that meaning. If the speaker is angry, but the listener is smiling, the "Context Checker" sounds the alarm.

Why This Matters

The paper tested old deepfake detectors (designed for speakers) on these new "listening" fakes. The results were terrible—the old detectors were like trying to catch a thief with a net designed for fish. They failed completely.

However, MANet was a superstar. It caught the fakes with incredible accuracy because it was built specifically to spot the weirdness of fake listening.

The Big Takeaway

We used to think deepfake detection was all about checking the speaker's lips. This paper says, "Stop! Look at the listener's face too."

In the future, if you are on a video call with a stranger, don't just listen to what they say. Watch how the other person reacts. If their nods and smiles feel a little "off" or don't match the conversation's mood, you might be talking to a robot, not a human.

In short: The paper introduces a new way to spot fakes by focusing on the person who is listening, using a smart system that checks if their reactions match the story being told.

1. Problem Definition

Current deepfake detection research predominantly focuses on Speaking Deepfake Detection (SDD), where the manipulated subject is actively speaking. These methods rely heavily on detecting inconsistencies between lip movements and audio (lip-sync errors) or facial expression anomalies during speech.

However, the authors identify a critical gap: in realistic interactive scenarios (e.g., video calls, conferences), participants alternate between speaking and listening. Attackers can manipulate the "listening" state to generate realistic non-verbal feedback (nodding, smiling, head shaking) based on the speaker's audio. This creates "Listening Deepfakes."

The Challenge: Existing SDD methods fail in this context because listeners do not exhibit lip-sync constraints. The forgery cues are subtle (micro-expressions, head pose dynamics) and rely on semantic consistency rather than strict temporal synchronization.
The Opportunity: Listening Head Generation (LHG) technology is less mature than Speaking Head Generation (SHG), meaning synthesized listening reactions often contain detectable artifacts and semantic misalignments that current detectors miss.

2. Key Contributions

The paper makes three primary contributions to the field:

Task Introduction: It formally defines the Listening Deepfake Detection (LDD) task, shifting the paradigm from speaker-centric to listener-centric forgery analysis.
Dataset Construction (ListenForge): The authors introduce ListenForge, the first dataset specifically designed for LDD.
- Source: Built using the ViCo and NoXi corpora.
- Generation: Utilizes five state-of-the-art Listening Head Generation (LHG) methods: ViCo, PCHG, DSPN, ListenFormer, and Trans-VAE.
- Scale: Contains 10,655 audio-visual clips (8,746 training, 954 validation, 955 test) with corresponding speaker audio to enable cross-modal analysis.
Proposed Method (MANet): A novel Motion-aware and Audio-guided Network designed to detect subtle inconsistencies in listener videos by leveraging speaker audio semantics.

3. Methodology: MANet

The proposed MANet framework consists of two pre-trained encoders (Video and Audio) and two specialized modules:

A. Motion-Aware Module (MAM)

Since listener videos are largely static compared to speaking videos, forgery artifacts often appear in subtle, infrequent motions (blinking, nodding).

Mechanism: The module computes the temporal difference between adjacent frames ( $M_t = V_{t+1} - V_t$ ) to isolate motion features.
Attention Mechanisms: It employs a Spatial-Channel Attention (SCA) mechanism:
- Spatial Attention: Focuses on specific facial regions where dynamic anomalies occur (e.g., unnatural chin movement).
- Channel Attention: Re-weights feature channels to amplify those correlated with forgery artifacts while suppressing noise.
Goal: To capture high-frequency, subtle motion inconsistencies that standard forward propagation might overlook.

B. Audio-Guided Module (AGM)

This module addresses the unique nature of LDD, where the listener's reaction must be semantically consistent with the speaker's audio (e.g., smiling when a joke is told).

Asymmetric Fusion: Unlike standard multimodal fusion which treats audio and video equally, AGM uses an asymmetric paradigm.
- Query: Speaker's audio features (providing semantic context).
- Key/Value: Listener's visual features (refined by MAM).
Mechanism: A Cross-Attention mechanism allows the model to adaptively retrieve and aggregate visual responses that correlate with the current speech semantics.
Goal: To detect semantic inconsistencies (e.g., a neutral face while the speaker tells a joke) rather than just visual artifacts.

4. Experimental Results

The authors evaluated MANet against existing SDD models (Xception, MesoNet, AVTFD, etc.) on the ListenForge dataset.

Baseline Performance: Existing SDD models performed poorly on ListenForge (e.g., AVTFD achieved ~54% AUC), confirming that speaker-focused detectors cannot generalize to listening scenarios.
MANet Performance: MANet achieved state-of-the-art results:
- Test Set AUC: 97.24%
- Test Set Accuracy: 89.74%
Comparative Gains: MANet outperformed the best existing multimodal baseline (AVTFD) by 42.94% in AUC.
Ablation Studies:
- Removing the Motion-Aware Module significantly dropped performance, proving the necessity of capturing subtle motion differences.
- Using the Audio-Guided Module with speaker audio as a query yielded better results than concatenating features or using speaker video, confirming that audio semantics are the most critical guide for listener forgery detection.

5. Significance and Future Directions

Paradigm Shift: The paper argues that deepfake detection must evolve beyond the "speaking-centric" view to cover the full spectrum of interactive communication.
Vulnerability Analysis: It highlights that current LHG technologies are the "weak link" in deepfake generation, making them easier to detect than mature speaking forgeries, but requiring specialized tools.
Practical Impact: The proposed dataset and method provide a foundation for securing interactive media, preventing identity fraud, and detecting personalized disinformation campaigns in video conferencing and social media.
Future Work: The authors plan to develop unified frameworks that can simultaneously detect forgeries in both speaking and listening states within a single interactive system.

Conclusion: This work establishes Listening Deepfake Detection as a distinct and critical research area, providing the necessary dataset (ListenForge) and a specialized architecture (MANet) to effectively identify forgeries that traditional methods miss.