💻 computer science

Leave No Stone Unturned: Uncovering Holistic Audio-Visual Intrinsic Coherence for Deepfake Detection

This paper introduces HAVIC, a deepfake detector that leverages holistic audio-visual intrinsic coherence learned from authentic videos to achieve superior generalization and performance on a new high-fidelity dataset (HiFi-AVDF), significantly outperforming existing state-of-the-art methods in cross-dataset scenarios.

Original authors: Jielun Peng, Yabin Wang, Yaqi Li, Long Kong, Xiaopeng Hong

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jielun Peng, Yabin Wang, Yaqi Li, Long Kong, Xiaopeng Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to spot a fake video. In the past, deepfake detectors were like detectives who only looked at one clue: either the picture (visuals) or the sound (audio). They looked for tiny, specific glitches left behind by old, clumsy forgers—like a blurry edge on a face or a weird echo in a voice.

But today's forgers are using super-advanced AI (like the ones making movies or generating videos from text). They are so good that they don't leave those old, obvious glitches. They are like master magicians who make the picture and sound look perfect individually, but they might still be slightly out of sync or just "feel" wrong when you put them together.

This paper introduces a new detective named HAVIC (Holistic Audio-Visual Intrinsic Coherence). Instead of looking for specific glitches, HAVIC looks for natural harmony.

Here is how it works, broken down into simple concepts:

1. The "Natural Harmony" Detective

Think of a real human conversation like a perfectly orchestrated orchestra.

The Visuals: The singer's lips move exactly when the note is sung.
The Audio: The voice sounds like it's coming from the person's mouth, not a speaker in the corner.
The Context: If the singer is talking about a beach, the background looks like a beach.

Old detectors were like a technician checking if a single instrument was out of tune. HAVIC is like a conductor listening to the entire orchestra. It asks: "Does the sound match the movement? Does the movement match the scene? Does the whole thing feel like a real, living moment?"

2. The Three Levels of "Feeling Real"

The authors say HAVIC checks for "coherence" (harmony) on three levels, like checking a painting for authenticity:

Level 1: The Single Modality Check (The Texture)
- Analogy: Looking at a painting to see if the brushstrokes look like real paint or if the texture is weird.
- What HAVIC does: It checks if the video alone looks natural (no weird face deformities) and if the audio alone sounds natural (no robotic distortions).
Level 2: The Micro-Check (The Lip-Sync)
- Analogy: Watching a puppet show. If the puppet's mouth opens for the word "Hello," but the sound comes out a split second late, you know it's fake.
- What HAVIC does: It checks the tiny, millisecond timing between a sound (like a "p" or "b" sound) and the exact shape of the lips.
Level 3: The Macro-Check (The Story)
- Analogy: Imagine a video of a person screaming in terror, but the audio is a cheerful lullaby. The story doesn't match.
- What HAVIC does: It checks the big picture. If the audio says "I'm at the beach," but the video shows a snowy mountain, HAVIC knows something is wrong, even if the face and voice look perfect.

3. How HAVIC Learned to Be a Detective

You can't teach a detective to spot fakes just by showing them fakes, because forgers keep changing their tricks. Instead, HAVIC was trained on thousands of real, authentic videos.

The Training: Imagine showing a student millions of real videos and saying, "Learn what 'real' feels like." HAVIC learned the "laws of physics" for how sound and vision naturally interact.
The Test: When it sees a new video, it compares it against those "laws of physics." If the video breaks the rules of natural harmony (even slightly), it flags it as a fake.

4. The New "Exam" (The Dataset)

The authors realized that old tests for deepfakes were too easy. They were like giving a math genius a test with only addition problems. The new forgers can easily pass those.

So, the team created a new, super-hard exam called HiFi-AVDF.

What is it? A collection of videos made by the world's most advanced commercial AI tools (like Sora, Kling, and Veo).
Why is it special? These videos are so realistic that even humans struggle to tell them apart from reality. It's the ultimate stress test for any detector.

5. The Results

When they put HAVIC to the test:

Old Detectors: They stumbled. They got confused by the new, high-quality fakes.
HAVIC: It aced the test. It caught the fakes that others missed by a huge margin (improving detection by nearly 10% on the hardest tests).

The Bottom Line

This paper argues that to catch the next generation of AI fakes, we can't just look for "glitches." We have to understand how the world naturally works.

HAVIC is a detective that doesn't just look for broken pieces; it listens for the song of reality. If the song is even a tiny bit off-key, it knows it's a fake. This makes it much harder for bad actors to fool it, no matter how advanced their AI tools become.

1. Problem Statement

The rapid advancement of generative AI has led to hyper-realistic audio-visual deepfakes that threaten personal security and social trust. Existing detection methods suffer from two primary limitations:

Single-Modality or Artifact Reliance: Most detectors rely on low-level artifacts specific to certain generators (e.g., frequency signals, texture patterns) or single-modal inconsistencies. These methods fail to generalize when faced with unseen generators (cross-generator shift) or advanced models like Sora and DALL·E.
Inadequate Inter-Modal Analysis: While some methods check for audio-visual discrepancies (e.g., lip-sync), they often miss deeper semantic or structural incoherencies. As lip-sync technology matures, simple synchronization checks are becoming less effective.

The authors argue that robust detection must be grounded in intrinsic audio-visual coherence across three levels:

Modality-Specific Structural Coherence: Physical plausibility within a single modality (e.g., unnatural deformities or acoustic distortions).
Inter-Modal Micro-Coherence: Precise segment-level alignment (e.g., phoneme articulation matching lip shapes).
Inter-Modal Macro-Coherence: Instance-level semantic consistency (e.g., the audio describing an action that matches the visual scene).

2. Methodology: HAVIC Framework

The authors propose HAVIC (Holistic Audio-Visual Intrinsic Coherence), a two-stage framework designed to learn and leverage these coherence priors.

Stage 1: Holistic Coherence Priors (HCP) Pre-training

The model is pre-trained on large-scale authentic videos using a self-supervised approach adapted from Masked Autoencoders (MAE). It employs three synergistic objectives to learn intrinsic coherence:

Modality-Specific Hierarchical Reconstruction Loss ( $L_{rec}$ ):
- Masks a large portion of audio and visual tokens.
- Uses hierarchical decoders to reconstruct masked tokens from encoder features at multiple layers.
- Goal: Learn structural plausibility and physical laws within each modality.
Fine-grained Audio-Visual Contrastive Loss ( $L_{cl}$ ):
- Partitions high-level features into temporal segments.
- Uses a soft negative pair strategy to align corresponding audio-visual segments while distinguishing non-corresponding segments within the same sample.
- Goal: Learn micro-coherence (temporal alignment).
Cross-modal Semantic Reconstruction Loss ( $L_{cross}$ ):
- Forces one modality to reconstruct the global semantic representation of the other (e.g., using video to predict audio semantics).
- Goal: Learn macro-coherence (semantic consistency).

Stage 2: Holistic Adaptive Aggregation (HAA) Classification

In the fine-tuning stage, the model is adapted for detection on labeled data:

Feature Extraction: The pre-trained encoders and interaction modules extract hierarchical features and interaction-aware features.
Adaptive Feature Aggregation: A learnable scoring network dynamically weighs tokens at different layers and modalities. This allows the model to focus on the most discriminative cues (low-level artifacts, high-level semantics, or cross-modal discrepancies) depending on the specific sample.
Classification: A main classifier predicts overall authenticity, supported by auxiliary classifiers on uni-modal features to prevent over-reliance on a single modality.

3. Key Contributions

A. The HAVIC Framework

A novel two-stage detector that moves beyond artifact detection to model holistic intrinsic coherence. By pre-training on real data to understand "what is real," the model generalizes better to unseen forgeries.

B. The HiFi-AVDF Dataset

The authors introduce a new, high-fidelity benchmark dataset to address the lack of modern deepfake data:

Content: 3,810 videos (1,905 real + 1,905 fake) featuring both Text-to-Video (T2V) and Image-to-Video (I2V) forgeries.
Generators: Synthesized using cutting-edge commercial models including Sora 2, Veo 3.1, Kling 2.5, Seedance 1.0, PixVerse V5, and WAN 2.5.
Significance: Unlike legacy datasets (e.g., FaceForensics++), HiFi-AVDF reflects "in-the-wild" conditions with high-fidelity, tightly synchronized audio-visual manipulations.

C. Comprehensive Evaluation

Extensive experiments demonstrate that HAVIC outperforms state-of-the-art (SOTA) methods, particularly in cross-dataset scenarios where generalization is critical.

4. Experimental Results

Intra-Dataset Performance (FakeAVCeleb): HAVIC achieved 99.8% Accuracy and 99.9% AUC, outperforming visual-only and other multi-modal baselines.
Cross-Dataset Generalization (KoDF): When trained on FakeAVCeleb and tested on KoDF, HAVIC achieved 99.2% AP and 98.9% AUC, significantly outperforming previous methods.
Generalization to High-Fidelity Deepfakes (HiFi-AVDF): This is the most challenging scenario. HAVIC achieved an average 75.41% AP and 75.77% AUC.
- This represents a massive improvement over existing SOTA methods, with gains of +9.39% AP and +9.37% AUC.
- Other methods struggled significantly (often dropping below 65% AP) against T2V/I2V generators, whereas HAVIC maintained robustness.
Ablation Studies: Removing the pre-training phase or any of the three coherence losses resulted in significant performance drops, validating the necessity of learning holistic priors.

5. Significance and Impact

Paradigm Shift: The paper shifts the focus from detecting specific "fingerprints" of generators to understanding the fundamental structural and semantic coherence of real-world media. This approach is inherently more robust against evolving generation technologies.
Benchmarking Standard: The release of HiFi-AVDF provides the community with a necessary tool to evaluate detectors against the latest generation of AI, moving beyond outdated GAN-based datasets.
Robustness: The framework demonstrates that learning "what is real" via self-supervised pre-training on authentic data is a superior strategy for generalization compared to supervised learning on specific fake patterns.
Limitations: The current work focuses on human face videos. Extending this to non-human scenarios (objects, animals) remains a future challenge. Additionally, while robust, performance still drops on the most advanced generators, indicating the need for continued research.

In conclusion, HAVIC represents a significant step forward in deepfake detection by treating the problem as a holistic coherence verification task rather than a simple anomaly detection task, supported by a rigorous new dataset tailored for the era of generative AI.