🤖 AI

Less is More: Modality-Decoupling for General AIGC Audio-Video Detection

This paper introduces DAV-Det, a decoupled audio-visual detection system that independently models forensic evidence from each modality using multi-granularity visual representations and a gated temporal-spectral audio architecture, achieving first place in the IJCAI-ECAI 2026 General AIGC Audio-Video Detection Challenge by challenging the assumption that cross-modal inconsistencies are always reliable for forgery detection.

Original authors: Jielun Peng, Yabin Wang, Yaqi Li, Jincheng Liu, Xiaopeng Hong, Athanasios V. Vasilakos

Published 2026-07-29

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jielun Peng, Yabin Wang, Yaqi Li, Jincheng Liu, Xiaopeng Hong, Athanasios V. Vasilakos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you can't trust your eyes or ears. Thanks to a rapid explosion in Artificial Intelligence, computers have learned to create fake videos and sounds that look and sound incredibly real. This isn't just about swapping a celebrity's face; it's about generating entire scenes, animals, and objects that never existed. This technology, known as AIGC (AI-Generated Content), is a double-edged sword: it's amazing for creativity but terrifying for misinformation and fraud. To fight back, scientists have been building "digital lie detectors." For years, the most popular strategy for catching these fakes relied on a simple idea: in a real video, what you see and what you hear should match perfectly. If a person is talking, their lips should move in sync with the sound. If the audio and video don't line up, it's a fake. It's like checking if a puppet's mouth is moving in time with the voice behind it; if they are out of sync, you know it's a trick.

However, a new study by researchers at Harbin Institute of Technology suggests that this old rule of thumb is broken when it comes to general videos. They found that in many real-world scenarios—like a nature documentary or a cooking show—the sound and the picture might not be tightly connected even when everything is 100% real. In these cases, looking for a mismatch is like trying to find a thief by checking if their shoes match their hat; sometimes, real people just don't wear matching outfits. The researchers argue that instead of forcing the audio and video to talk to each other to find a lie, we should let them work as independent detectives. They propose a new system called DAV-Det, which treats the audio and video as separate clues. The visual detective looks for tiny, weird textures or patterns in the picture, while the audio detective listens for unnatural glitches in the sound waves. Only after both have made their own verdict do they combine their opinions to decide if the whole thing is a forgery. This "less is more" approach, which avoids forcing a connection where none exists, turned out to be the key to winning a major international competition for spotting AI fakes, achieving a top score of 0.8460.

The Big Mistake: Assuming Everything is Connected

For a long time, the best way to catch a deepfake was to look for the "uncanny valley" of connection. If a video shows a person speaking, the AI often struggles to make the lips move perfectly with the words. So, detectors were built to measure how well the audio and video matched. If the match was bad, the system screamed "Fake!"

The authors of this paper decided to test this assumption on a much wider scale. They looked at two types of videos:

Human-centric videos: Like a person talking to the camera. Here, the old rule works great. Real videos have high audio-visual similarity, while fakes have low similarity.
General videos: Like a video of a dog barking, a car engine revving, or a landscape with wind noise.

When they tested the "matching" rule on general videos, it fell apart completely. In fact, it got worse than random guessing! They found that in many real general videos, the audio and video are naturally less similar than in some fake ones. Trying to use the "mismatch" rule here was actively hurting the detection. It's like trying to find a fake painting by checking if the frame matches the canvas; in a real museum, the frame might not match the canvas at all, but the painting is still real. The paper explicitly rules out the idea that audio-visual correspondence is a reliable cue for general AIGC detection.

The New Strategy: Two Independent Detectives

Since forcing the audio and video to compare notes was failing, the team proposed a "Decoupled" approach. Instead of a single brain trying to process both at once, they built two specialized brains that work separately and then share their conclusions.

The Visual Detective (DAV-Det's Eye)
The visual detector doesn't just look at the whole picture; it zooms in on three different levels of detail, like a detective examining a crime scene:

Global Level: It looks at the big picture to see if the whole scene feels "off" or semantically wrong.
Patch Level: It breaks the image into tiny squares (patches) to find subtle texture glitches, like a weird blur on a specific object that shouldn't be there.
Segment Level: It groups nearby patches into larger chunks (segments) to spot regional inconsistencies, like a weird shadow that doesn't belong to a specific area.

The system uses a clever trick called "Weak Supervision." Since it doesn't always know exactly which tiny patch is fake, it assumes that if the whole image is fake, at least some of the tiny patches must be suspicious. It learns to spot these suspicious patches and uses them to build a stronger case. It also practices on "degraded" images (images with added noise or blur) to make sure it doesn't get confused by bad camera quality.

The Audio Detective (DAV-Det's Ear)
The audio detector listens for two types of clues:

Time-based clues: Is the rhythm of the sound natural? Are the transitions between sounds smooth, or do they jump weirdly?
Frequency clues: Does the sound have strange high-frequency artifacts that real microphones usually don't pick up?

To catch these, the audio detector uses a "gated" system. Imagine a bouncer at a club who decides which parts of the sound are important and which should be ignored. This bouncer (the gating network) highlights the most suspicious parts of the sound wave, allowing the detector to focus on the temporal (time) and spectral (frequency) irregularities that AI often leaves behind.

The Final Verdict: Decision-Level Fusion

Once both detectives have done their work, they don't mix their data together into a giant soup. Instead, they keep their findings separate and make a final decision based on their individual scores.

For a simple "Real vs. Fake" check: If either the audio detective or the visual detective thinks it's fake, the system flags it as fake. It's a "better safe than sorry" approach.
For a detailed check (Four Classes): The system calculates the probability of four scenarios:
1. Real Video + Real Audio (RR)
2. Fake Video + Fake Audio (FF)
3. Fake Video + Real Audio (FR)
4. Real Video + Fake Audio (RF)

By assuming the audio and video are independent, the system multiplies their probabilities to figure out which of these four scenarios is most likely.

The Results: Winning the Game

The team tested their new system, DAV-Det, in the "General AIGC Audio-Video Detection Challenge" at the IJCAI-ECAI 2026 DDL 2.0 Workshop. This was a tough competition with over 200,000 video-audio samples covering everything from humans to animals and objects.

The results were impressive:

Rank: They took 1st place out of all competing teams.
Score: They achieved a final score of 0.8460.
Performance: They scored 0.9617 on the binary (Real/Fake) detection task and 0.6873 on the more difficult four-class classification task.

When they tested their system on older, human-centric deepfake datasets (where the old "matching" rule usually works), DAV-Det still crushed the competition, reaching an accuracy of 0.998 and an AUC of 0.999. This suggests that their method is robust enough to handle both the tricky general videos and the traditional face-swapping fakes.

What's Next?

The authors are honest about what their system can't do yet. Their visual detective is great at spotting static image flaws but doesn't explicitly track how things move across frames (temporal motion). Also, their method of combining the two detectives relies on simple math rules (like taking the maximum score) rather than a complex, learning-based fusion.

However, the core message is clear: in the world of general AI fakes, sometimes the best way to find the truth is to stop looking for a connection that isn't there. By letting the audio and video speak for themselves, the system found a more reliable path to the truth. As the paper suggests, "Less is More"—decoupling the modalities allows for a more robust detection system that doesn't get fooled by the natural lack of synchronization in real-world videos.

Technical Summary: DAV-Det for General AIGC Audio-Video Detection

1. Problem Definition and Motivation

The rapid advancement of Artificial Intelligence Generated Content (AIGC) has expanded the scope of audio-visual forgery beyond human-centric deepfakes (e.g., face swaps, lip-syncing) to general scenes involving animals, objects, and environments. While existing detection methods often rely on the assumption of audio-visual content correspondence—the idea that authentic human speech aligns with lip movements—this paper argues that this assumption fails in general AIGC scenarios.

The authors empirically demonstrate that in general-scene datasets (e.g., MVAD), real samples often exhibit lower audio-visual similarity than fake samples, contrary to the behavior observed in human-centric datasets (e.g., FakeAVCeleb). Consequently, traditional feature-level fusion methods that seek cross-modal inconsistencies may be counterproductive, yielding performance worse than random guessing in general scenarios. The core problem addressed is the need for a robust detection framework that does not rely on the premise of strong audio-visual alignment.

2. Methodology: DAV-Det

The proposed solution, DAV-Det (Decoupled Audio-Visual AIGC Detection), adopts a "Less is More" philosophy by decoupling the modeling of audio and visual modalities. Instead of feature-level cross-modal interaction, the system independently extracts forensic evidence from each modality and performs decision-level fusion.

2.1 Visual AIGC Detector

The visual component is designed to capture forgery cues at multiple spatial granularities, ranging from global semantic anomalies to local texture artifacts.

Backbone: Utilizes DINOv3 to extract a global CLS token and a sequence of patch tokens.
Multi-Granularity Representation:
- Global: Directly uses the CLS token.
- Patch-level: A scoring network estimates forgery logits for each patch, which are aggregated via softmax-weighted summation.
- Segment-level: Patch tokens are clustered into spatial segments using agglomerative clustering. A separate scoring network aggregates these segments.
Training Strategies:
- Weak Supervision: Since only image-level labels are available, the patch and segment networks are trained using Multiple Instance Learning (MIL). The top- $k$ forgery logits are aggregated to predict the image label. A margin loss is introduced to maximize the gap between suspicious and non-suspicious instances.
- Degraded-Original Consistency Learning (DOCL): To improve robustness against real-world degradations (noise, blur, compression), the model is trained on both original and perturbed images, enforcing consistency between their multi-granularity representations.
- Auxiliary Classifiers: Three auxiliary classifiers operate on the global, patch, and segment representations to prevent over-reliance on a single feature level during training.

2.2 Audio AIGC Detector

The audio component models acoustic artifacts by exploiting both temporal dynamics and spectral irregularities.

Backbone: Uses the audio encoder of PEAV to extract a global CLS token and a sequence of frame tokens.
Gated Temporal-Spectral Dual-Branch Architecture:
- A gating network generates a feature-wise gating map to adaptively modulate frame tokens.
- Temporal Branch: Uses self-attention to model temporal dependencies, followed by pooling strategies (average, max, and learnable query-based) to summarize temporal dynamics.
- Spectral Branch: Employs channel attention to emphasize spectral patterns, followed by similar pooling strategies to capture spectral artifacts.
Integration: The global CLS token attends to the temporal and spectral representations via cross-attention to produce a unified audio representation, which is concatenated with the pooled features for final classification.

2.3 Decision-Level Fusion

During inference, the audio and visual detectors operate independently.

Binary Classification: A max-based fusion strategy is used. If either the audio or visual modality predicts a high probability of forgery, the sample is classified as fake.
Four-Class Classification: An independence assumption is used to compute joint probabilities for the four categories: Real-Real (RR), Fake-Fake (FF), Fake-Real (FR), and Real-Fake (RF). The class with the highest joint probability is selected.

3. Key Contributions

Empirical Insight: The paper demonstrates that the assumption of audio-visual content correspondence, foundational to most multi-modal deepfake detectors, does not hold in general AIGC scenarios and can be detrimental to detection performance.
DAV-Det Architecture: A novel decoupled system that independently models modality-specific forensic cues using multi-granularity visual representations and a gated temporal-spectral audio design, fusing decisions rather than features.
State-of-the-Art Performance: The method achieved 1st place in the General AIGC Audio-Video Detection Challenge of the IJCAI-ECAI 2026 DDL 2.0 Workshop with a final score of 0.8460.

4. Experimental Results

Competition Performance: On the DDL-GAV benchmark (based on the MVAD dataset), DAV-Det outperformed the top 5 teams. It achieved a Binary AUC of 0.9617 and a Four-Class F1 score of 0.6873.
Generalization: When evaluated on the human-centric FakeAVCeleb dataset, DAV-Det achieved an accuracy of 0.998 and AUC of 0.999, surpassing existing audio-visual methods (e.g., AVFF, PIA). This suggests the decoupled approach is effective even in scenarios where audio-visual correspondence exists.
Ablation Studies:
- Visual: Adding patch and segment branches progressively improved performance. Removing weak supervision, auxiliary classifiers, or the DOCL strategy resulted in performance drops, confirming their necessity.
- Audio: The temporal-spectral dual-branch design significantly outperformed a baseline using only the CLS token, verifying that both temporal dynamics and spectral irregularities provide complementary forensic evidence.

5. Significance and Limitations

Significance:
The paper posits that for general AIGC detection, decision-level fusion is a more robust alternative to feature-level fusion. By decoupling modalities, the system avoids the pitfalls of assuming content alignment in scenarios where audio and video may be semantically unrelated. This approach provides a scalable solution for detecting forgeries in diverse, non-human-centric contexts.

Limitations:
The authors acknowledge two primary limitations:

Temporal Visual Modeling: The visual detector does not explicitly model fine-grained temporal dependencies (e.g., motion inconsistencies across frames), focusing instead on spatial granularities.
Fusion Heuristics: The decision-level fusion relies on heuristic rules (max-based for binary, independence assumption for four-class) rather than adaptive, learnable fusion mechanisms.

Future Work:
The authors plan to explore temporal visual modeling to capture dynamic forgery traces and develop more principled, adaptive decision-level fusion strategies (e.g., learnable calibration) to further enhance detection capabilities.