💬 NLP

Revealing the Truth with ConLLM for Detecting Multi-Modal Deepfakes

This paper introduces ConLLM, a hybrid framework that combines pre-trained model embeddings with contrastive learning and large language model reasoning to overcome modality fragmentation and shallow inter-modal reasoning, thereby significantly improving detection accuracy across audio, video, and audio-visual deepfakes.

Original authors: Gautam Siddharth Kashyap, Harsh Joshi, Niharika Jain, Ebad Shabbir, Jiechao Gao, Nipun Joshi, Usman Naseem

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gautam Siddharth Kashyap, Harsh Joshi, Niharika Jain, Ebad Shabbir, Jiechao Gao, Nipun Joshi, Usman Naseem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Too-Perfect" Lie

Imagine a world where someone can create a video of a politician saying something they never said, or a voice recording of a CEO authorizing a fake transaction. These aren't just bad photos; they are deepfakes—hyper-realistic lies made by computers.

The paper explains that current "lie detectors" have two main blind spots:

The "Silo" Problem (Modality Fragmentation): Imagine a security guard checking a person's ID (face) and listening to their voice separately. The guard might say, "The face looks real!" while a different guard says, "The voice sounds fake!" Because they aren't talking to each other, they miss the contradiction. Current AI models often do this too—they look at the video and listen to the audio in isolation, failing to see the big picture.
The "Surface-Level" Problem (Shallow Reasoning): Imagine a guard who only checks if the lips are moving in time with the sound. If the lips match the sound, the guard says, "All clear!" But what if the person is saying something that makes no sense for their personality or the situation? Current models often miss these subtle, logical inconsistencies because they don't "think" deeply enough about the meaning.

The Solution: ConLLM (The "Super-Team" Detective)

The authors propose a new system called ConLLM. Think of it as a high-tech detective team with a specific two-step process to catch these sophisticated liars.

Step 1: The Specialized Scouts (Embedding Extraction)

First, the system sends the video and audio to different "scouts" who are experts in their specific fields.

The Audio Scout: Uses a model called XLS-R to listen to the voice.
The Video Scout: Uses a model called VideoMAE to watch the face and body movements.
The Duo Scout: Uses VATLM to watch how the voice and face work together.

These scouts don't just guess; they take detailed notes (called "embeddings") about what they see and hear. This ensures no detail is lost before the team meets up.

Step 2: The Roundtable Discussion (Refinement)

This is where the magic happens. The notes from the scouts are brought to a "roundtable" where two powerful tools work together:

The "Truth Aligner" (Contrastive Learning): Imagine a game of "spot the difference." This tool forces the audio notes and video notes to line up perfectly if they are real. If the audio says "happy" but the video shows a "sad" face, the tool pushes them apart, flagging them as a mismatch. This fixes the "Silo Problem."
The "Big Brain" (Large Language Model - LLM): This is the team's smartest member, similar to the AI behind chatbots like GPT. It doesn't just look at the data; it reasons about it. It asks questions like, "Does this person's speech pattern match their known behavior?" or "Is the story they are telling logically consistent?" This fixes the "Surface-Level Problem" by catching semantic lies that simple pattern matching misses.

The Results: Catching More Liars, Faster

The paper claims this new team is significantly better than previous detectives:

Audio: It reduced the error rate in catching fake voices by up to 50%.
Video: It improved the accuracy of spotting fake videos by up to 8%.
Combined (Audio-Visual): It boosted accuracy by about 9% when checking both voice and video together.

Why is it so good?
The authors ran tests to see what made the team work. They found that:

Using the specialized "scouts" (Pre-Trained Models) was crucial. Without them, the system was much worse.
The "Big Brain" (LLM) reasoning was the secret sauce that caught the subtle, logical lies.
The system is also efficient. It runs faster and uses less computer memory than other massive AI models, making it more practical for real-world use.

The Bottom Line

ConLLM is a new way to detect deepfakes that stops treating the eyes and ears as separate things. Instead, it uses a team of specialists to gather evidence, a "truth aligner" to check for contradictions, and a "big brain" to reason through the logic of the lie. The result is a system that is much harder to fool, catching both the obvious fakes and the clever, subtle ones.

Technical Summary: Revealing the Truth with ConLLM for Detecting Multi-Modal Deepfakes

1. Problem Statement

The rapid proliferation of deepfake technology, driven by generative modeling and multimodal synthesis (e.g., GANs, VAEs, transformer-based TTS), poses severe threats to social and political stability by enabling hyper-realistic synthetic media. While existing detection methods have made progress, they face two critical limitations that hinder generalization and accuracy:

Modality Fragmentation: Current approaches often process audio, video, and text in isolation or with poor alignment. This leads to contradictory cues (e.g., a realistic face paired with robotic speech) and poor generalization across diverse and adversarial deepfake modalities.
Shallow Inter-Modal Reasoning: Existing multimodal detectors frequently rely on surface-level fusion strategies. They fail to model complex semantic relationships and subtle inconsistencies across modalities, resulting in missed detections where lip-sync matches audio but the speech contradicts known behavioral patterns.

2. Methodology: ConLLM Framework

To address these challenges, the authors propose ConLLM (Contrastive Learning with Large Language Models), a hybrid framework designed for robust multimodal deepfake detection across audio, video, and audio-visual modalities. The architecture follows a two-stage modular design:

Stage 1: Embedding Extraction

The framework utilizes Pre-Trained Models (PTMs) to extract modality-specific embeddings without premature fusion, preserving unique characteristics of each modality:

Audio: XLS-R converts speech signals into robust representations.
Video: VideoMAE captures spatiotemporal features.
Audio-Visual: VATLM jointly models both modalities to encode cross-modal interactions.
Projection: These high-dimensional embeddings are mapped via modality-specific projection functions ( $\psi_m$ ) into a shared latent space ( $R^{d_s}$ ), ensuring compatibility for subsequent alignment.

Stage 2: Cross-Modal Refinement

This stage employs two mechanisms to refine the aligned embeddings:

Contrastive Learning: A contrastive loss function ( $L_{contrast}$ ) is applied to align embeddings. It minimizes the distance between embeddings of authentic samples across modalities while maximizing the distance between manipulated ones. This directly addresses modality fragmentation by forcing semantically consistent representations closer in the latent space.
LLM-Based Reasoning: The aligned embeddings are processed by a GPT-style transformer module ( $\phi_{Transformer}$ ). Using multi-head self-attention, this module captures inter-modal semantic dependencies and contextualizes the embeddings. This step refines the features to detect subtle semantic inconsistencies that shallow fusion methods miss.

Finally, the refined embeddings are concatenated and passed through a classification head to produce the final prediction (Real vs. Manipulated).

3. Key Contributions

The paper outlines three primary contributions:

Novel Architecture: The proposal of ConLLM, a two-stage hybrid framework that integrates modality-specific PTM extraction, contrastive alignment, and LLM-based semantic refinement to simultaneously tackle modality fragmentation and shallow inter-modal reasoning.
Comprehensive Evaluation: A rigorous assessment of ConLLM across three modalities (audio, video, and audio-visual) using six widely-used benchmarks, demonstrating its generalizability.
Performance Gains: The framework outperforms state-of-the-art baselines, achieving significant reductions in error rates and improvements in accuracy. Ablation studies confirm that the specific components (PTMs, contrastive learning, and LLM reasoning) contribute consistently to these gains.

4. Experimental Results

The authors evaluated ConLLM on six datasets: ASVSpoof 2019 (ASV), DECRO (D-E, D-C), Celeb-DF (CDF), WildDeepfake (WD), FakeAVCeleb (FAFC), and DeepFake Detection Challenge (DFDC).

Performance Metrics

Audio Deepfakes: ConLLM reduced the Equal Error Rate (EER) by up to 50% compared to prior works. On the ASV dataset, it achieved an EER of 0.21%, and 0.01% on the D-E dataset.
Video Deepfakes: The model improved video accuracy by up to 8%, achieving 98.75% accuracy on CDF and 85.00% on WD, with AUC scores of 99.98% and 94.50% respectively.
Audio-Visual Tasks: ConLLM achieved approximately 9% accuracy gains over baselines, reaching 98.75% accuracy on FAFC and 96.50% on DFDC.

Efficiency and Generalization

Computational Efficiency: ConLLM demonstrated superior efficiency compared to large multimodal models (e.g., AudioFlamingo, Video-LLM). It reduced inference time, memory usage, and FLOPs significantly (e.g., 55ms inference time on ASV vs. 130ms for AudioFlamingo 3).
Cross-Lingual Generalization: The model showed strong adaptability across languages, achieving low EERs (1.5% and 2.0%) in cross-lingual transfer tasks (Chinese to English and vice versa).
Ablation Studies:
- Replacing PTMs with traditional CNNs degraded performance significantly (e.g., EER rose from 0.21% to 0.41% on ASV).
- Removing contrastive learning increased EERs and decreased accuracy, confirming its role in alignment.
- Removing LLM-based refinement reduced the model's ability to capture semantic nuances.
- Concatenation-based fusion outperformed non-concatenation strategies (e.g., averaging).

5. Significance and Claims

The paper claims that ConLLM represents a significant advancement in deepfake detection by moving beyond unimodal analysis and shallow fusion. By decoupling feature extraction from semantic integration, the framework guarantees contextual robustness and discriminative power.

The authors emphasize that the integration of contrastive learning and large-scale pretrained language models allows the system to detect fine-grained semantic inconsistencies that previous methods miss. While acknowledging limitations regarding dependency on training data diversity and computational demands for real-time deployment, the study positions ConLLM as a practical, high-performance solution for mitigating the risks of manipulated media in real-world scenarios. The code and data are made available to support further research in this domain.