Referee: Reference-aware Audiovisual Deepfake Detection

1. Problem Statement

Deepfake technology, driven by advanced generative AI, poses significant societal risks by creating high-fidelity multimodal content (video and audio) that is difficult to distinguish from reality. Existing detection methods face critical limitations:

• Overfitting to Artifacts: Current approaches often rely on detecting transient spatiotemporal artifacts (e.g., boundary inconsistencies, frequency anomalies, or lip-sync errors). These methods struggle to generalize to unseen manipulation techniques or high-fidelity generators.
• Vulnerability to Conditions: Methods relying on fine-grained visual cues (like lip movements) are easily degraded by occlusions (hands, microphones), low resolution, or poor lighting.
• Inefficiency of Identity-Based Methods: Prior identity-based approaches either rely on visual-only comparisons (ignoring audio) or require excessively long reference footage (e.g., >5 minutes) and massive external datasets (e.g., millions of utterances) to function, making them impractical for real-world deployment.
• Lack of Cross-Modal Integration: Existing multimodal methods often use naive feature concatenation, failing to capture intricate cross-modal dependencies between voice and facial appearance.

2. Methodology: The Referee Framework

The authors propose Referee, a novel framework that shifts the detection paradigm from finding transient artifacts to verifying cross-modal biometric consistency using a single-shot reference example. The architecture consists of three main components:

A. Audiovisual Feature Extraction

• Encoders: Utilizes pre-trained audiovisual encoders (based on Synchformer) to extract modality-specific representations.
• Segmentation: Input streams are divided into fixed-length segments. Visual and audio sequences are concatenated with a learnable modality-separation token to form a joint representation for both the Target (suspect) and Reference (ground truth) videos.

B. Identity Bottleneck (IDB)

• Goal: To isolate speaker identity from confounding variables like speech content, facial expressions, and head poses.
• Mechanism: The IDB employs a set of learnable identity queries ($Q_{ID}$) that interact with the joint audiovisual features via a transformer-based cross-attention mechanism.
• Output: This process compresses the complex audiovisual sequence into a unified, compact set of identity tokens ($T_{ID}$) that represent the speaker's biometric signature, effectively filtering out transient noise. Weight sharing is used between the target and reference branches to ensure consistent encoding.

C. ID Matching Module & Deepfake Classification

• Cross-Attention Matching: The identity tokens from the target are aligned with those from the reference using a cross-attention mechanism. The target tokens act as the Query, and reference tokens act as Key/Value.
• Reference-Aware Tokens: This generates "reference-aware" identity tokens ($\bar{T}_{ID}$) that explicitly encode the similarity or discrepancy between the target and the reference.
• Dual-Task Learning:
  1. Identity Verification (Auxiliary Task): An MLP head predicts whether the target and reference belong to the same identity. This is supervised by an Identity Matching Loss ($L_{ID}$), treating any forged content as a distinct identity from the reference.
  2. Deepfake Classification (Main Task): The refined identity tokens are concatenated with the target's audiovisual features and a [CLS] token. This sequence is processed by an AV-Transformer to jointly reason about lip-speech synchronization and identity integrity. A final classifier outputs the deepfake probability.
• Total Loss: $L_{all} = L_{RF}$ (Deepfake classification) + $L_{ID}$ (Identity matching).

3. Key Contributions

1. Reference-Aware Detection: Introduces a framework that leverages a single-shot reference example to detect identity inconsistencies, moving beyond reliance on low-level synthesis artifacts.
2. Identity Bottleneck (IDB): Proposes a novel module with learnable queries to extract robust, cross-modal identity features, effectively disentangling identity from transient factors.
3. Cross-Modal Biometric Matching: Implements a mechanism to explicitly correlate voice and facial appearance, enabling the model to detect forgeries that preserve lip-sync but violate biometric identity.
4. Data Efficiency: Demonstrates that high-performance detection does not require million-scale external datasets; the model is trained effectively with only ~14k authentic clips.

4. Experimental Results

The model was evaluated on FakeAVCeleb, FaceForensics++ (FF++), and KoDF (Korean Deepfake dataset).

• Cross-Dataset Generalization:
  • On KoDF (cross-lingual, unseen techniques), Referee achieved a state-of-the-art 99.4% AUC and 99.4% AP, significantly outperforming previous methods (e.g., ICSAV at 99.2% AUC).
  • On FF++, it achieved 79.78% AUC and 91.00% AP, outperforming other reference-based methods that rely on massive datasets like VoxCeleb2.
• Intra-Dataset Performance:
  • On FakeAVCeleb, Referee achieved 99.2% Accuracy and 99.7% AUC, surpassing strong baselines like AVFF and FRADE.
• Robustness Analysis:
  • Spatial Perturbation: Maintained >98.9% accuracy even with 90% random cropping of the video.
  • Reference Length: Achieved 99.71% AUC with only a 1-second reference clip, proving high data efficiency.
  • Cross-Lingual: Showed superior resilience in zero-shot cross-lingual scenarios where other reference-based models failed.

5. Significance

Referee represents a paradigm shift in deepfake detection:

• From Artifacts to Biometrics: It moves the focus from detecting ephemeral generation artifacts (which evolve rapidly) to verifying fundamental biometric consistency (which is harder to forge without breaking identity coherence).
• Generalization: By anchoring detection to cross-modal identity priors, the model generalizes effectively to unseen manipulation methods, languages, and degradation conditions.
• Practicality: The ability to work with short reference clips and without massive pre-training datasets makes it highly suitable for real-world deployment in forensic and security applications.

In conclusion, Referee establishes a new frontier in audiovisual forensics by demonstrating that explicitly correlating reference-based biometric priors is the key to achieving reliable, generalized, and data-efficient deepfake detection.