SynthForensics: A Multi-Generator Benchmark for Detecting Synthetic Video Deepfakes

1. Problem Statement

The landscape of synthetic media has shifted from manipulated content (e.g., face-swapping existing videos) to purely synthetic content generated from scratch by Text-to-Video (T2V) models.

• The Gap: Existing deepfake detection benchmarks (e.g., FaceForensics++, Celeb-DF) are designed to detect manipulation artifacts (blending, warping). They are obsolete for detecting "born-synthetic" videos generated by state-of-the-art open-source T2V models (like Wan2.1, CogVideoX), which produce high-fidelity content without relying on a source video for manipulation.
• The Threat: Efficient, open-source T2V models are democratizing the ability to create indistinguishable synthetic content on consumer hardware, posing a severe threat to media authenticity. Current detectors, trained on manipulation data, fail to generalize to these new generative artifacts.

2. Methodology: The SynthForensics Benchmark

The authors introduce SynthForensics, the first human-centric benchmark specifically designed for detecting purely synthetic video.

A. Data Construction Pipeline

• Paired-Source Protocol: The benchmark uses a rigorous 1-to-1 pairing strategy. 1,363 pristine real videos from FF++ and DFD serve as anchors.
• Structured Prompt Engineering:
  • A Vision-Language Model (VideoLLaMA 3) extracts 7 structured metadata fields from each real video (short/dense description, background, style, shot type, lighting, atmosphere).
  • These fields are dynamically reassembled into model-specific prompts for five distinct T2V generators to ensure semantic consistency while varying the generation architecture.
• Generators: Five architecturally diverse, state-of-the-art open-source models were selected based on quality and diversity:
  1. Wan2.1 (14B params, DiT)
  2. SkyReels-V2 (14B params, DiT with RL)
  3. CogVideoX (5B params, Expert Transformer)
  4. Self-Forcing (1.3B params, Causal DiT, consumer-grade efficiency)
  5. MAGI-1 (24B params, Autoregressive)
• Human-in-the-Loop Validation: Every generated video underwent manual inspection for anatomical consistency, temporal artifacts, and ethical compliance. Rejected videos triggered prompt rewriting or parameter adjustment.
• Dataset Scale: 6,815 unique synthetic videos (~9.5 hours, >550k frames).
• Compression Variants: Each video is provided in four versions to test robustness: Raw, Lossless (Canonical), Light Compression (CRF23), and Heavy Compression (CRF40).

B. Evaluation Protocols

The paper evaluates 9 state-of-the-art detectors (5 frame-level, 4 video-level) across three protocols:

1. Zero-Shot: Testing pre-trained detectors (trained on manipulation data) on SynthForensics.
2. Fine-Tuning: Adapting detectors to SynthForensics to test learnability and backward compatibility.
3. Training from Scratch: Training on a subset of generators to test generalization to unseen generators (Out-of-Domain).

3. Key Contributions

1. First Human-Centric Synthetic Benchmark: A dataset of 6,815 videos from 5 diverse T2V models, addressing the lack of resources for "born-synthetic" detection.
2. Rigorous Paired-Source Design: By pairing real and synthetic videos with identical semantic descriptions, the benchmark isolates generative artifacts from semantic content, forcing detectors to learn low-level synthesis signatures.
3. Comprehensive Robustness Testing: Inclusion of four compression levels and diverse aspect ratios (landscape/portrait) to simulate real-world distribution conditions.
4. Full Reproducibility: Release of all generation metadata, including specific prompts, negative prompts, seeds, and inference parameters for every video.

4. Key Results

A. Zero-Shot Performance (The "Crisis")

Current detectors are fundamentally unprepared for synthetic video.

• Performance Collapse: When moving from manipulation benchmarks (FF++) to SynthForensics, the mean performance drop is 29.19% AUC.
• Catastrophic Failure: Some methods (e.g., RECCE) drop to 40.21% AUC, performing worse than random chance (50%).
• Compression Sensitivity: Under heavy compression (CRF40), top models lose over 30 AUC points.
• Generator Variance: Detectors perform significantly better on some generators (e.g., SkyReels-V2) than others (e.g., CogVideoX), indicating they are overfitting to specific generator signatures rather than learning general synthetic features.

B. Training and Generalization

• Learnability: The dataset is highly learnable. Fine-tuning on SynthForensics boosts detection performance to near-perfect levels (e.g., RECCE jumps from 40% to 98.42% AUC).
• Forward Generalization: Models trained from scratch on a subset of generators generalize well to unseen generators (Mean 93.81% AUC), proving that diverse training data enables the learning of universal synthetic artifacts.
• Backward Incompatibility (The Trade-off): A critical finding is the severe incompatibility between detecting synthetic video and detecting traditional manipulation.
  • Models fine-tuned or trained on SynthForensics suffer catastrophic forgetting on legacy benchmarks (FF++, DFD).
  • For example, GenConViT maintains high performance on legacy data, but models like FTCN and ProDet drop to near-random chance (~55% AUC) on traditional deepfakes after adapting to synthetic data. This suggests that manipulation artifacts and generation artifacts are distinct forensic domains requiring specialized strategies.

5. Significance and Conclusion

• Paradigm Shift: The paper demonstrates that the deepfake detection field must pivot from detecting alterations to detecting synthesis. Current SOTA detectors are fragile and ineffective against modern T2V outputs.
• Future Directions: The authors highlight that a single "universal" detector may not be feasible due to the trade-off between detecting legacy manipulations and new synthetic content. Future detectors may need modular approaches or domain-specific training.
• Community Impact: By releasing a high-quality, reproducible benchmark with full metadata, SynthForensics provides the necessary infrastructure to develop robust detection systems capable of safeguarding multimedia authenticity in the era of generative AI.

In summary, SynthForensics exposes the critical vulnerability of current forensic tools against open-source T2V models and provides the first rigorous framework to develop the next generation of detectors capable of identifying content that is "born synthetic."