Technical Summary: Advancing Reliable Synthetic Video Detection (SAFE Challenge)

Problem Statement

The rapid advancement of generative video technologies (diffusion, transformer, and adversarial architectures) has blurred the line between authentic and synthetic media. While these systems offer creative opportunities, they pose significant challenges for media provenance, journalism, and fraud prevention. Existing benchmarks often rely on fixed, openly available datasets that are susceptible to overfitting and are frequently limited to facial or human-centric content. Furthermore, many detectors fail to generalize to unseen generation techniques or remain robust against common post-processing operations (e.g., compression, resizing) encountered during real-world online dissemination. The SAFE (Synthetic Video Detection) Challenge was designed to address these gaps by establishing a fully blind, model-submission-based evaluation framework that simulates real-world forensic constraints.

Methodology

Challenge Design and Infrastructure

The SAFE Challenge was hosted on the Hugging Face platform over a 90-day period, culminating at the ICCV 2025 APAI Workshop. The competition utilized a fully blind evaluation protocol:

• No Data Release: Participants did not have access to the test dataset.
• Model Submission: Teams submitted containerized scripts to a private repository.
• Sequestered Evaluation: Submissions were executed in an offline, containerized environment on identical hardware (NVIDIA L40S GPU, 48GB VRAM) to ensure fairness and reproducibility.
• Blindness: Participants had no knowledge of the specific video sources, generators, or post-processing applied to the private test set.

Dataset Construction

The dataset comprised 6,000 video samples totaling 20 hours of content, split into public (for leaderboard feedback) and private (for final evaluation) sets.

• Real Content: Sourced from 21 diverse categories including wildlife trail cams, drone footage, social media, lectures, dashcams, and pre-digital archive footage. Audio was stripped, and clips were truncated to 5–10 seconds.
• Synthetic Content: Generated using 13 state-of-the-art models (10 Text+Image-to-Video, 1 Audio-Driven, 2 Physical Environment Simulators).
  • Alignment: To ensure semantic alignment, an automated pipeline using Qwen2.5-VL extracted representative frames and generated text prompts from the real videos. These identical inputs were fed to all generators, isolating generator performance from content variability.
• Task 1 (Baseline): Detection of unmodified synthetic vs. real videos.
• Task 2 (Robustness): Detection of synthetic vs. real videos after applying 14 post-processing techniques, including up/downscaling, H.264/H.265/AV1 compression, motion blur, noise, frame interpolation, and "laundering" (recording a screen with a camera).

Evaluation Metrics

The primary metric was Balanced Accuracy (BAC), defined as the average of True Positive Rate (TPR) and True Negative Rate (TNR). Area Under the Curve (AUC) was also reported. The public leaderboard provided anonymized source-specific metrics to prevent overfitting, while the final ranking relied on the private set.

Key Contributions

1. Realistic, Blind Benchmark: Unlike prior open-dataset efforts, SAFE enforced a black-box testing scenario where models were evaluated on sequestered data without access to metadata or file-format cues, closely mirroring operational forensic workflows.
2. Dual-Task Evaluation: The challenge simultaneously measured generalization (Task 1: unseen generators and sources) and robustness (Task 2: resilience to common online transformations).
3. Comprehensive Dataset: The inclusion of 13 diverse high-quality generators and 21 real-world sources (ranging from 240p to 4K) provided a broader semantic domain than previous face-centric benchmarks.
4. Empirical Analysis of SOTA Detectors: The paper aggregates findings from over 600 submissions by 12 teams, identifying architectural trends and performance bottlenecks.

Results

Task 1: Generalization Performance

• Strong Baseline Performance: Top teams achieved high AUC scores on unmodified content (e.g., GRIP-UNINA: 0.93, ISPL-Realynx: 0.92), significantly outperforming the baseline (0.74).
• Cross-Generator Transferability: Detectors trained on specific generators (e.g., DASH team training only on WAN2.2) demonstrated surprising ability to generalize to unseen generators, suggesting that architecture and training strategy may be more critical than exhaustive generator coverage.
• Source Variability: Performance remained relatively consistent across diverse real-world sources (public vs. private splits), though specific sources like "dashcam" and "TV talk shows" showed lower detection rates, likely due to unique forensic or semantic characteristics.

Task 2: Robustness to Post-Processing

• Significant Degradation: All teams experienced a performance drop of 5 to 25 AUC points when videos underwent post-processing compared to unmodified content.
• Vulnerability Factors:
  • Compression: H.264, H.265, and AV1 compression caused substantial drops (e.g., from ~0.88 to ~0.77).
  • Noise and Blur: Gaussian blur and additive noise further reduced AUC to ~0.74 and ~0.72, respectively.
  • Laundering: The "real camera" technique (digital-to-analog-to-digital re-recording) proved most difficult, dropping AUC to 0.62. Interestingly, this operation increased false alarm rates, causing laundered real videos to score closer to synthetic videos.

Architectural Insights

Analysis of top-performing teams (Table 9) revealed common successful strategies:

• Backbones: Utilization of large, pre-trained vision backbones (specifically DINOv2) designed for general computer vision tasks.
• Augmentation: Widespread use of autoencoding augmentation, where real videos are reconstructed via VAEs to create synthetic training examples that are perfectly content-matched to real inputs.
• Training Data: Some top teams achieved high performance with minimal overlap between their training data and the challenge generators, reinforcing the transferability of learned features.

Significance and Claims

The paper claims that the SAFE Challenge provides a structured, multi-factor test of synthetic video detection under controlled yet realistic blind conditions. The findings highlight measurable progress in cross-generator generalization, demonstrating that modern detectors can effectively distinguish synthetic content when unmodified. However, the study also underscores persistent vulnerabilities to post-processing artifacts, indicating that current methods are not yet robust enough for the noisy, compressed reality of online video dissemination.

The authors conclude that while detection of unmodified synthetic video is promising, the field must address the degradation caused by standard online transformations (compression, re-encoding) to achieve reliable, large-scale forensic utility. The challenge design and results offer a baseline for future research into robust, generalizable detection architectures.