ActivityForensics: A Comprehensive Benchmark for Localizing Manipulated Activity in Videos

1. Problem Definition

The paper addresses the emerging challenge of Temporal Forgery Localization, specifically focusing on Activity-Level Forgeries.

• Context: While existing benchmarks (e.g., ForgeryNet, Lav-DF) focus on appearance-level forgeries (face swapping, object removal), recent advances in video generation and editing have enabled activity-level manipulations. These alter human actions to distort event semantics (e.g., changing a neutral gesture into a misconduct gesture) while maintaining high visual consistency with the surrounding video.
• The Gap: Current methods often rely on architectures designed for Temporal Action Localization, which prioritize high-level semantic understanding. However, detecting activity forgeries requires sensitivity to subtle, low-level temporal and visual artifacts (texture irregularities, motion discontinuities) rather than semantic cues. Existing datasets lack the scale and diversity to train models for this specific task, and current models struggle to generalize across different manipulation mechanisms.

2. Methodology

A. Dataset Construction: ActivityForensics

The authors introduce ActivityForensics, the first large-scale benchmark for localizing manipulated activities.

• Scale: Contains over 6,000 forged video segments seamlessly blended into authentic contexts.
• Grounding-Assisted Data Construction: To overcome the labor-intensive nature of manual forgery creation, they propose an automated pipeline:
  1. Grounding: Uses video captioning and temporal grounding models to extract activity descriptions and their precise temporal boundaries.
  2. Manipulation: Employs Large Language Models (LLMs) to semantically alter the activity descriptions (e.g., changing "waving" to "giving a thumbs-up").
  3. Synthesis: Conditions video generation (e.g., Wan, Scifi, FCVG) and editing (e.g., VACE, LTX) models on the manipulated descriptions and grounding information to synthesize the forged segments.
  4. Integration: Seamlessly merges the generated segments back into the original video, ensuring high visual and temporal realism.
• Diversity: The dataset covers diverse manipulation paradigms (text-driven, pose-driven, frame interpolation) and varying segment durations.

B. Proposed Model: Temporal Artifact Diffuser (TADiff)

The authors propose TADiff, a simple yet effective baseline designed to mitigate semantic bias and amplify artifact cues.

• Motivation: Standard action localization models overfit to semantic content. TADiff aims to force the model to learn features invariant to semantics but sensitive to artifacts.
• Architecture: Built upon ActionFormer (a multi-scale transformer encoder).
• Mechanism:
  1. Forward Process (Noise Injection): Injects stochastic Gaussian noise into the temporal feature space. This perturbs the representation away from the semantic manifold, breaking the coupling between high-level semantics and the features.
  2. Reverse Process (Denoising): Uses a lightweight temporal convolutional denoiser with Feature-wise Linear Modulation (FiLM) and Denoising Diffusion Implicit Model (DDIM) updates.
  3. Objective: The model is trained to denoise the features to recover an "artifact-sensitive" signal. The loss function focuses solely on localization (classification confidence and boundary regression), not content reconstruction.
• Effect: This process regularizes the feature manifold, encouraging the network to rely on subtle temporal inconsistencies rather than semantic context.

3. Key Contributions

1. New Task & Dataset: Introduction of ActivityForensics, the first large-scale dataset specifically for manipulated activity localization, constructed via a novel grounding-assisted automated pipeline.
2. Comprehensive Evaluation Protocols: Establishment of three evaluation settings to test robustness:
   • Intra-domain: Training and testing on the same manipulation models.
   • Open-world: Training on known models, testing on unseen commercial models (e.g., Vidu).
   • Cross-domain: Transferring between different generation paradigms (e.g., text-driven vs. pose-driven).
3. Novel Baseline (TADiff): A diffusion-based feature regularizer that effectively suppresses semantic bias and enhances sensitivity to low-level forgery artifacts, outperforming state-of-the-art action localization and forgery detection models.

4. Experimental Results

The paper benchmarks TADiff against state-of-the-art methods (ActionFormer, UMMAFormer, DiGIT) across the three evaluation settings.

• Intra-Domain Performance: TADiff achieves the highest Average Precision (AP) and Average Recall (AR). Notably, it improves AP@0.95 (strict boundary localization) by +9.78% over the ActionFormer baseline, demonstrating superior temporal precision.
• Open-World Performance: TADiff maintains strong performance on unseen commercial models (Vidu), showing a +5.82 AP gain. The diverse training data prevents the performance drop typically seen in domain shifts.
• Cross-Domain Generalization: In transfer tasks (e.g., A→B and B→A), TADiff consistently outperforms baselines, proving robustness against shifts in manipulation mechanisms.
• Ablation Studies:
  • Both noise injection and denoising modules are necessary; using both yields the best results.
  • Optimal performance is achieved with 3 denoising steps for intra-domain and 4 steps for open-world settings.
• Feature Analysis (t-SNE): Visualizations show that without TADiff, real and forged features overlap significantly (Fisher score: 1.74). With TADiff, the classes separate clearly (Fisher score: 2.64), confirming the reduction of semantic bias and enhancement of artifact discriminability.

5. Significance

• Advancing Video Forensics: The work shifts the focus from static appearance forgery to dynamic activity forgery, a critical area as generative AI becomes more capable of altering human behavior in videos.
• Digital Integrity: By providing a rigorous benchmark and a robust detection method, the paper contributes to the development of tools necessary to preserve media authenticity and public trust in the era of deepfakes.
• Methodological Insight: The introduction of diffusion-based feature regularization offers a new paradigm for handling tasks where semantic cues are misleading, suggesting potential applications in other anomaly detection domains.

The dataset and code are publicly available at https://activityforensics.github.io.