Beyond Static Artifacts: A Forensic Benchmark for Video Deepfake Reasoning in Vision Language Models

This paper introduces Forensic Answer-Questioning (FAQ), a large-scale benchmark and corresponding instruction-tuning set designed to enhance Vision-Language Models' ability to detect video deepfakes by evaluating and improving their temporal reasoning capabilities across three hierarchical levels: facial perception, temporal grounding, and forensic reasoning.

The Gist

Imagine you are a detective trying to spot a fake video. In the past, you might have looked at a single frozen photo of a face. If the eyes looked weird or the skin texture was off, you'd know it was a fake. This is what current AI models are good at: spotting static glitches in a single picture.

But deepfakes are moving pictures. They are videos. And the real "smoking gun" isn't just a weird-looking face; it's how that face moves over time. Maybe the mouth doesn't quite sync with the voice, or a blink happens too fast, or a shadow moves in the wrong direction. These are temporal inconsistencies—clues that only exist when you watch the video play out.

Current AI models are like detectives who only look at crime scene photos. They miss the clues that happen in motion. This paper introduces a new training program called FAQ (Forensic Answer-Questioning) to teach AI how to be a real video detective.

Here is how they did it, broken down into simple concepts:

1. The Problem: The "Frozen Photo" Trap

Think of current AI models as students who only studied for a test using flashcards. They are great at recognizing a specific object (like a "blurred nose") on a card. But when you hand them a movie and ask, "Is this real?", they get confused because they don't know how to watch the story unfold. They miss the dynamic clues, like a smile that starts too late or an eye that doesn't blink naturally.

2. The Solution: The "Three-Level Detective Academy"

The authors created a massive new training dataset called FAQ. Instead of just showing the AI a picture and asking "Is this fake?", they built a three-level curriculum to train the AI step-by-step, like a detective academy:

• Level 1: The "Eagle Eye" (Facial Perception)
  • The Task: Look at a specific part of the face (like the mouth) in a video frame. Is it sharp and clear, or blurry and weird?
  • The Analogy: This is like teaching a student to spot a smudge on a single page of a book. It's the basics of seeing visual flaws.
• Level 2: The "Time Traveler" (Temporal Grounding)
  • The Task: The AI has to find when and where the weirdness happens. "Between 2 seconds and 3 seconds, the nose looked pixelated."
  • The Analogy: Now the student isn't just looking at a page; they are watching the movie. They have to point to the exact moment the actor's lip-sync failed. This teaches the AI to connect the visual glitch to the time it happened.
• Level 3: The "Sherlock Holmes" (Forensic Reasoning)
  • The Task: The AI watches the whole video, gathers all the clues (the blurry eyes, the weird timing, the pixelated nose), and makes a final verdict: "This is a fake."
  • The Analogy: This is the final exam. The detective must synthesize all the tiny clues they found throughout the movie to solve the case.

3. How They Built It: From "Clicks" to "Questions"

The researchers didn't just ask an AI to guess. They started with human experts.

1. The Human Clicks: Humans watched deepfake videos and clicked on exactly where and when something looked fake (e.g., "The chin looks weird from 3:00 to 3:10").
2. The Robot Translator: They used a smart computer program to turn those human clicks into multiple-choice questions.
   • Human Input: "Click on the mouth at 4.5s, it's blurry."
   • AI Question: "Look at the mouth between 4.0s and 5.0s. Is it clear or blurry?"
3. The Distractors: They made sure the wrong answers (distractors) were tricky. They didn't just say "It's fake." They offered plausible but wrong options, forcing the AI to actually look and think rather than just guessing.

4. The Results: From Novice to Expert

They tested this new training method on various AI models.

• Before Training: The AI models were like novices. They could spot a blurry nose in a still photo but failed miserably at spotting a fake video.
• After Training (FAQ-IT): Once the models were "tuned" with this new dataset, they became experts.
  • They got much better at spotting fakes in videos they had never seen before.
  • They learned to look for the movement of the forgery, not just the static image.
  • Even when the video was compressed (lower quality), they held up better than before.

The Big Takeaway

This paper is a wake-up call for the AI world. It says: "Stop just looking at the photos; start watching the movie."

By teaching AI to reason about time and motion—not just static pixels—we can build much smarter systems to detect deepfakes. The FAQ benchmark is like a new gym for AI, where it learns to lift the heavy weights of temporal reasoning, making it a much stronger guardian against digital deception.

Technical Summary

1. Problem Statement

Current Vision-Language Models (VLMs) applied to deepfake detection primarily excel at identifying spatial artifacts (static visual inconsistencies within a single frame) but fail to leverage temporal inconsistencies (dynamic anomalies across video frames).

• Limitation of Existing Data: Most existing training datasets rely on static image extraction or limited question templates that provide only spatial discriminative information.
• The Gap: Deepfake videos often contain subtle temporal cues (e.g., flickering textures, unnatural motion trajectories) that are critical for detection but are ignored by models trained solely on static data.
• Core Challenge: How to effectively guide VLMs to discover, localize, and reason about these temporal inconsistencies through Question-Answering (QA) training data.

2. Methodology: The FAQ Benchmark

The authors propose Forensic Answer-Questioning (FAQ), a large-scale, multiple-choice question (MCQ) benchmark designed to bridge the gap between static perception and temporal reasoning.

A. Data Construction Pipeline

1. Data Collection & Curation:
   • Sourced from FaceForensics++ (FF++), collecting ~5,000 deepfake videos and 1,000 authentic videos.
   • Filtering: Used YOLOv8 to ensure high-quality facial presence (average confidence > 0.78), resulting in ~4,500 high-quality manipulated videos.
2. Spatio-Temporal Clustering:
   • Converted sparse human clicks (spatiotemporal coordinates) into coherent forgery segments using a clustering algorithm based on spatial ($\tau_s$) and temporal ($\tau_t$) proximity.
   • This generated 14,392 forged video segments with precise start/end times.
3. Landmark Extraction & Trajectory Tracking:
   • Used dlib to extract facial landmarks (eyes, nose, mouth, jaw, ears) and tracked their motion trajectories to identify specific manipulated regions.
4. Description Parsing & QA Generation:
   • Leveraged an LLM (gpt-oss-120b) to decompose raw video descriptions into "atomic annotations" (specific region + specific artifact type).
   • Generated 33,000 QA pairs across three hierarchical levels, ensuring distractors are visually and temporally plausible to prevent models from relying on linguistic shortcuts.
5. Human Verification:
   • A rigorous multi-stage human validation process ensured the correctness of answers, the plausibility of distractors, and temporal consistency.

B. Three-Level Task Hierarchy

The benchmark evaluates VLMs progressively:

1. Level 1: Facial Perception (Static): Tests basic visual quality.
   • Tasks: Region Perception (blur/clarity) and Edge Perception (boundary definition).
2. Level 2: Temporal Deepfake Grounding (Spatio-Temporal): Tests the ability to locate dynamic artifacts.
   • Tasks: Type Understanding (identifying artifact type given time/region), Region Grounding (identifying region given time/type), and Temporal Grounding (identifying time range given region/type).
3. Level 3: Forensic Reasoning (Holistic): Tests synthesis of evidence for a final verdict.
   • Tasks: Forgery Analysis (sequentially identifying artifacts without explicit cues) and Final Assessment (determining authenticity based on synthesized evidence).

3. Key Contributions

1. First Temporal-Focused QA Benchmark: FAQ is the first benchmark specifically designed to evaluate and train VLMs on temporal inconsistencies in deepfake videos, moving beyond static image analysis.
2. Comprehensive Generation Pipeline: Developed a reproducible, semi-automated pipeline that transforms static human annotations into dynamic, time-aware QA pairs with carefully designed distractors.
3. Instruction-Tuning Set (FAQ-IT): Created a corresponding instruction-tuning dataset to fine-tune VLMs, demonstrating that converting temporal cues into QA pairs is a viable training paradigm.
4. Hierarchical Evaluation: Introduced a three-level framework that isolates and measures specific capabilities: perception, grounding, and reasoning.

4. Experimental Results

The authors evaluated 13 VLMs (including Open-source like LLaVA, Qwen, InternVL, and Closed-source like GPT-4o) and fine-tuned two models (Qwen2.5-VL and LLaVA-NeXT) using FAQ-IT.

• Zero-Shot Performance: Existing VLMs struggle significantly with Level 2 and Level 3 tasks (Temporal Grounding and Reasoning), with average accuracies often below 30% for complex reasoning, highlighting the current lack of temporal forensic capability.
• Fine-Tuning Gains:
  • Models trained on the full FAQ-IT dataset showed massive improvements.
  • LLaVA-NeXT saw a 48.8% average accuracy increase.
  • Qwen2.5-VL saw a 30.8% increase.
  • Crucially, training on static-only data (FAQ-IT♠) yielded negligible or unstable gains, proving that temporal cues are the driving force behind the performance boost.
• Downstream Detection (Cross-Dataset):
  • Models fine-tuned on FAQ achieved state-of-the-art performance on standard benchmarks like FF++, Celeb-DF, DeeperForensics, and WildDeepfake.
  • On FF++, fine-tuned models achieved detection accuracies of ~73-74%, significantly outperforming baselines.
• Robustness & Analysis:
  • Compression: Performance degrades under heavy compression (c40), suggesting that aggressive compression destroys the subtle high-frequency temporal artifacts the models rely on.
  • Frame Sampling: Optimal performance was achieved with 16 frames per video; fewer frames lacked context, while more frames introduced redundancy.
  • Ablation: Mixed supervision (all three levels) outperformed single-level or multi-stage training, preventing catastrophic forgetting and preserving multimodal alignment.

5. Significance

• Paradigm Shift: The paper shifts the deepfake detection paradigm from purely spatial artifact detection to temporal reasoning, aligning VLMs more closely with how human forensic experts analyze video.
• Interpretability: By framing detection as a QA task, the model provides not just a binary "fake/real" label but also explainable reasoning (e.g., "The mouth shows texture inconsistencies between 2.0s and 3.5s").
• Foundation for Future Research: FAQ provides a standardized, high-quality benchmark to drive the development of next-generation VLMs capable of understanding dynamic video forgeries, which is critical for combating the rising threat of AIGC-generated misinformation.