Technical Summary: VideoASMR-Bench

Problem Statement

The boundary between real and synthetic videos is increasingly blurred due to the advent of frontier video generation models (VGMs) such as Sora and Veo. While current benchmarks (e.g., VBench, VideoPhy) evaluate semantic alignment and gross physical consistency (e.g., object disappearance, gravity), they lack the discriminative power to assess fine-grained audio-visual perception and sensory immersion. Existing evaluation suites often treat audio as a secondary modality or ignore it entirely, failing to capture the subtle artifacts required to distinguish high-fidelity AI-generated content from reality. This gap necessitates a high-sensitivity assessment framework to guide the next stage of video perception and generation research.

Methodology

The authors introduce VideoASMR-Bench, a benchmark designed to stress-test video models using Autonomous Sensory Meridian Response (ASMR) content. ASMR videos (e.g., tapping, brushing, whispering) are selected because they demand precise temporal-rhythmic alignment, fine-grained material interactions, and sustained sensory immersion, making them an ideal testbed for detecting subtle generation failures.

Benchmark Construction

The benchmark consists of three core components:

1. ASMR Video Suite: 1,500 high-quality real ASMR videos curated from social media (YouTube), filtered for view count (>900k), and processed to remove watermarks and artificial backgrounds. The suite is divided into "easy" (short, single-step actions in uniform scenes) and "hard" (longer, multi-step activities in complex environments) levels.
2. ASMR Prompt Suite: Text descriptions generated for each video, detailing environments, subjects, actions, temporal dynamics, and corresponding sounds. These serve as conditioning signals for synthesis.
3. ASMR Image Suite: High-quality reference images (first frames of the real videos) used as conditioning inputs for image-to-video generation.

Using these suites, the authors synthesized 2,235 fake ASMR videos across nine state-of-the-art VGMs (including Sora2, Veo3.1, HunyuanVideo, Wan2.2, and OpenSora-V2) under four different generation settings:

• Image-to-Video (Img2Vid)
• Text-to-Video (Text2Vid)
• Text-to-Image-to-Video (Text2Img2Vid)
• Image-only generation with minimal prompts

Evaluation Framework

The paper proposes an adversarial understanding–generation evaluation where VGMs attempt to produce realistic fake videos to fool Video Understanding Models (VLMs), while VLMs attempt to detect them.

• VLM Evaluation: 11 VLMs (including GPT-4o, Gemini-3-Pro, and various Qwen/GLM models) are evaluated on their ability to classify videos as real or fake. Models are required to provide step-by-step reasoning (Chain-of-Thought) before outputting a binary decision.
• VGM Evaluation: VGMs are scored based on the proportion of their generated videos that are incorrectly classified as real by the VLMs (Realism Score).
• Human Baseline: 10 human annotators (mix of experts and non-experts) provide ground truth labels.

Key Results

The evaluation reveals significant limitations in current state-of-the-art models regarding fine-grained ASMR detection:

1. VLMs Struggle with Detection: Even top-tier proprietary VLMs fail to reliably detect AI-generated ASMR videos.

   • Gemini-3-Pro, the best-performing VLM, achieved an average accuracy of 70.57%, significantly lagging behind the human baseline of 87.50%.
   • Most open-source models performed poorly, with many achieving accuracy below 60%.
   • VLMs exhibit a strong bias toward classifying videos as "real," misclassifying up to ~71% of fake videos.

2. VGMs Produce Indistinguishable Content for Machines: Current frontier VGMs can generate ASMR videos that are largely indistinguishable from real ones to VLMs.

   • Veo3.1-fast was the most successful generator, with only 12.54% of its videos correctly identified as fake by the ensemble of VLMs.
   • Open-source models like HunyuanVideo and Wan2.2 also achieved low detection rates (14.77% and 16.10% respectively), outperforming Sora2 in some metrics when watermarks were removed.

3. Critical Observations:

   • Watermark Reliance: VLMs often rely on watermarks as a "detection shortcut." When Sora2 watermarks were removed, the detection accuracy of models like Gemini-2.5-Pro and GPT-5 dropped by approximately 30 percentage points.
   • Audio Importance: Incorporating audio cues significantly improved detection accuracy (average gain of ~5 points), highlighting that current VGMs often fail to achieve high-fidelity audio-visual synchronization, particularly in ASMR contexts.
   • Reasoning and Preference: Explicit reasoning traces and preference-based evaluation (comparing a real vs. fake video) generally improved detection accuracy and stability compared to direct binary judgment.

Significance and Contributions

The paper claims the following contributions:

• First ASMR Domain Study: To the best of the authors' knowledge, this is the first work to systematically investigate both video generation and understanding within the ASMR domain, utilizing it as a rigorous testbed for fine-grained physical realism and sensory immersion.
• Scalable Benchmark: VideoASMR-Bench provides a high-quality, diverse dataset (1,500 real videos) paired with an extensible prompt and image suite, allowing the benchmark to scale dynamically with future video models via conditional synthesis.
• Adversarial Framework: The introduction of an adversarial evaluation protocol spanning 11 VLMs and 9 VGMs demonstrates that while VGMs are rapidly approaching human-level realism in specific sensory domains, VLMs remain unreliable detectors, often relying on superficial cues like watermarks rather than intrinsic perceptual analysis.

The authors conclude that while current models can fool machines, they still fail to fully deceive human annotators, suggesting that the next frontier in video generation research lies in achieving true sensory immersion and perfect audio-visual coherence.