VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

1. Problem Statement

Despite rapid advancements in Multimodal Large Language Models (MLLMs) for video understanding, current evaluation benchmarks suffer from two critical limitations:

1. Inflated Scores: High accuracy on existing benchmarks often masks deficiencies in fine-grained visual understanding and complex reasoning. Models may guess correct answers based on language priors or coarse semantics without truly "seeing" the video.
2. Lack of Evidence Verification: Standard evaluations measure answer correctness but fail to verify whether the model's prediction is grounded in precise spatio-temporal evidence (i.e., identifying the specific time intervals and spatial regions that support the answer). This allows for "hallucinated" correctness where the answer is right, but the reasoning is flawed.

The authors argue that as AI enters its "second half," progress must be measured not just by QA accuracy, but by the validity and verifiability of the evidence underlying predictions.

2. Methodology

A. Dataset Construction: VideoZeroBench

The authors introduce VideoZeroBench, a challenging, hierarchical benchmark designed to test the limits of long-video understanding.

• Scale & Content: Comprises 500 manually annotated open-ended questions across 138 long videos (average duration: 667.1 seconds, ranging from 30s to 3037s).
• Domains: Covers 13 diverse domains including Sports, Gaming, Instructional, Daily Vlogs, Driving, News, etc.
• Annotation:
  • Questions: Designed to be precise and verifiable (e.g., specific numbers, short phrases), avoiding multiple-choice guessing. They require subtle clue discovery and integration of non-salient evidence.
  • Evidence: Each question is paired with ground-truth temporal intervals (start/end timestamps) and spatial bounding boxes (normalized coordinates on key frames).
  • Atomic Capabilities: Questions are categorized into 11 atomic abilities, grouped into:
    • Detailed Perception: Counting, small-object perception, OCR, action recognition.
    • Spatio-Temporal Reasoning: Spatial orientation, object tracking, scene transitions, multi-segment dependency.
    • Semantic & Cross-Modal: World knowledge, audio perception.

B. Five-Level Hierarchical Evaluation Protocol

To disentangle answering generation, temporal grounding, and spatial grounding, the authors propose a progressive 5-level evaluation framework:

• Level-1 (Answer with Evidence): Model receives the video + question + full temporal and spatial evidence. Measures pure evidence integration and reasoning capability.
• Level-2 (Answer with Temporal Evidence): Model receives video + question + temporal intervals only. Tests fine-grained spatial perception without spatial hints.
• Level-3 (Standard QA): Model receives video + question without any hints. Standard end-to-end QA accuracy.
• Level-4 (Correct Answer + Temporal Grounding): Model must answer correctly AND identify the correct temporal intervals. Accuracy = $P(\text{Correct Answer}) \times P(\text{Temporal IoU} > 0.3)$.
• Level-5 (Correct Answer + Spatio-Temporal Grounding): The most stringent level. Model must answer correctly, identify temporal intervals, AND provide accurate spatial bounding boxes. Accuracy = $P(\text{Correct Answer}) \times P(\text{Temporal IoU} > 0.3) \times P(\text{Visual IoU} > 0.3)$.

3. Key Contributions

1. First Evidence-Grounded Video Benchmark: VideoZeroBench is the first benchmark to rigorously verify spatio-temporal evidence for long-video QA, moving beyond simple answer correctness.
2. Hierarchical Evaluation Framework: The 5-level protocol allows researchers to diagnose specific failure modes (e.g., is the model failing to find the right time, the right object, or to reason correctly?).
3. High-Quality Manual Annotation: Unlike many benchmarks relying on automated generation, this dataset features human-verified questions and evidence, ensuring high difficulty and logical consistency.
4. Comprehensive Analysis: The paper provides deep insights into model failures across atomic abilities, video categories, evidence span lengths, and input modalities (audio vs. visual).

4. Experimental Results

The authors evaluated 17 state-of-the-art models, including proprietary giants (Gemini-3-Pro, GPT-5.2, Seed-2.0) and open-source models (Qwen3-VL, InternVL3.5).

• Performance Drop: There is a drastic performance decline as constraints tighten.
  • Level-3 (Standard QA): The best model, Gemini-3-Pro, achieves only 17.0% accuracy.
  • Level-5 (Full Grounding): When requiring correct answers plus accurate spatio-temporal grounding, the best model drops to 1.0%. Most models score 0%.
• Key Findings:
  • Hallucination is Common: Models often provide correct answers without identifying the supporting evidence, indicating they are guessing or relying on priors rather than visual grounding.
  • Spatial Bottleneck: Fine-grained spatial perception (small objects, spatial orientation) is a major weakness. Even when temporal hints are provided (Level-2), accuracy remains low.
  • Counting & Small Objects: These tasks are particularly difficult, with most models scoring below 8-12%.
  • Thinking-with-Videos: "Agentic" models that use iterative reasoning (e.g., VideoChat-R1.5) show marginal improvements but fail to solve the grounding bottleneck; they still cannot locate precise evidence.
  • Human Gap: Humans achieve 67.6% accuracy on a subset, highlighting a massive gap between current MLLMs and human-level video understanding.

5. Significance and Conclusion

VideoZeroBench reveals that current video MLLMs are far from achieving reliable, evidence-based reasoning. The "second half" of AI progress requires shifting focus from surface-level answer accuracy to verifiable spatio-temporal grounding.

• Implications: The results suggest that simply scaling models or adding "thinking" loops is insufficient without solving the fundamental problem of precise spatio-temporal localization in long, cluttered videos.
• Future Directions: Research must prioritize detailed visual grounding, fine-grained perception, and mechanisms that force models to justify answers with specific visual evidence.
• Availability: The benchmark and code are made publicly available to serve as a rigorous diagnostic tool for the next generation of video MLLMs.

In summary, VideoZeroBench acts as a "reality check," demonstrating that while models can sometimes guess the right answer, they currently lack the ability to reliably "see" and "point to" the evidence that proves they understand the video.