Technical Summary: VidNum-1.4K

Problem Statement

Current Vision-Language Models (VLMs) have achieved significant progress in video understanding, yet a critical scientific question remains: do these models genuinely construct a stable "world model" to understand dynamic scenes, or do they merely rely on memorized dataset priors and superficial correlations? While existing benchmarks address video understanding, they are fragmented regarding numerical reasoning. Standard VideoQA datasets (e.g., TGIF-QA, NExT-QA) treat counting as a minor subset of semantic tasks without explicit quantitative difficulty hierarchies. Conversely, dedicated repetition-counting benchmarks (e.g., QUVA Repetition, Countix) focus on cycle estimation for repetitive actions as regression tasks, failing to assess multi-step logical deductions, cross-event consistency, or reasoning across frequent shot cuts. There is a distinct lack of a comprehensive evaluation protocol for compositional numerical reasoning in video, where models must perform arithmetic operations, comparisons, and logical deductions grounded in temporal evidence.

Methodology: VidNum-1.4K Benchmark

To address this gap, the authors introduce VidNum-1.4K, a comprehensive benchmark comprising 1,379 strictly human-annotated multiple-choice questions (MCQs) based on independent video clips.

Data Construction

The benchmark was constructed via a rigorous, fully manual, multi-stage pipeline involving approximately 300 full-time annotators divided into four isolated groups to ensure data quality and prevent bias:

1. Source Collection: Videos were curated from a global pool spanning five macro-categories (Knowledge, Life Record, Sports, Artistic Performance, Film & TV) and diverse fine-grained scenarios (e.g., Aviation, Biology, Daily Life).
2. Creation & Verification: Group A generated questions based strictly on visual evidence, enforcing a "visual-description-only" policy to prevent models from using language priors (e.g., avoiding proper nouns like "Cristiano Ronaldo" in favor of "the man in the red jersey"). Group B attempted to answer these questions to filter out trivial items or those solvable without video input.
3. Audit: Groups C and D performed holistic reviews and final audits to ensure answer correctness, refine timestamps, and eliminate ambiguity, resulting in the final "Gold Standard" set.

Hierarchical Structure

The benchmark is structured into a three-level hierarchy designed to test progressive cognitive and perceptual demands:

• Level 1: Homogeneous Counting: Focuses on basic visual grounding and object permanence. Models must count a single type of object, action, or event under minimal constraints, requiring the maintenance of an "internal counter" during continuous motion (e.g., tracking doors as the camera pans).
• Level 2: Constrained and Heterogeneous Counting: Increases complexity by requiring discrimination between multiple entity types or single entities defined by multi-attribute constraints (e.g., counting specific dog breeds). This level tests robust re-identification across temporal gaps and shot cuts to maintain numerical consistency.
• Level 3: Compositional Numerical Reasoning: Evaluates high-order cognitive capabilities including comparison, calculation, estimation, and sequential ordering. These tasks require multi-step arithmetic operations strictly grounded in temporal evidence (e.g., isolating time windows, counting players under specific constraints, and logically comparing the results).

Evaluation Protocol

The authors evaluated a diverse suite of state-of-the-art (SOTA) open-source and closed-source VLMs using two protocols:

1. Direct Answer (NoCoT): Models provide a final choice directly.
2. Zero-shot Chain-of-Thought (CoT): Models are prompted to articulate intermediate observations and calculation steps before committing to an answer.
   Visual input was standardized using a frame sampling strategy (1 FPS for videos <48s; 48 uniform frames for longer videos).

Key Contributions

1. VidNum-1.4K Benchmark: The introduction of a dedicated, diverse video QA benchmark specifically designed to evaluate numerical reasoning and "world understanding" capabilities, featuring a unique three-level hierarchy from perception to compositional reasoning.
2. Comprehensive Empirical Evaluation: A rigorous assessment of representative open-source and closed-source VLMs, revealing a significant performance gap and identifying specific bottlenecks in current architectures.

Results

The evaluation reveals a striking reasoning gap across all tested models:

• Overall Performance: Even the most advanced closed-source model, Gemini-3.1-pro, barely reaches a 60% accuracy threshold under the CoT setting. Representative open-source families (e.g., InternVL series) struggle significantly, typically performing in the 25%–45% range.
• Task-Specific Bottlenecks:
  • Level 3 Sensitivity: Chain-of-Thought prompting acts as a critical catalyst for Level 3 (Compositional) reasoning, suggesting models possess raw perceptual grounding but falter in logical aggregation without explicit guidance.
  • Action vs. Object/Event: Action-based counting consistently yields lower scores than object or event counting, indicating difficulties in tracking continuous, fluid boundaries of dynamic actions.
  • Numerical Consistency: The inclusion of multi-shot videos exposes a severe lack of consistency, with models frequently losing count or double-counting across scene cuts.
• Scaling Behavior: Analysis of the InternVL3 series (8B to 78B parameters) shows that increasing model capacity yields general performance improvements, particularly for Level 3 reasoning (accuracy rising from 48.8% to 57.1% with CoT). However, Level 1 and Level 2 performance remains stagnant or improves only modestly, suggesting that fine-grained instance tracking and cross-shot re-identification are fundamental perceptual bottlenecks not easily solved by parameter scaling alone.
• CoT Trade-offs: While CoT generally improves accuracy on complex tasks, it introduces a "gain-with-regression" phenomenon where some previously correct direct answers become incorrect, indicating that current models rely on unstable heuristic "intuition" rather than robust causal reasoning.

Significance

The paper positions VidNum-1.4K as a demanding diagnostic testbed for the next generation of numerical video intelligence. The findings demonstrate that current VLMs lack a stable "internal world model" capable of handling the inherent complexity of real-world multimedia content, particularly regarding temporal dynamics and compositional logic. By combining fragmented temporal contexts with multi-step reasoning requirements, the benchmark serves as a stringent stress test to expose these weaknesses, aiming to guide the development of models with more reliable and physically grounded video understanding.