Technical Summary: SVI-Bench

Problem Statement

Current video intelligence systems excel at perception—recognizing visible objects and actions—but fail to perform Strategic Video Intelligence (SVI). SVI is defined as a progressive cognitive stack requiring:

1. Perception: Parsing who is where and doing what.
2. Causal Reasoning: Explaining why events unfold and predicting outcomes.
3. Strategic Simulation: Generating plausible alternative futures and goal-directed strategies.
4. Agentic Synthesis: Autonomously integrating multimodal evidence to produce expert-level analysis.

Existing benchmarks fall short of evaluating this full stack. General video benchmarks (e.g., Kinetics, ActivityNet) focus on perception and short-term temporal reasoning but lack verifiable ground truth for causal or strategic questions. Synthetic environments (e.g., CLEVRER) offer verifiable causal ground truth but lack the complexity of real-world multi-agent systems. Real-world benchmarks often lack objective outcomes for strategic reasoning. There is a critical gap in evaluating AI systems on complex, dynamic multi-agent interactions where reasoning, simulation, and planning are required.

Methodology

The SVI-Bench Framework

The authors introduce SVI-Bench, a large-scale benchmark using team sports (basketball, soccer, hockey) as a "dynamic microworld." Sports were chosen because they combine the complexity of real-world multi-agent interaction (10–22 agents under adversarial pressure) with verifiable ground truth (explicit rules, definitive outcomes, and objective statistics).

Data Engine and Construction

SVI-Bench is constructed via a data engine that transforms five raw data sources into a dense, cross-referenced corpus:

1. Broadcast Video: ~35K hours of professional footage.
2. Play-by-Play Logs: ~15M timestamped event records with spatial coordinates.
3. Expert Commentary: ~15K hours of transcribed broadcast narration.
4. Game Reports: ~23K post-game journalist recaps.
5. Structured Statistics: ~103K records of player/team performance metrics.

The construction pipeline involves:

• Temporal Alignment: Synchronizing logs, commentary, and reports with video segments using game-clock timestamps.
• Cross-Modal Entity Resolution: Linking references to players, teams, and events across modalities into identity graphs.
• LLM-Powered Instance Generation: Using Large Language Models to synthesize question-answer pairs, distractors, and annotations based on the aligned multimodal context, guided by pillar-specific prompt templates.
• Quality Control: Automated checks against event logs followed by human expert review on representative subsets.

Evaluation Suite

The benchmark organizes evaluation into 9 tasks across four cognitive pillars:

• Pillar 1: Dynamic Scene Understanding (Perception)
  • T1 Structured Play Description: Dense captioning of multi-agent scenes (10s clips).
  • T2 Fine-Grained Action QA: Multiple-choice questions on specific interactions (10s clips).
  • T3 Compositional Video Retrieval: Retrieving videos based on complex attribute compositions.
• Pillar 2: Causal Reasoning
  • T4 Strategic Reasoning QA: Open-ended explanation of strategic causes over full games (55–150 min).
  • T5 Outcome Forecasting: Predicting future game states or events (3–15 min segments).
  • T6 Long-Form Narrative Synthesis: Generating expert-level game reports from full games.
• Pillar 3: Strategic Simulation
  • T7 Motion-Conditioned Generation: Generating video where agents follow prescribed trajectories.
  • T8 Goal-Conditioned Action Generation: Generating video where agents execute actions to achieve a specific goal.
• Pillar 4: Agentic Synthesis
  • T9 Cross-Corpus Agentic Reasoning: An autonomous agent must plan multi-step retrieval across ~1.8M video clips and ~33K documents to answer complex queries.

Key Results

The Performance Cliff

Evaluations of strong multimodal and agentic baselines (including GPT-5.2, Gemini 3, and fine-tuned open-source models) reveal a consistent "capability cliff":

• Perception: Models perform competently. Fine-tuned LLaVA-Video-7B achieves 73.91% accuracy on fine-grained action QA (T2), nearly matching human experts (75.78%).
• Reasoning: Performance degrades sharply. On strategic reasoning (T4), models score only ~2.0/5.0, while humans score 4.2/5.0. Outcome forecasting (T5) drops to 44.82% accuracy (vs. 58.9% for humans).
• Simulation: Multi-agent coordination is difficult. Even fine-tuned video generation models achieve only ~0.51 mIoU on motion-conditioned generation, meaning roughly half of generated player positions deviate significantly from trajectories.
• Agency: The most severe drop occurs in agentic tasks. In the default setting (analyzing raw video), the strongest model (GPT-5.2) achieves only 4.6% accuracy on cross-corpus reasoning (T9).

Oracle Experiments

To isolate the impact of perception versus reasoning, the authors replaced raw video inputs with ground-truth textual descriptions (play-by-play logs):

• Reasoning (T4, T5): Oracle access provided only marginal gains (e.g., T5 accuracy rose from 38.2% to 41.9%), suggesting that perception is not the primary bottleneck for these tasks; the limitation lies in strategic reasoning itself.
• Agentic Synthesis (T9): Oracle access improved accuracy significantly from 4.6% to 54.0%. This indicates that while visual perception is a major bottleneck for T9, the remaining gap (46%) is due to failures in multi-step planning, cross-modal reasoning, and evidence integration.

Human-Model Gap

Human experts significantly outperform models on reasoning and forecasting tasks but are closely matched on perception. Furthermore, humans demonstrate calibrated confidence (knowing when they are uncertain), whereas models often exhibit high confidence on incorrect answers.

Significance and Claims

The paper claims SVI-Bench is the first large-scale benchmark to evaluate the full spectrum of video intelligence from perception to strategic agency in real-world multi-agent environments. Its significance lies in:

1. Defining a New Capability Stack: Formalizing Strategic Video Intelligence as a progressive hierarchy (Perception $\to$ Reasoning $\to$ Simulation $\to$ Agency) rather than isolated skills.
2. Identifying a Critical Bottleneck: Demonstrating that current state-of-the-art models, while competent at "seeing," fail to "reason," "simulate," or "plan" in complex dynamic environments. The performance cliff suggests that scaling perception alone is insufficient for strategic intelligence.
3. Providing a Verifiable Testbed: Offering a domain (sports) where causal and strategic questions have objective ground truth, enabling rigorous evaluation of causal and agentic capabilities that are currently unmeasurable in other real-world domains.

The authors conclude that progress in SVI requires advancements in three specific areas: video models with stronger long-horizon reasoning, generative models with explicit multi-agent dynamics, and multimodal agents capable of planning and retrieving evidence at scale.