VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos?

1. Problem Statement

While Text-to-Video (T2V) generation models have advanced rapidly, existing evaluation benchmarks are becoming insufficient for assessing state-of-the-art systems. The authors identify three critical limitations in current evaluation paradigms:

• Saturation of Traditional Metrics: Current benchmarks (e.g., VBench, EvalCrafter) focus on frame-level aesthetics, image fidelity, and explicit semantic alignment. As models improve, these metrics saturate, failing to differentiate between models with clear capability gaps.
• Lack of Temporal Causality: Existing benchmarks rarely evaluate event-level temporal causality, a core property distinguishing video from other modalities. They often rely on fully specified prompts where the outcome is explicitly described, removing the need for the model to infer consequences.
• Absence of World Model Assessment: Current benchmarks do not systematically assess World Model capabilities—the ability to understand implicit physical laws, natural constraints, and common sense knowledge that are not explicitly stated in the prompt.

2. Methodology: The VideoVerse Benchmark

To address these gaps, the authors introduce VideoVerse, a comprehensive benchmark designed to evaluate T2V models on their ability to synthesize videos that adhere to complex temporal causality and world knowledge.

A. Benchmark Construction

• Data Sources: Prompts are curated from three domains: Daily Life (from ActivityNet), Scientific Experiments (high-school level physics/chemistry), and Science Fiction (from VidProM).
• "Hidden Semantics" Design: Unlike previous benchmarks that explicitly describe outcomes (e.g., "a rubber duck bounces"), VideoVerse prompts omit consequences. For example, a prompt might say "pour vinegar into baking soda," expecting the model to implicitly generate the resulting bubbles based on chemical knowledge.
• Prompt Engineering:
  1. Event Extraction: GPT-4o extracts causal event chains from raw video captions.
  2. Human Refinement: Independent annotators (with at least a bachelor's degree) rewrite these into final prompts, integrating specific evaluation dimensions while preserving the causal event structure.
• Scale: The benchmark consists of 300 high-quality prompts covering 815 events and 793 binary evaluation questions.

B. Evaluation Dimensions

VideoVerse defines 10 evaluation dimensions split into Dynamic (requiring temporal understanding) and Static (evaluable in a single frame) categories:

C. Evaluation Protocol

• VLM-Based QA: The authors use Gemini 2.5 Pro as an evaluator to answer binary questions (Yes/No) about the generated videos.
• Mitigating Hallucination: Unlike previous methods that feed the full prompt to the VLM (causing it to assume the video contains what the prompt says), VideoVerse provides dimension-specific binary questions to the VLM, forcing it to verify visual evidence only.
• Scoring Strategy:
  • Event Following: Uses the Longest Common Subsequence (LCS) algorithm to measure the alignment between the ground-truth event order and the VLM's detected event order.
  • Other Dimensions: Scores are cumulative based on the number of correctly answered binary questions.
• Sub-Question Decomposition: For complex dynamic dimensions (Mechanics, Interaction, Material Properties), evaluation questions are decomposed into finer-grained sub-questions to detect partial failures (e.g., an axe hitting wood but not splitting it).

3. Key Contributions

1. VideoVerse Benchmark: A novel benchmark with 300 prompts and 10 dimensions specifically targeting World Model capabilities and hidden semantics, moving beyond surface-level visual quality.
2. Evaluation Pipeline: A robust, human-aligned QA-based pipeline using advanced VLMs that minimizes hallucination by decoupling the prompt from the evaluation question.
3. Comprehensive Analysis: The first systematic comparison of open-source vs. closed-source T2V models regarding their world modeling abilities, revealing significant gaps.

4. Experimental Results

The authors evaluated leading open-source (e.g., CogVideoX, Hunyuan, Wan, OpenSora) and closed-source models (Veo-3, Sora-2, Minimax-Hailuo).

• Performance Gap: While open-source models perform comparably to closed-source models on basic dimensions (e.g., 2D layout, attribution), a massive gap emerges in World Model dimensions.
  • Sora-2 and Veo-3 achieve the highest overall scores.
  • Wan2.2-A14B leads open-source models but still lags significantly behind closed-source systems in world knowledge.
• World Model Deficit: Even the best models (Sora-2, Veo-3) achieve only ~72% on World Model dimensions (excluding Event Following), indicating they are far from true world simulators.
• Specific Failures:
  • Interaction: Models often fail to show physical consequences (e.g., shaving a beard does not shorten the hair).
  • Natural Constraints: Models fail to simulate physical phenomena (e.g., dry ice not sublimating).
  • Event Following: Models struggle to maintain strict causal chains over time.
• Video Length: The study found that video length (5s vs. 10s) is not the primary bottleneck for Event Following; the limitation lies in the model's reasoning capabilities.

5. Significance and Future Directions

• Paradigm Shift: VideoVerse shifts the T2V evaluation focus from "visual fidelity" to "logical and physical consistency," aligning benchmarks with the goal of creating true World Models.
• Research Direction: The results highlight that current models rely heavily on surface-level text matching. Future research must focus on implicit reasoning, physical simulation, and causal inference rather than just larger backbones or more training data.
• Efficiency: The design of VideoVerse (multiple dimensions per prompt) offers a cost-effective way to benchmark increasingly large models, addressing the high inference costs of modern T2V systems.

In conclusion, VideoVerse demonstrates that while T2V models have achieved cinematic visual quality, they currently lack the deep world understanding required to synthesize videos that are physically and logically plausible without explicit textual guidance.