💻 computer science

WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation

This paper introduces WBench, a comprehensive multi-turn benchmark designed to systematically evaluate interactive video world models across five key dimensions—video quality, setting adherence, interaction adherence, consistency, and physics compliance—using 289 test cases and validated automatic metrics to reveal that no current state-of-the-art model excels in all areas.

Original authors: Kaining Ying, Hengrui Hu, Siyu Ren, Jiamu Li, Fengjiao Chen, Ziwen Wang, Xuezhi Cao, Xunliang Cai, Henghui Ding

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaining Ying, Hengrui Hu, Siyu Ren, Jiamu Li, Fengjiao Chen, Ziwen Wang, Xuezhi Cao, Xunliang Cai, Henghui Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a video game engine, but instead of coding the rules, you are teaching an AI to "dream" a world into existence. You give it a starting picture and say, "Now, walk forward, jump over that rock, and suddenly a helicopter appears." The AI then generates the next few seconds of video based on your command.

The problem is: How do we know if the AI is actually good at this?

Currently, there are many different ways to test these AI "world models," but they are like trying to judge a car by only testing its brakes, or only its speed, or only its paint job. There is no single, unified test that checks everything at once.

Enter WBENCH. Think of WBENCH as the ultimate "Driver's License Exam" for these interactive video AIs.

The Big Idea: The "World Simulator" Test

The authors created a comprehensive test suite called WBENCH. Instead of just asking the AI to make a pretty video, they put it in a virtual driving school where it has to handle a multi-turn conversation with a user.

Here is how the test works, broken down into five simple categories:

The Look (Video Quality): Does the video look smooth and pretty, or is it flickering and blurry? This is like checking if the car's paint is shiny and the tires are round.
The Memory (Setting Adherence): If you told the AI, "It's a snowy mountain with a red robot," does it keep the snow and the red robot throughout the whole video? Or does the robot suddenly turn blue and the snow melt? This tests if the AI remembers the rules of the world it created.
The Obedience (Interaction Adherence): If you say "Turn left," does the camera actually turn left? If you say "The robot jumps," does it jump? This is the "steering wheel" test. The paper found that some AIs are great at making pretty pictures but terrible at actually listening to your commands.
The Stability (Consistency): If the camera spins around in a circle and comes back to the start, does the world look exactly the same as it did before? Or did the buildings shift, or did the robot disappear? This tests if the AI has a stable "memory" of the world's layout.
The Physics (Physical Compliance): If a ball falls, does it fall down? If a car crashes, does it crumple? Or does the ball float up to the sky and the car pass through the wall like a ghost? This tests if the AI understands how the real world (or the fantasy world) works.

The Test Drive: What They Did

The researchers built a dataset of 289 different "scenarios" (like a snowy mountain, a busy city, or a fantasy castle). For each scenario, they created a sequence of 1,058 instructions (turns).

They tested 20 different AI models (including big names like Kling, Wan, Genie 3, and others). They asked these models to follow instructions like:

"Walk forward."
"Jump."
"Switch the view from behind the character to the character's own eyes."
"Make it rain."

The Results: No Perfect Driver

After running the tests, the paper found some surprising things:

No "Super AI" exists yet: Just like no single car is the fastest, safest, and most fuel-efficient all at once, no single AI model passed every part of the test. Some were great at making pretty videos but terrible at following directions. Others were great at navigating but forgot what the world looked like after a few seconds.
Navigation is tricky: Moving the camera (walking or turning) is actually a separate skill from making the video look good. An AI can make a beautiful movie but fail to move the camera when asked.
The "Long Game" is hard: The longer the conversation goes on (the more turns you take), the more the AI starts to make mistakes. It's like a human trying to remember a complex story; after a while, they start mixing up the details.
Open Source is catching up: Some of the models that are free for anyone to use performed just as well as, or even better than, the expensive, closed-source models on specific tasks.

The Bottom Line

WBENCH is a new, standardized ruler for measuring interactive video AIs. It proves that while these models are getting better at making movies, they still struggle to be reliable "world simulators" that can consistently follow instructions, remember the past, and obey the laws of physics over a long period of time.

The paper doesn't say these models are ready to replace human game designers or run self-driving cars yet. Instead, it says: "Here is exactly where they are failing, so developers know what to fix next." It's a diagnostic tool to help the next generation of AI become truly interactive.

Technical Summary: WBENCH - A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation

1. Problem Statement

Interactive video world models are rapidly advancing, capable of generating controllable video sequences in response to user actions across domains like gaming, autonomous driving, and embodied interaction. However, the field lacks a unified standard for systematic evaluation. Existing benchmarks are fragmented: some focus solely on perceptual video quality (e.g., VBench) without testing interactive control, while others evaluate specific world modeling aspects (e.g., navigation or memory) but fail to cover the full spectrum of open-domain scenes, perspectives, and interaction types. Consequently, it is difficult to fairly compare models with different native interfaces (text, camera pose, discrete actions) or to diagnose specific failure modes in long-horizon generation, consistency, and physical compliance.

2. Methodology

2.1 Dataset Construction (WBENCH)

The authors introduce WBENCH, a comprehensive multi-turn benchmark comprising 289 test cases and 1,058 interaction turns. The dataset is constructed based on a "setting-first" principle:

World Settings: Each case defines an initial world state $W$ comprising four attributes: Scene (6 categories: nature, urban, indoor, workspace, fantasy, sports), Style (photorealistic to stylized), Perspective (First-Person or Third-Person), and Subject (human, animal, vehicle, robot, or other).
Interaction Sequences: Each case includes a multi-turn sequence of user controls drawn from four interaction types:
1. Navigation: Camera or ego-agent motion using WASD (translation) and arrow keys (rotation).
2. Subject Action: Actions performed by the primary subject (manipulation, locomotion, tool use, combat, gestures).
3. Event Editing: Externally imposed environmental changes (weather, time-of-day, object appearance).
4. Perspective Switching: Transitions between first-person and third-person views.
Unified Control: To enable fair comparison across paradigms, navigation interactions are represented in three aligned forms: text prompts, 6-DoF camera poses, and discrete keyboard actions.

2.2 Evaluation Suite

WBENCH evaluates models across five complementary dimensions using 22 fine-grained automatic sub-metrics, all validated against human judgments:

Video Quality: Measures perceptual quality (aesthetic, imaging, flickering, dynamic degree, smoothness, HPSv3-Norm) independent of conditioning.
Setting Adherence: Evaluates faithfulness to the initial world setting, including Scene Adherence (maintaining visible elements and revealing off-screen elements) and Subject Adherence (appearance and motion style).
Interaction Adherence: Assesses correct execution of user requests.
- Navigation: Uses MegaSaM for pose estimation to compute normalized Absolute Trajectory Error (nATE) against synthetic ground truth.
- Semantic Interactions (Event Editing, Subject Action, Perspective Switching): Uses Vision-Language Models (VLMs) with structured binary checks (e.g., event occurrence, completion, anomaly absence).
Consistency: Measures stability of geometry, appearance, and perspective over time. Includes metrics for spatial consistency (round-trip trajectories), segment continuity (cut detection), perspective consistency (subject tracking), and reconstruction consistency (geometric and photometric reprojection).
Physical Compliance: Assesses adherence to physical rules via Causal Fidelity (VLM-based evaluation of rendering physics and causal logic) and Visual Plausibility (fine-tuned Qwen3-VL scoring of low-level artifacts).

2.3 Experimental Protocol

The benchmark was applied to 20 state-of-the-art models spanning three paradigms:

Text-driven (9 models): e.g., Kling 3.0, Wan 2.7, Seedance 1.5. Evaluated on the full 289 cases via iterative last-frame forwarding.
Camera-controlled (5 models): e.g., HY-World 1.5, LingBot-World. Evaluated on a shared navigation subset (158 cases) using native pose inputs.
Action-conditioned (6 models): e.g., Genie 3, Matrix-Game 3.0. Evaluated on the navigation subset using discrete action inputs.
Human Alignment: A study with 400 annotators performing blind pairwise comparisons confirmed that the automated metrics achieve a Spearman correlation ( $\rho \ge 0.94$ ) with human preferences.

3. Key Results

No Dominant Model: No single model performs strongly across all five dimensions. Performance is highly specialized based on the model's paradigm and training data.
Navigation Independence: Navigation capability is largely decoupled from other dimensions. Strong rendering or physics performance does not guarantee controllable movement. Conversely, physical correctness correlates more strongly with video quality and consistency than with control capabilities.
Control vs. Consistency: Camera control does not guarantee perspective consistency. Models with native camera control (e.g., HY-World 1.5) excel in navigation but often struggle to maintain coherent subject motion in third-person views compared to text-driven models.
Degradation Over Turns: Navigation performance degrades significantly over multi-turn sequences (dropping ~33 points from Turn 1 to Turn 4+), as pose errors compound. Semantic interactions (event editing, subject action) degrade more moderately. Perspective switching remains the most difficult task overall (average score ~30.7).
Open-Source Competitiveness: Open-source models (e.g., HY-World 1.5, LingBot-World, Matrix-Game 3.0) achieve leading scores in specific dimensions (navigation, consistency, action-conditioned control), matching or surpassing closed-source alternatives.
Structured Difficulty: Benchmark difficulty varies by setting. First-person navigation is easier than third-person. Sports/game scenes and animal subjects present the highest difficulty due to complex dynamics, while workspace scenes and robots are easier due to static geometry.

4. Key Contributions

Unified Benchmark: A comprehensive framework spanning five evaluation dimensions with 22 fine-grained sub-metrics, addressing the gap in existing benchmarks that cover only partial competencies.
Multi-turn Dataset: A dataset covering both perspectives, four interaction types, and a unified navigation interface, enabling fair cross-paradigm comparison.
Diagnostic Pipeline: A fully automatic evaluation pipeline applied to 20 models, establishing diagnostic baselines and surfacing actionable insights (e.g., the independence of navigation from other capabilities) for future model development.

5. Significance and Claims

The paper claims that WBENCH provides the first unified protocol for evaluating interactive world models across open-domain scenes, both perspectives, and all major interaction types. By revealing that current models have not yet unified high-fidelity rendering with reliable controllability, consistency, and physics, the benchmark highlights the specific gaps in the field. The authors assert that their findings—particularly the decoupling of navigation from other capabilities and the structured difficulty based on scene/subject types—offer a roadmap for developing more robust world models. The work is positioned as a diagnostic tool to guide future research rather than a solution to the underlying generation challenges.

Limitations Acknowledged: The current test set focuses on discrete action sequences rather than continuous control, and the physical dimension relies partly on LLM-based evaluation which may have reliability limits for subtle effects.