💻 computer science

WorldMark: A Unified Benchmark Suite for Interactive Video World Models

This paper introduces WorldMark, the first unified benchmark suite and online arena that enables fair, standardized comparison of interactive video world models by providing a common set of scenes, action sequences, and evaluation metrics across diverse systems.

Original authors: Xiaojie Xu, Zhengyuan Lin, Kang He, Yukang Feng, Xiaofeng Mao, Yuanyang Yin, Kaipeng Zhang, Yongtao Ge

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaojie Xu, Zhengyuan Lin, Kang He, Yukang Feng, Xiaofeng Mao, Yuanyang Yin, Kaipeng Zhang, Yongtao Ge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a food critic trying to judge the best chefs in the world. But here's the catch: Chef A only lets you taste their food in their own private kitchen with their own ingredients. Chef B only lets you taste theirs in a different kitchen with different spices. Chef C refuses to let you taste anything unless you speak a language only they understand.

Because everyone is cooking in their own isolated world, you can never truly say, "Chef A is better than Chef B." You can only say, "Chef A's food is good in Chef A's kitchen."

This is exactly the problem with Interactive Video World Models (AI that creates video games or virtual worlds you can walk through). Right now, every AI model has its own rules, its own "kitchen," and its own way of taking orders. Comparing them fairly is impossible.

Enter WorldMark. Think of WorldMark as the first standardized "Taste-Test" arena for these AI world creators.

The Three Big Problems WorldMark Solves

1. The "Language Barrier" (Unified Action Mapping)

The Problem: One AI understands "WASD" (the keyboard keys for moving forward/back/left/right). Another only understands "Pose your hand like this." A third only understands "Say 'Go forward' in a sentence."
The WorldMark Solution: WorldMark acts like a universal translator.

Imagine you have a single remote control with standard buttons: Move Forward, Move Back, Turn Left, Turn Right.
WorldMark takes your simple press of "Forward" and instantly translates it into whatever specific language that specific AI speaks.
Result: Now, every AI is trying to do the exact same task at the exact same time, using the exact same instructions. It's an apples-to-apples comparison.

2. The "Missing Playbook" (The Test Suite)

The Problem: Before, every AI was tested on its own favorite scenes (like a specific Minecraft castle or a specific city). If an AI was bad at forests but good at cities, it could just hide in the city tests.
The WorldMark Solution: WorldMark provides a standardized obstacle course with 500 different challenges.

The Scenes: It includes 50 different starting images, ranging from realistic cities and nature spots to cartoonish or "stylized" art.
The Views: You can test the AI from a "First-Person" view (like wearing a GoPro) or a "Third-Person" view (watching a character from behind, like in a video game).
The Difficulty: The tests get harder.
- Easy: Walk forward for 20 seconds.
- Medium: Walk forward, then turn around.
- Hard: Walk in a complex patrol route for a full minute without the world falling apart.

3. The "Scorecard" (The Evaluation Toolkit)

The Problem: How do you grade the video? Is it pretty? Did it move when you told it to? Did the world make sense?
The WorldMark Solution: They created a three-part report card for every video generated:

Visual Quality (The "Pretty" Score): Is the video blurry? Is the lighting weird? Does it look like a nice photo or a mess?
Control Alignment (The "Obedience" Score): If you told the AI to turn right, did it actually turn right? Or did it just slide sideways?
World Consistency (The "Logic" Score): This is the big one. If you walk around a corner and come back, is the chair still there? Did the character's face melt? Did the sky turn into water? This checks if the AI remembers the world it created.

What Did They Find? (The Plot Twist)

When they ran all the models through this standardized test, they discovered some surprising things that previous private tests missed:

Pretty doesn't mean Smart: One model (YUME) made the most beautiful, artistic-looking frames. But if you walked around in its world, the logic fell apart. It was like a beautiful painting that changes every time you blink.
Smart doesn't mean Pretty: Another model (Genie 3) kept the world perfectly consistent (the chair stayed the chair), but the images were a bit less "artistic" than the others.
The "Third-Person" Nightmare: When the AI had to show a character from behind (like a video game), many models completely lost their minds. They couldn't keep the camera steady or the character in frame. It's like a camera operator who suddenly forgets how to hold a camera.
Specialists aren't Generalists: An AI trained specifically on Minecraft couldn't handle a realistic city scene at all. It's like a chef who is amazing at baking cakes but burns toast.

Why This Matters

Before WorldMark, it was like watching a boxing match where the fighters wore blindfolds and fought in different rooms. We didn't know who was actually the strongest.

WorldMark removes the blindfolds, puts everyone in the same ring, and gives them the same opponent. Now, researchers can finally see which AI is truly building a stable, interactive world and which one is just hallucinating pretty pictures.

They even built a website (World Model Arena) where anyone can watch these AI models fight side-by-side in real-time, like a live leaderboard for the best virtual world creators.

1. Problem Statement

The field of interactive video world models (e.g., Genie, YUME, HY-World, Matrix-Game) is advancing rapidly, but progress is hindered by a lack of standardized evaluation.

Fragmented Benchmarks: Current models are evaluated on private datasets with bespoke scenes, trajectories, and control interfaces, making fair cross-model comparison impossible.
Incompatible Inputs: Models use heterogeneous control mechanisms (e.g., natural language captions, 6-DoF pose parameters, gamepad signals, action vectors), preventing "apples-to-apples" testing.
Missing Metrics: Existing public benchmarks (like VBench or WorldScore) focus on text-to-video quality or passive camera trajectories but fail to evaluate action controllability, long-horizon world consistency, and interactive responsiveness under unified conditions.
The Gap: Without standardized test conditions (identical scenes, identical action sequences, and a unified control interface), it is impossible to determine if a model truly advances world modeling or merely excels under favorable, proprietary test conditions.

2. Methodology: The WorldMark Framework

WorldMark introduces a standardized infrastructure designed to create a "common playing field" for interactive Image-to-Video (I2V) world models. It consists of five core components:

A. Unified Action Interface & Mapping

To address input heterogeneity, WorldMark defines a shared WASD-style action vocabulary (Forward, Backward, Strafe Left/Right, Yaw Left/Right) parameterized by duration.

Action-Mapping Layer: A modular adapter translates this shared vocabulary into each model's native control format (e.g., text prompts for YUME, pose parameters for HY-World, gamepad signals for Genie 3).
Calibration: Adapters calibrate per-model parameters (step size, yaw rate) to ensure semantically identical instructions are executed faithfully across different architectures.

B. Hierarchical Test Suite (500 Cases)

The benchmark comprises a curated dataset designed to test models across diverse conditions:

Image Suite: 50 reference images (expanded to 100 via third-person synthesis) covering three scene categories (Nature, City, Indoor), two styles (Photorealistic, Stylized), and two viewpoints (First-person, Third-person).
Action Suite: 15 standardized action sequences ranging from simple translations to complex cyclic patrols and 360° rotations.
Difficulty Tiers:
- Easy: Single-segment, 20s (basic compliance).
- Medium: Two-segment, 40s (smooth transitions).
- Hard: Three-segment, 60s (sustained consistency over long horizons).
Context-Aware Filtering: A Vision-Language Model (VLM) analyzes each reference image to select only physically plausible action sequences (e.g., preventing lateral movement in enclosed spaces).

C. Modular Evaluation Toolkit

WorldMark evaluates generated videos across three dimensions using a mix of geometric and VLM-based metrics:

Visual Quality:
- Aesthetic Quality: Human-perceived appeal (LAION predictor).
- Imaging Quality: Low-level distortions like noise/blur (MUSIQ).
Control Alignment:
- Translation Error: Scale-invariant Euclidean distance between ground-truth and estimated camera positions (via DROID-SLAM).
- Rotation Error: Geodesic angular deviation between predicted and ground-truth rotation matrices.
World Consistency:
- Reprojection Error: 3D spatial coherence via Dense Bundle Adjustment.
- State/Content/Style Consistency: VLM-based judgments tracking object stability, hallucination (popping), and aesthetic uniformity over time.

D. Evaluation Workflow

The pipeline follows: Image Selection $\rightarrow$ Action Mapping (via adapters) $\rightarrow$ Video Generation (under standardized conditions) $\rightarrow$ Metric Evaluation. This allows researchers to plug in custom metrics while maintaining standardized inputs.

3. Key Contributions

First Standardized Benchmark: WorldMark is the first benchmark enabling unified cross-model comparison for interactive I2V world models, supporting six major models with heterogeneous interfaces.
Unified Action Mapper: A novel translation layer that normalizes diverse control inputs (text, pose, gamepad) into a single semantic vocabulary, enabling fair comparison.
Comprehensive Test Suite: A 500-case hierarchical suite covering first/third-person views, photorealistic/stylized styles, and varying difficulty levels.
Open Ecosystem: Release of all data, evaluation code, and model outputs, plus the launch of World Model Arena (warena.ai) for live, side-by-side human voting and leaderboard tracking.

4. Experimental Results

The authors evaluated six models: YUME 1.5, Matrix-Game 2.0, HY-World 1.5, HY-GameCraft, Open-Oasis, and Genie 3.

Key Findings:

Visual Quality vs. World Consistency are Uncorrelated:
- YUME 1.5 produced the most aesthetically pleasing frames (highest Aesthetic/Imaging Quality) but ranked poorly on long-horizon coherence and world consistency.
- Genie 3 (closed-source) maintained the most consistent worlds (lowest Reprojection Error, highest State/Content Consistency) but had only moderate visual fidelity compared to YUME.
Control Alignment $\neq$ Overall Quality:
- HY-GameCraft exhibited the lowest translation/rotation errors (precise control) but suffered from poor visual fidelity.
- Genie 3 had higher trajectory errors yet preserved a globally coherent world, suggesting that precise geometric control does not guarantee high-quality generation.
Third-Person Failure Mode:
- Switching to third-person views caused a drastic performance drop for many models. For example, Matrix-Game 2.0's rotation error increased by roughly 20x in third-person compared to first-person, highlighting the difficulty of maintaining camera control around a visible character.
Domain Transfer Issues:
- Open-Oasis, trained specifically on Minecraft, failed across all metrics when tested on real-world or stylized scenes, indicating a lack of generalization.
Human Alignment: Automated metrics showed a strong Spearman correlation ( $\rho > 0.9$ ) with human preference rankings, validating the benchmark's reliability.

5. Significance

Resolves Evaluation Fragmentation: WorldMark eliminates the "private benchmark" barrier, allowing the community to objectively rank models based on identical inputs.
Reveals Critical Trade-offs: The study exposes that current models often optimize for one dimension (e.g., aesthetics or control) at the expense of others (e.g., consistency), guiding future research toward balanced architectures.
Identifies Open Challenges: The severe degradation in third-person generation and the difficulty of long-horizon consistency highlight specific areas where current world models need improvement.
Community Resource: By providing a modular toolkit and an online arena, WorldMark accelerates the iterative development of interactive world models, moving the field from isolated experiments to a unified, competitive landscape.