💻 computer science

WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models

This paper introduces WorldArena, a unified benchmark and the EWMScore metric designed to systematically evaluate embodied world models across both perceptual fidelity and functional utility in downstream decision-making tasks, revealing that high visual quality does not always guarantee strong task performance.

Original authors: Yu Shang, Zhuohang Li, Yiding Ma, Weikang Su, Xin Jin, Ziyou Wang, Lei Jin, Xin Zhang, Yinzhou Tang, Haisheng Su, Chen Gao, Wei Wu, Xihui Liu, Dhruv Shah, Zhaoxiang Zhang, Zhibo Chen, Jun Zhu, Yonghon

Published 2026-02-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu Shang, Zhuohang Li, Yiding Ma, Weikang Su, Xin Jin, Ziyou Wang, Lei Jin, Xin Zhang, Yinzhou Tang, Haisheng Su, Chen Gao, Wei Wu, Xihui Liu, Dhruv Shah, Zhaoxiang Zhang, Zhibo Chen, Jun Zhu, Yonghong Tian, Tat-Seng Chua, Wenwu Zhu, Yong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a professional chef. To see if they are ready for a real kitchen, you wouldn't just look at a high-definition photo of a delicious meal they made (that’s just "looking good"). You would actually watch them cook, see if they follow a recipe, check if they can handle a knife without cutting themselves, and see if they can actually produce a meal that tastes right.

Currently, scientists are trying to build "World Models"—AI "brains" for robots. These brains are supposed to "dream" or predict what will happen next when a robot moves (e.g., "If I move my arm left, the cup will tip over").

The problem? Most researchers are only checking if the AI's "dreams" look like pretty movies. They are checking the "photo quality," but they aren't checking if the "physics" actually make sense.

This paper introduces WorldArena, which is like a comprehensive "Final Exam" for these AI brains.

The Three Parts of the Exam

Instead of just looking at the pictures, WorldArena tests the AI in three specific ways:

1. The "Eye Test" (Perception Quality)
This is the most basic level. Does the video look sharp? Is the lighting realistic? Does the object stay the same shape, or does it morph into a blob halfway through? It’s like checking if a movie has high-definition graphics or if it looks like a blurry, glitchy mess.

2. The "Kitchen Test" (Functional Utility)
This is where the real challenge begins. The researchers test if the AI can actually do things. They use three metaphors:

The Data Engine (The Practice Chef): Can the AI generate enough realistic "practice videos" to help a robot learn how to cook without needing a real kitchen?
The Policy Evaluator (The Food Critic): If we show a robot a video of its own actions, can the AI accurately predict if the robot succeeded or failed? (Is the AI a reliable judge?)
The Action Planner (The Head Chef): Can the AI actually plan a sequence of moves to complete a task, like "pick up the hammer and hit the block"?

3. The "Human Touch" (Human Evaluation)
Since math and code can sometimes miss the "vibe" of reality, they brought in real people to watch the videos and say, "Yes, that looks like a real robot," or "No, that looks like a hallucination."

The Big Discovery: The "Pretty vs. Practical" Gap

The most important finding in this paper is a bit of a wake-up call. The researchers found a massive gap between looking good and being useful.

Think of it like this: Imagine an AI that generates a stunning, 4K, cinematic video of a robot picking up a glass. It looks like a Hollywood movie! But, if you actually try to use that AI to guide a real robot, the robot fails. Why? Because in the "movie," the glass might pass through the robot's hand like a ghost, or the physics might be slightly "off."

The paper proves that just because an AI can make a beautiful video doesn't mean it understands how the physical world actually works.

Why does this matter?

By creating WorldArena, the researchers have built a standardized "arena" where all AI developers can compete. It moves the goalposts from "Make a pretty video" to "Make a brain that a robot can actually trust to navigate the real world."

Technical Summary: WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models

1. Problem Statement

As Embodied AI advances, Embodied World Models (EWMs) have become essential for enabling agents to reason about environmental dynamics through action-conditioned prediction. However, the authors identify two critical gaps in current evaluation protocols:

Fragmentation of Evaluation: Existing benchmarks focus heavily on perceptual fidelity (video generation quality, such as visual realism and motion smoothness) but largely overlook functional utility—the model's ability to serve as a reliable proxy for downstream embodied tasks like data synthesis, policy evaluation, and action planning.
Insufficient Model Coverage: Current benchmarks primarily target general-purpose, text-conditioned video models, leaving specialized, action-conditioned, and robot-centric world models largely unassessed.

2. Methodology

The authors propose WorldArena, a unified benchmark that evaluates EWMs across three complementary dimensions:

A. Perceptual Dimension (Video Quality):
The benchmark assesses open-loop prediction through 16 numerical metrics categorized into six sub-dimensions:

Visual Quality: Image clarity (MUSIQ), aesthetic appeal (LAION), and feature distribution similarity (V-JEPA).
Motion Quality: Motion intensity (Dynamic Degree, Flow Score) and temporal coherence (Motion Smoothness).
Content Consistency: Stability of subjects (DINO), backgrounds (CLIP), and pixel-level textures (Photometric Consistency).
Physics Adherence: Realism of robot-object interactions (VLM-based) and spatial-temporal trajectory alignment (NDTW).
3D Accuracy: Geometric consistency (Depth Accuracy) and perspective plausibility (VLM-based).
Controllability: Adherence to instructions (VLM-based), semantic alignment (CLIP), and action-following diversity.

B. Functional Dimension (Embodied Task Utility):
The benchmark evaluates closed-loop performance through three specific roles:

Embodied Data Engine: Measuring the performance gain of downstream policy models ( $\pi_{0.5}$ ) when trained on synthetic data generated by the world model.
Embodied Policy Evaluator: Assessing the correlation between task success rates in the world model versus a high-fidelity physical simulator (RoboTwin 2.0).
Embodied Action Planner: Evaluating the success rate of the world model when used directly as an end-to-end controller in a simulator.

C. Human Evaluation:
To capture qualitative nuances (e.g., physical plausibility and instruction adherence) that automated metrics might miss, the authors conduct large-scale human studies involving scoring and head-to-head win-rate comparisons.

D. EWMScore:
The authors introduce EWMScore, a holistic, single-index metric derived from the arithmetic mean of normalized perceptual metrics, designed to provide an interpretable measure of overall generative performance.

3. Key Contributions

First Unified Benchmark: The first framework to integrate both perceptual (visual) and functional (task-oriented) evaluations for EWMs.
Comprehensive Metric Suite: A robust set of 16 automated metrics and a new holistic index (EWMScore) that aligns well with human judgment (Pearson $r = 0.825$ ).
Systematic Large-Scale Evaluation: A rigorous study of 14 representative models, ranging from general video generators (e.g., Veo 3.1, Wan 2.6) to specialized robotic models (e.g., CtrlWorld, RoboMaster).

4. Key Results

The Perception–Functionality Gap: A significant finding is that high visual quality does not guarantee high functional utility. While commercial models (Veo 3.1, Wan 2.6) dominate in aesthetic and visual metrics, they do not necessarily outperform specialized models in embodied tasks.
Data Engine Limitations: Most world models currently provide insufficient data quality for effective policy training; only a few (e.g., RoboMaster, WoW) showed competitive performance in data synthesis.
Action-Conditioning Advantage: Action-conditioned models (like CtrlWorld) demonstrate superior physical adherence and higher correlation with real-world simulators compared to text-conditioned models.
Correlation Analysis: EWMScore correlates strongly with human perception but shows much weaker correlations with action planning ( $r = 0.360$ ), highlighting that current models lack the "predictive signals" necessary for complex reasoning despite looking realistic.

5. Significance

WorldArena shifts the paradigm of EWM evaluation from "looking real" to "being useful." By providing a public leaderboard and a multi-dimensional framework, it offers a roadmap for the community to move beyond video generation toward developing truly functional, physically grounded simulators that can drive the next generation of autonomous robotic intelligence.