World Reasoning Arena

1. Problem Statement

Current World Models (WMs) are primarily evaluated on narrow metrics such as next-state prediction and visual fidelity (e.g., pixel-level accuracy or short-term video generation). These existing benchmarks fail to assess the richer simulation capabilities required for intelligent agents, specifically:

• Lack of High-Level Control: Models struggle to follow complex, multi-step natural language instructions (especially those targeting the environment rather than just the agent).
• Long-Horizon Instability: Models suffer from error accumulation, leading to incoherent or physically implausible simulations over extended time horizons.
• Passive Prediction vs. Active Reasoning: Current evaluations treat WMs as passive predictors rather than active simulators capable of "thought experiments" (comparing alternative futures) to support goal-directed planning and decision-making.

There is a significant gap between current model capabilities and the human-level ability to simulate, reason, and plan in complex, dynamic environments.

2. Methodology: The WR-Arena Benchmark

The authors introduce WR-Arena, a comprehensive benchmark designed to evaluate WMs across three fundamental dimensions of "Next World Simulation." The framework moves beyond single-turn perceptual testing to multi-turn, goal-oriented evaluation.

Core Evaluation Dimensions

1. Action Simulation Fidelity:

   • Goal: Evaluate the ability to interpret semantically meaningful, multi-step instructions and generate diverse counterfactual rollouts.
   • Sub-tasks:
     • Agent Simulation: Driving a controllable entity through high-level behaviors while keeping background dynamics stable.
     • Environment Simulation: Applying high-level scene interventions (e.g., changing weather, moving objects) and simulating causal consequences while the agent's policy remains neutral.
   • Metric: Judged by Vision-Language Models (VLMs) on action faithfulness and precision.

2. Long-horizon Forecast:

   • Goal: Assess the ability to sustain accurate, coherent, and physically plausible simulations across extended interactions.
   • Sub-metrics:
     • Transition Smoothness: Measures optical flow continuity and penalizes abrupt velocity changes or "jerks" at round boundaries.
     • Generation Consistency: Tracks content alignment and style consistency over time, applying additive penalties for error accumulation as the rollout progresses.

3. Simulative Reasoning and Planning:

   • Goal: Determine if the WM can act as an internal simulator for goal-directed reasoning, enabling an agent to compare alternative futures.
   • Sub-tasks:
     • Step-Wise Simulation: Selecting the correct next observation from distractors based on a specific action (robotic manipulation).
     • Open-Ended Simulation: Planning in realistic, unstructured household environments (e.g., Agibot dataset).
     • Structured Simulation: Planning in controlled tabletop environments (e.g., Language Table dataset) to test fine-grained reasoning.
   • Protocol: A VLM agent proposes candidate actions; the WM simulates outcomes; the VLM selects the best action to reach a goal. Success is measured by trajectory completion rates.

3. Key Contributions

• WR-Arena Benchmark: The first comprehensive benchmark specifically designed to evaluate WMs on Action Simulation Fidelity, Long-horizon Forecast, and Simulative Reasoning/Planning simultaneously.
• Task Taxonomy: A structured categorization of evaluation tasks that distinguishes between agent-centric and environment-centric control, as well as structured vs. open-ended planning scenarios.
• Novel Metrics: Introduction of specific metrics like Multi-round Smoothness (MRS) and Additive Penalty (AP) to quantify error accumulation and temporal coherence, moving beyond simple frame-to-frame similarity.
• Large-Scale Evaluation: A systematic evaluation of state-of-the-art (SOTA) models, including dedicated World Models (Cosmos, V-JEPA 2, PAN) and Video Generation Models (KLING, MiniMax, Gen-3, WAN 2.x).

4. Experimental Results

The authors evaluated 11 models across the three dimensions. Key findings include:

• Performance Gap: Current models struggle significantly with Environment Simulation (no model surpassed 60% accuracy), revealing a fundamental limitation in modeling scene-level interventions compared to agent-centric actions.
• Long-Horizon Degradation: All models exhibit error accumulation over time. Most models drop below 60% consistency after 7 rounds. PAN demonstrated the most stable performance, maintaining high consistency due to its self-forcing training strategy.
• Planning Utility: Only models that produce semantically actionable rollouts significantly improved downstream planning performance.
  • PAN achieved the highest gains in planning success rates: +26.7% in open-ended settings and +23.4% in structured settings compared to a VLM-only baseline.
  • Pure video generators (e.g., WAN 2.1) and embedding-based models (e.g., V-JEPA 2) showed inconsistent or negligible improvements in planning, often failing to provide causally grounded simulations.
• Model Comparison:
  • PAN (a general, interactable world model with Generative Latent Prediction) achieved the most balanced performance, excelling in instruction following, temporal stability, and planning utility.
  • Commercial Video Generators (KLING, MiniMax) showed strong short-term visual fidelity but failed in long-horizon coherence and instruction alignment.
  • Cosmos and V-JEPA 2 showed promise in specific domains but lacked the unified optimization of understanding, prediction, and control required for robust planning.

5. Significance and Conclusion

• Diagnostic Tool: WR-Arena exposes that high visual fidelity does not equate to reliable world understanding. Models can generate beautiful videos but fail to respect physical constraints or follow complex instructions over time.
• Guideline for Future Research: The results suggest that next-generation WMs must jointly optimize understanding, prediction, and control. Treating these as separate modules leads to brittle performance.
• Path Forward: The benchmark establishes that for WMs to be useful in real-world applications (e.g., autonomous driving, robotics), they must be capable of counterfactual reasoning and long-horizon stability, not just next-frame prediction.
• Open Source: The code and datasets for WR-Arena are publicly available to facilitate further research and development in this field.

In summary, the paper argues that the field must shift from evaluating "how real a video looks" to "how well the model simulates reality for the purpose of reasoning and planning," with PAN currently emerging as the most capable architecture for these advanced tasks.