Technical Summary: SpatialWorld

Problem Statement

Spatial reasoning is a foundational capability for Multimodal Large Language Models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation paradigms (e.g., static Visual Question Answering or pre-recorded video understanding) or simulator-specific pipelines. These approaches fail to assess general interactive spatial understanding because they do not capture the dynamic, partially observable nature of real-world environments. In physical spaces, agents cannot acquire complete information from a single view; they must actively navigate to gather progressive visual evidence, update spatial beliefs, and plan subsequent actions. Furthermore, existing embodied benchmarks often couple agents to specific simulators, sensor assumptions, or low-level action interfaces, making it difficult to distinguish between general spatial reasoning and adaptation to a specific environment's quirks.

Methodology

The authors introduce SPATIALWORLD, a unified benchmark designed to evaluate the interactive spatial understanding of multimodal agents in complex, real-world tasks across heterogeneous 3D environments.

Core Design Principles

1. Pure Egocentric Vision: Agents operate under strict vision-only partial observability. They receive only first-person RGB observations without privileged state information (e.g., depth maps, global occupancy grids, or object coordinates).
2. Cross-Platform Unification: The benchmark integrates eight heterogeneous simulation backends (AI2-THOR, ProcTHOR, VirtualHome, CARLA, EmbodiedCity, and multi-agent variants, plus lightweight 3D games) under a shared, simulator-agnostic protocol.
3. Native MLLM Action Interface: Instead of low-level motor commands, agents express decisions via a unified, text-based action space. This includes high-level primitives for Navigation (e.g., Move), Viewpoint & Posture (e.g., Rotate), Interaction (e.g., Pick/Place), and Task-Control (e.g., EndTask). This allows off-the-shelf MLLMs to decompose tasks via chain-of-thought reasoning without simulator-specific fine-tuning.
4. Execution-Based Verification: Success is determined by terminal-state verifiers that check if the final environment state satisfies the goal, rather than matching a predefined action trajectory. This accommodates diverse reasoning paths and open-ended exploration.

Benchmark Construction

• Scale: The benchmark comprises 760 human-annotated tasks spanning six scenario categories: Daily Household, Work & Study, Entertainment, Travel, Social Collaboration, and Digital Spatial Games.
• Complexity: Tasks are categorized by complexity levels: Navigation (movement only), Interaction (object state changes), and Hybrid (long-horizon navigation + manipulation).
• Pipeline: A rigorous three-stage construction process involves task design, human execution to generate reference trajectories, and cross-validation by expert reviewers to ensure feasibility and instruction clarity.

Evaluation Protocol

The evaluation treats the task as a Partially Observable Markov Decision Process (POMDP). The primary metric is Task Success Rate (TSR), measuring the fraction of tasks where the terminal goal is fully satisfied. A secondary metric, Step Efficiency (SE), evaluates the ratio of the agent's step count to a human-annotated reference length, distinguishing efficient agents from those relying on trial-and-error.

Key Results

The authors evaluated 15 advanced MLLMs (including proprietary models like GPT-5, Gemini-3.1-Pro, and open-source models like Qwen-3.5-397B-A17B) on the full benchmark.

• Low Overall Performance: Robust spatial task solving remains a significant challenge. The strongest model, GPT-5, achieved an average TSR of only 17.4%, while the leading open-source model, Qwen-3.5-397B-A17B, reached 14.1%.
• Success-Efficiency Mismatch: There is a clear disconnect between task success and execution efficiency. Models with higher TSRs do not necessarily achieve higher efficiency; some succeed only through redundant exploration or task-specific shortcuts.
• Domain-Specific Variations: Model rankings vary substantially across domains:
  • GPT-5 leads in Daily Household, Travel, and Social Collaboration tasks.
  • Qwen-3.5-397B-A17B ties GPT-5 in Work & Study and leads in Physical Entertainment.
  • Gemini-3.1-Pro achieves the highest scores in Digital Games.
• Indoor vs. Outdoor: A pronounced domain shift exists. GPT-5 and Qwen-3.5 lead in indoor environments (requiring fine-grained object grounding), while Gemini-3-Flash and GPT-5 perform better in outdoor environments (requiring long-horizon navigation).
• Complexity Bottlenecks: Hybrid tasks combining long-horizon navigation with multi-step manipulation are demonstrably harder (mean TSR ~4.2%) than pure interaction tasks (mean TSR ~50.2%).

Significance and Claims

The paper positions SPATIALWORLD as a rigorous testbed that exposes multiple, separable bottlenecks in spatial reasoning, long-horizon planning, and action execution, rather than reducing evaluation to a single leaderboard score.

• Diagnostic Value: The benchmark reveals that while current models excel at static scene perception, they struggle profoundly with dynamic physical environments. The low success rates and high variance suggest a fundamental gap in robust interactive spatial reasoning.
• Paradigm Shift: The authors envision SPATIALWORLD as a foundational tool to shift MLLM research from passive observation to the realization of general-purpose spatial agents capable of seamless interaction in the real world.
• Limitations: The authors acknowledge that the benchmark operates in simulated environments rather than physical robotic platforms and that the current scale (760 tasks) is modest compared to automatically generated datasets due to the rigorous human annotation required for quality and reproducibility.

In summary, the paper argues that evaluating spatial reasoning requires moving beyond static datasets to unified, interactive, vision-only protocols that reflect the true constraints of physical world interaction. SPATIALWORLD provides the infrastructure to measure these capabilities systematically.