SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes

Technical Summary: SpatialAct

Problem Statement
Current Vision-Language Models (VLMs) have demonstrated proficiency in observation-conditioned spatial perception and reasoning. However, it remains unclear whether these models can build coherent spatial understanding, act upon it, and refine their actions through multi-turn feedback in dynamic 3D environments. Existing benchmarks largely evaluate models as passive observers (answering questions about static scenes) or entangle high-level spatial reasoning with low-level control in embodied tasks. This leaves a "missing middle ground": the evaluation of whether VLM agents can perform high-level semantic spatial actions, observe resulting state transitions, and maintain consistent reasoning across multiple rounds of interaction. The authors define this capability as action-conditioned spatial reasoning.

Methodology
To address this gap, the authors introduce SpatialAct, a simulator-grounded benchmark designed to probe the reasoning-to-action capabilities of VLM agents in 3D scenes.

• Benchmark Construction: SpatialAct comprises 333 scenes across three layout scenarios: Abstract Geometric (procedurally generated), Urban Architectural (derived from RAISECity), and Indoor Scene (derived from InternScenes). The dataset contains 4,355 QA pairs. Each scene is rendered from both top-view and isometric-view perspectives to provide complementary spatial information.
• Hierarchical Task Design: The benchmark employs a three-level diagnostic structure:
  1. Basic Spatial Abilities: Five fundamental tasks adapted from psychometric theories: Object Meaning, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization.
  2. Single-step Error Detection and Fix: A decomposed task where models identify a spatial error and propose a single corrective action.
  3. Multi-turn Interactive Refinement: The core challenging setting. Models interact with a 3D simulator to iteratively repair abnormal spatial configurations (e.g., collisions, wall conflicts, orientation errors). The model issues executable commands (move, rotate, scale), which are executed in the simulator, and the updated multi-view renderings are returned for continued reasoning.
• Evaluation Metrics: For multi-turn refinement, the authors define five metrics:
  • Repair Rate: The proportion of errors reduced.
  • Scene Success Rate: The proportion of scenes where all errors are resolved.
  • Effective Repair Turn Ratio: The frequency of turns that successfully reduce errors.
  • Premature Stop Rate: The frequency of stopping while errors remain.
  • Average Completion Tokens per Scene: A measure of reasoning cost.

Key Contributions

1. Formalization of Action-Conditioned Spatial Reasoning: The paper establishes this as a distinct evaluation axis for VLM agents, focusing on high-level simulator-executable actions that alter the 3D environment state.
2. Introduction of SpatialAct: A comprehensive benchmark with 333 scenes and 4,355 QA pairs, featuring a hierarchical design that connects basic spatial understanding to multi-turn action-conditioned refinement.
3. Systematic Evaluation and Diagnosis: The authors evaluate leading proprietary and open-source VLMs, revealing a significant "reasoning-to-action gap." They provide a diagnostic framework to analyze failure modes, including error sensitivity and the impact of scene complexity.

Results

• Reasoning-to-Action Gap: While current VLMs perform well on isolated spatial reasoning tasks (e.g., achieving ~80% accuracy in basic categories), they struggle significantly in multi-turn interactive refinement. The strongest model, Gemini-3.1 Pro, achieved a Repair Rate of 0.411 and a Scene Success Rate of 0.206. In contrast, human participants achieved a Repair Rate of 0.911 and a Scene Success Rate of 0.763.
• Model Performance: Proprietary models (Gemini-3.1 Pro, GPT-5.4) substantially outperformed open-source models (Kimi-K2.5, Qwen3.6 variants) and GLM-5V-Turbo. Many open-source models showed Repair Rates close to zero or negative, indicating they often failed to correct errors or introduced new ones.
• Failure Modes: Analysis revealed two coupled failure reasons: diagnosis errors (misidentifying spatial violations) and reasoning-to-action errors (issuing incorrect corrective actions). Models struggled to maintain coherent spatial beliefs across long-horizon state transitions.
• Context and Complexity: Increasing context window size did not significantly improve Repair Rate or Scene Success Rate, suggesting diminishing returns from token budget expansion. Performance degraded as initial scene complexity (number of errors) increased, highlighting difficulties in tracking multiple errors and managing coupled spatial dependencies.
• Error Sensitivity: Models were more effective at repairing orientation errors than conflict errors (collisions/wall conflicts), indicating a bottleneck in constraint-aware, multi-object coordination.

Significance
The paper claims that SpatialAct reveals a clear disconnect between basic spatial understanding and stable multi-turn spatial refinement. The results suggest that current VLM agents lack robust spatial state tracking under action-induced environment changes, even when low-level control is abstracted away. The benchmark serves as a diagnostic tool to identify underlying causes of model failures, such as the inability to maintain cross-turn consistency and perform constraint-aware planning. The authors emphasize that bridging this gap requires improvements in spatial state tracking, coordination across feedback loops, and dependable context-grounded action generation, rather than just enhanced perception.