Technical Summary: SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning

Problem Statement

Spatial reasoning—the ability to determine object locations, relationships, and movements in 3D space—remains a fundamental challenge for Vision-Language Models (VLMs). While tool-augmented agents attempt to bridge this gap by integrating specialist perception modules (e.g., depth estimators, segmenters), their effectiveness is often constrained by the action interface used to invoke these tools.

Existing approaches typically rely on one of two interfaces, both of which exhibit limitations for open-ended, complex 3D/4D reasoning:

1. Single-pass code execution: The agent writes a complete Python program before execution. This forces a commitment to a full analysis strategy before observing any intermediate results (e.g., segmentation masks or depth maps), preventing adaptation to unexpected visual evidence.
2. Structured tool-calls: The agent invokes tools via predefined commands (e.g., JSON/XML). While structured, this interface often lacks the flexibility to compose operations dynamically or utilize external scientific libraries (e.g., numpy, scipy) for task-specific computations that emerge only at test time.

Neither design adequately supports the iterative, compositional reasoning required for complex spatial tasks where computations (like finding the nearest point via KD-tree or estimating a surface plane via RANSAC) cannot be anticipated by a fixed API.

Methodology: SpatialClaw

The authors propose SpatialClaw, a training-free framework that redefines the action interface for spatial agents by treating code as the action interface. Instead of a one-shot program or a structured command dispatch, SpatialClaw utilizes a persistent, multi-turn Python kernel.

Core Architecture

The system operates through an iterative loop comprising five stages:

1. Planning: A separate planner LLM (isolated from the main agent) generates a high-level analysis plan based on the question and tool documentation, without seeing the input images.
2. Code Generation: The main VLM-backed agent generates a single executable Python cell per step. This cell is conditioned on the question, the plan, and all prior execution feedback.
3. Code Execution: The code runs in a persistent kernel pre-loaded with:
   • Input Data: InputImages (frames) and Metadata.
   • Perception Primitives: Tools like tools.Reconstruct (Depth Anything 3) and tools.SAM3 (Segment Anything 3) for depth, camera pose, and segmentation.
   • Scientific Libraries: numpy, scipy, and matplotlib for geometric computation and visualization.
   • State Management: Intermediate results (masks, point clouds, variables) persist as Python variables across turns, allowing the agent to inspect, compose, and revise previous steps.
4. Feedback Assembly: The system captures stdout, variable summaries, execution errors, and visual outputs (via show()) and appends them to the agent's context for the next step.
5. Answer Submission: The agent terminates by calling ReturnAnswer() once sufficient evidence is gathered.

Key Design Principles

• Stateful Iteration: Unlike single-pass methods, the agent can observe intermediate evidence (e.g., a failed segmentation) and revise its strategy (e.g., switching from a centroid-based distance to a KD-tree nearest-neighbor search) without restarting.
• Expressive Composition: The agent is not limited to a fixed set of tool calls. It can freely combine perception outputs with numerical primitives to solve novel problems, such as chaining a depth estimate with a segmentation mask in a way specific to the query.
• Disciplined Reasoning: A unified system prompt enforces general spatial reasoning principles (e.g., cross-checking visual evidence, preferring metric computation over pixel impressions) without requiring benchmark-specific fine-tuning.

Key Contributions

1. Rethinking the Action Interface: The paper argues that the action interface is a critical determinant of agent capability in spatial reasoning. It introduces SpatialClaw, which replaces rigid single-pass or structured interfaces with a persistent, code-as-action interface that enables flexible composition and revision.
2. Comprehensive Evaluation: The framework is evaluated across 20 spatial reasoning benchmarks covering single-image, multi-view, general spatial, video/4D, and general video understanding tasks.
3. Generalization without Adaptation: SpatialClaw achieves consistent improvements across six VLM backbones (ranging from 26B to 397B parameters from Qwen and Gemma families) without any model-specific tuning, benchmark-specific engineering, or changes to the system prompt.

Results

Evaluated on 20 benchmarks, SpatialClaw achieves an average accuracy of 59.9%, outperforming the recent state-of-the-art spatial agent (SpaceTools) by +11.2 points.

• Performance Gains: The most significant improvements are observed in dynamic 4D video reasoning and multi-view inference (e.g., +18.3% on DSI-Bench, +14.3% on MindCube), categories that demand chained geometric computations across frames and viewpoints.
• Interface Comparison: In controlled comparisons using the same toolset and backbones, SpatialClaw consistently outperforms both Single-Pass Code and Structured Tool-Call baselines.
• Ablation Studies:
  • Removing predefined utility wrappers (keeping only core perception tools and scientific libraries) results in minimal performance degradation, suggesting the persistent kernel and scientific primitives are the primary drivers of success.
  • Removing perception tools entirely (keeping only the code interface) still yields a +2.7% gain over a no-tool baseline, isolating the value of the action interface itself.
• Failure Analysis: The primary failure modes are attributed to perception quality (VLM hallucinations, tool limitations) rather than the action interface, indicating that while the interface enables better reasoning, it is bounded by the quality of the underlying perception tools.

Significance and Claims

The paper claims that the design of the action interface is a highly impactful yet underexplored axis for improving spatial reasoning agents. By enabling agents to treat code as an orchestration space for sequencing, composing, and diagnosing perception tools, SpatialClaw demonstrates that expressive action interfaces can drive significant performance gains independent of model capacity.

The authors emphasize that SpatialClaw is training-free, allowing existing VLMs to be extended with stronger spatial reasoning capabilities without additional data or fine-tuning. This is particularly valuable for domains like robotics and embodied AI where data collection is expensive. The work suggests that future improvements in spatial reasoning may rely less on scaling model parameters and more on designing interfaces that facilitate flexible, iterative, and self-correcting reasoning loops.