Technical Summary: SpaceTools – Tool-Augmented Spatial Reasoning via Double Interactive RL

Problem Statement

Vision-Language Models (VLMs) have demonstrated strong qualitative visual understanding but struggle with the metrically precise spatial reasoning required for embodied applications, such as robotics. While recent models excel at open-ended visual questions, they lack the geometric precision, 3D awareness, and multi-step reasoning capabilities needed to translate perception into physical action.

Traditional approaches to enhancing spatial reasoning rely on fine-tuning models on task-specific datasets, which necessitates large-scale annotations and extensive data engineering. Furthermore, while the "agentic" paradigm suggests VLMs could utilize external tools (e.g., depth estimators, segmentation models) to augment their capabilities, current implementations often rely on handcrafted prompting strategies or fixed, predefined tool pipelines. These limitations prevent models from discovering optimal tool-use patterns dynamically. Additionally, applying Reinforcement Learning (RL) directly to a large set of heterogeneous tools creates a prohibitively large search space, causing exploration to fail.

Methodology

The paper introduces SpaceTools, a framework that enables VLMs to coordinate multiple computer vision and robotics tools through a novel training paradigm called Double Interactive Reinforcement Learning (DIRL).

1. Double Interactive Reinforcement Learning (DIRL)

DIRL is a two-phase training framework designed to overcome the search space challenges of multi-tool RL while avoiding the rigidity of fixed pipelines.

• Phase 1: Teaching Phase (Initialization)

  • Goal: Establish basic tool-use capabilities and provide a strong initialization for the policy.
  • Process: The authors combine two data sources:
    1. IRL-Trained Teacher: A model trained via Interactive RL (IRL) using only a single pointing tool. This constrained search space allows for reliable convergence and generates grounded reasoning traces.
    2. Universal Teacher: A frontier model (e.g., Claude Sonnet 4.5) prompted to solve tasks using the full set of tools (pointing, segmentation, 3D bounding boxes, etc.). Only trajectories leading to correct solutions are retained.
  • Training: The base VLM undergoes Supervised Fine-Tuning (SFT) on a dataset composed of these traces (1 part IRL-teacher, 3 parts Universal-teacher).

• Phase 2: Exploration Phase (Refinement)

  • Goal: Refine multi-tool coordination and learn to handle diverse reasoning tasks.
  • Process: The SFT-initialized policy undergoes Interactive RL (IRL) with access to the full toolset.
  • Mechanism: The model interacts with tools in a multi-turn dialogue. It generates reasoning (in <thought> tags), calls tools (in <tool> tags), receives tool outputs, and iterates until a final answer is produced (in <answer> tags).
  • Optimization: The authors use Group Relative Policy Optimization (GRPO) to update the policy based on task-specific rewards (e.g., accuracy for multiple-choice, IoU for bounding boxes, geometric consistency for grasping).

2. Toolshed: Scalable Tool Infrastructure

To support the computationally intensive nature of interactive training with diverse tools, the authors developed Toolshed, a distributed infrastructure.

• Function: It hosts compute-heavy computer vision tools (e.g., SAM2, Depth Pro, RoboRefer, GraspGen) and robotic tools as rapid on-demand services.
• Architecture: It decouples tool resource management from the RL/inference workloads, utilizing resource isolation and asynchronous parallel workers to achieve high throughput and utilization during training.

3. Reward Design

The RL framework employs normalized, task-specific rewards to guide learning:

• Multiple Choice: Binary reward (1 for correct, 0 otherwise).
• 2D Bounding Boxes: Mean IoU (MIoU).
• Pointing: Normalized Negative Distance to Centroid (NNDC), clipped with binary accuracy.
• Pose Estimation: IoU between convex hulls of predicted and ground-truth corners.
• Grasp Estimation: Normalized Negative Coordinate Error (NNCE) to reward geometric alignment.

Key Contributions

1. DIRL (Double Interactive Reinforcement Learning): A novel training paradigm that enables VLMs to learn effective coordination of a large set of heterogeneous tools. It uniquely combines teacher-guided supervision (via single-tool and frontier-model traces) with interactive exploration to stabilize learning in large action spaces.
2. Toolshed: A scalable, open-source platform designed to host diverse, compute-intensive vision and robotics tools, enabling their efficient use during interactive training loops.
3. SpaceTools: A VLM (based on Qwen2.5-VL-3B) trained via DIRL that achieves state-of-the-art performance on spatial reasoning benchmarks and demonstrates reliable real-world manipulation using a 7-DOF robot as a tool.

Experimental Results

Spatial Reasoning Benchmarks

SpaceTools was evaluated on five benchmarks: RoboSpatial-Home, BLINK, RefSpatial, CVBench, and BOP-ASK.

• Performance: SpaceTools achieved state-of-the-art results across nearly all benchmarks, outperforming proprietary models (e.g., GPT-5, Claude Sonnet 4.5, Gemini-ER 1.5) and specialized open-source spatial VLMs.
• Comparisons:
  • Surpassed Gemini-ER 1.5 by +7.5% on RoboSpatial.
  • Exceeded Claude Sonnet 4.5 by +24.4% on pose estimation.
  • Surpassed GPT-5 by +8.3% on grasp prediction.
• Ablation: Removing the IRL-trained teacher caused a sharp performance drop, particularly on tasks requiring fine spatial grounding. Removing the Stage 2 IRL phase also degraded performance, confirming that interactive exploration is critical for refining tool coordination. Tool-free SFT and RL baselines underperformed SpaceTools by significant margins (+12% to +16% on RoboSpatial).

Real-World Robot Manipulation

The authors evaluated SpaceTools on a 7-DOF robot performing pick-and-place tasks.

• Success Rate: SpaceTools achieved an 86% success rate on pick-and-place tasks, outperforming frontier models (GPT-5, Claude Sonnet 4.5) equipped with the same tools.
• Efficiency: SpaceTools demonstrated a Time-to-First-Movement (TTFM) of 10 seconds, significantly faster than the 30–36 seconds observed for other tool-augmented models, indicating more efficient reasoning and tool chaining.
• Behavior: Unlike baselines that sometimes hallucinated tool outputs or failed to chain tools coherently, SpaceTools learned to dynamically adapt strategies (e.g., falling back to self-estimation when a tool fails) and orchestrate a closed perception–action loop.

Significance and Claims

The paper claims that SpaceTools demonstrates that VLMs can acquire complex spatial reasoning capabilities through learned tool coordination rather than requiring architectural modifications or massive, task-specific data engineering.

• Scalability: The work addresses the scalability challenge of tool-augmented reasoning by showing that interactive RL, when properly initialized (via DIRL) and supported by robust infrastructure (Toolshed), can successfully navigate the combinatorial search space of multiple tools.
• Generalization: The model exhibits strong out-of-distribution generalization. For instance, a model trained primarily on one benchmark (RoboSpatial) transferred effectively to others (RefSpatial) where other fine-tuning approaches failed completely.
• Embodied AI: The successful integration of a real robot as a tool validates the framework's ability to bridge the gap between abstract spatial reasoning and physical manipulation, suggesting a viable path for deploying agentic VLMs in real-world robotic systems.

The authors conclude that while tool integration yields clear gains in precise spatial grounding, models still face challenges in interpreting nuanced tool outputs in high-level tasks, highlighting the continued need for research into agentic reasoning.