🤖 AI

LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models

LAST introduces a unified framework featuring an interactive sandbox (LAST-Box) and a progressive training strategy that enables multimodal large language models to effectively invoke specialized vision tools and interpret their outputs as hints, thereby significantly enhancing spatial reasoning performance and outperforming strong proprietary models.

Original authors: Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Yang Chen, Ziqiao Shang, Lan-Zhe Guo, Yu-Feng Li

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Yang Chen, Ziqiao Shang, Lan-Zhe Guo, Yu-Feng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read librarian named MLLM (Multimodal Large Language Model). This librarian has read every book in the world and can describe a picture of a living room in beautiful, poetic detail. They know what a "sofa" is, what "wood" looks like, and can tell you the general vibe of the room.

However, if you ask this librarian, "How far is the sofa from the coffee table?" or "Is that lamp taller than the bookshelf?", they often start guessing wildly. They might say the lamp is 10 feet tall when it's actually 3, or claim the sofa is right next to the door when it's actually in the corner. They are great at telling stories, but terrible at measuring reality.

This paper introduces a solution called LAST (Leveraging Tools as Hints) to fix this problem. Here is how it works, broken down into simple concepts:

1. The Problem: The "Hallucinating" Librarian

The authors noticed that even the smartest AI models struggle with spatial reasoning. They can recognize objects, but they can't accurately judge distances, sizes, or 3D layouts.

The Old Way: Researchers tried to make the librarian smarter by feeding them more and more books (data). But no matter how many books they read, the librarian still couldn't "internalize" the rules of physics and geometry. They kept guessing.
The New Idea: Instead of forcing the librarian to memorize geometry, why not give them a toolbox? Let the librarian ask an expert for help when they get stuck.

2. The Solution: The "LAST-Box" (The Magic Toolbox)

The authors built a special sandbox called LAST-Box. Think of this as a high-tech workshop where the librarian can grab specific tools without needing to know how to build them from scratch.

Atomic Tools (The Raw Tools): Imagine a toolbox containing a laser measure, a ruler, a depth camera, and a magnifying glass. These are powerful but complicated to use. If you ask the librarian to "use the laser measure," they might fumble and break it because the instructions are too technical.
Spatial Skills (The Pre-Made Kits): This is the genius part of LAST. The authors wrapped those complicated tools into easy-to-use "kits."
- Instead of saying "Run the depth algorithm on the sofa," the librarian just says, "Use the 'Estimate Depth' kit."
- The kit does all the hard math, takes a picture of the result, and hands the librarian a simple note: "The sofa is 2 meters away."
- This turns a complex, error-prone process into a simple, one-step instruction.

3. The Training: Teaching the Librarian to Use the Tools

You can't just hand a toolbox to a librarian and expect them to know how to use it. The authors used a three-stage training camp to teach the model:

Stage 1: The Warm-up (Learning to Read the Tools):
First, they showed the librarian pictures of what the tools produce (like depth maps or segmentation masks) and asked, "What does this squiggly line mean?" This teaches the librarian that a "depth map" isn't just a weird picture; it's a map of distance.
Stage 2: Supervised Learning (The Guided Practice):
Next, they gave the librarian practice problems with a teacher watching over their shoulder. If the librarian tried to guess the distance, the teacher said, "No, look at the tool output first." They taught the librarian a cycle: Think → Ask Tool → Read Result → Answer.
Stage 3: Reinforcement Learning (The Exam):
Finally, they let the librarian take a test alone. If they used the right tool and got the right answer, they got a gold star (reward). If they ignored the tool or used the wrong one, they got a gentle nudge to try again. This taught the model to choose the right tool for the right job on its own.

4. The Results: From Guessing to Precision

When they tested this new system (called LAST-7B):

It got smarter: The model improved its spatial reasoning by about 20% compared to its original version.
It beat the giants: Even though it's a smaller, open-source model, it outperformed massive, expensive, closed-source models (like the latest versions of GPT or Gemini) on tasks involving measuring distances and sizes.
It works for everyone: They even showed that if you give this "toolbox" to other smart models that can't be retrained, those models also get significantly better at spatial tasks.

The Big Picture

Think of LAST as giving a brilliant but clumsy artist a set of precision drafting tools.

Before: The artist tried to draw a perfect circle freehand. It looked okay, but it was wobbly.
After: The artist uses a compass (the tool). The circle is perfect.

The paper proves that we don't need to make AI models infinitely larger to make them smarter. Instead, we just need to teach them how to ask for help from specialized experts (tools) and how to understand the answers those experts give them. This turns a "guessing game" into a precise science.

1. Problem Statement

Multimodal Large Language Models (MLLMs) have made significant strides in general visual understanding but struggle with spatial reasoning, particularly in tasks requiring precise geometric layouts, scale estimation, and fine-grained topological relationships.

Limitations of Current Approaches: Existing methods rely heavily on data-driven scaling (fine-tuning with more spatial data), which fails to help models internalize structured geometric priors. MLLMs frequently suffer from hallucinations regarding spatial relations and fail to comprehend complex depth or perspective information.
The Tool-Augmentation Gap: While specialized vision models (e.g., for depth estimation or object localization) exist, integrating them into MLLMs is challenging due to two main issues:
1. Invocation Complexity: Spatial tools are heterogeneous and parameter-rich, making robust multi-step invocation difficult for MLLMs.
2. Output Interpretation: Tools produce low-level outputs (e.g., raw segmentation masks, depth maps) that MLLMs struggle to interpret and convert into high-level reasoning cues. Directly feeding these outputs often overwhelms the model, acting as noise rather than helpful hints.

2. Methodology

The authors propose LAST (Leveraging Tools as Hints), a unified framework designed to bridge the gap between MLLMs and specialized vision tools. The methodology consists of two core components:

A. LAST-Box: An Extensible Interactive Sandbox

LAST-Box acts as a middleware that abstracts heterogeneous vision models into a structured interface for MLLMs.

Atomic Operations: The system defines a set of foundational modules (e.g., GroundingDINO for detection, DepthAnything-v2 for depth, SAM for segmentation) as atomic operations.
Spatial Skills Abstraction: Instead of asking the MLLM to call raw APIs, LAST-Box composes atomic operations into reusable spatial skills (e.g., EstimateDepth, EstimateSize, SegmentObjects).
- Input/Output: These skills take raw images/text and return multimodal hints: a visual output (e.g., a cropped image with an overlay) coupled with a semantic textual description.
- Benefit: This shortens the tool-calling trajectory, shields the model from low-level parameter complexity, and provides interpretable, model-friendly cues.

B. Progressive Training Strategy

To teach MLLMs how to effectively use these tools, the authors propose a three-stage curriculum:

Stage 1: Warm-up (Tool Output Understanding):
- The model is trained to recognize and interpret tool outputs (e.g., distinguishing a depth map from a texture map) using QA pairs generated from tool-augmented views.
- Goal: Ensure the model understands the semantics of the visual feedback before using it for reasoning.
Stage 2: Supervised Fine-Tuning (SFT) (Instruction Following):
- The model learns the "Think-Act-Observe-Reason" cycle.
- A teacher model (GPT-4o) generates reasoning trajectories where the model decides to call a tool, analyzes the result, and derives an answer.
- Consistency Check: The training data includes scenarios where tools fail or provide partial results, forcing the model to learn when to trust the tool and when to revert to the original image.
Stage 3: Interactive Reinforcement Learning (IRL) (Adaptive Decision Making):
- Using the GRPO algorithm, the model is optimized to make autonomous tool invocation decisions without pre-defined reasoning paths.
- Reward Function: A composite reward encourages correct formatting, answer accuracy, and appropriate tool usage (evaluating if the tool call was useful for the final result).

3. Key Contributions

Atomic-to-Skill Tool Abstraction: A framework that composes low-level tool invocations into high-level, reusable spatial skills, significantly reducing the complexity of long-horizon tool calling.
Progressive Tool-Aware Training: A novel three-stage training paradigm (Warm-up $\to$ SFT $\to$ IRL) that systematically guides models from understanding tool semantics to proficient, adaptive tool invocation.
Open-Source High-Performance Model: The release of LAST-7B, a 7B parameter model that integrates the LAST-Box sandbox and training strategy, achieving state-of-the-art results on spatial benchmarks.

4. Experimental Results

The authors evaluated LAST-7B on four diverse spatial reasoning benchmarks: CVBench, EmbSpatial, SPBench-SI, and MSMU.

Performance Gains:
- LAST-7B achieved an average performance gain of ~18.3% over its backbone (Qwen2.5-VL-7B).
- It significantly outperformed specialized spatial models (e.g., SpaceLLaVA, Robopoint) and even surpassed strong proprietary closed-source APIs like GPT-5.1 and Gemini-2.5-Pro on average.
- Notable improvements were seen in metric-heavy tasks: Size Estimation (+32.96%), Relative Direction (+31.04%), and MSMU (+26.97%).
Tool Effectiveness:
- Tool Success Rate: LAST-7B achieved a high tool success rate (88.67%–99.67%).
- Impact of Success: When tools were used successfully, accuracy improved by up to 29% (e.g., on SPBench-SI) compared to unsuccessful attempts, proving the model effectively leverages the hints.
Generalizability:
- Applying the LAST-Box framework to closed-source models (Gemini-2.5-Pro) without fine-tuning also yielded significant performance boosts (e.g., +18% on 2D-REL), demonstrating the framework's utility as an external reasoning aid.
Ablation Studies:
- Removing any of the three training stages resulted in performance degradation, confirming the necessity of the full progressive curriculum.
- Using both text and image hints (multimodal) outperformed single-modality hints, validating the design of the multimodal output in LAST-Box.

5. Significance

Bridging the "Metric Gap": LAST successfully addresses the long-standing inability of MLLMs to perform precise geometric and metric quantification, transitioning them from coarse semantic perception to fine-grained spatial measurement.
Paradigm Shift: The paper advocates for a shift from purely data-driven scaling to tool-augmented learning, where MLLMs act as orchestrators of specialized, mature vision models rather than attempting to relearn these capabilities from scratch.
Practical Applicability: By providing an open-source, high-performance model and a modular sandbox, LAST offers a scalable solution for real-world applications requiring spatial reasoning, such as robotics navigation, embodied AI, and 3D scene understanding.