Technical Summary: PERIA – Building Tool-Augmented Visual Agents for Spatial Reasoning

Problem Statement

Recent Vision-Language Models (VLMs) have demonstrated strong multimodal understanding but remain limited in spatial reasoning tasks that require active evidence acquisition and multi-step visual interaction. These tasks—such as subway-map reasoning, visual probing, and vision reconstruction—demand more than reasoning over static visual representations. They require models to ground fine-grained evidence, extract task-relevant details, and reason about spatial relations through intermediate steps.

Existing tool-augmented VLMs show promise in structured tasks (e.g., chart understanding) where answers derive from explicit evidence. However, they often fail in complex spatial tasks because:

1. Lack of Global Perception: They miss local visual details or fail to recover global spatial structure.
2. Tool-Use Gap: Simply providing access to visual tools does not guarantee competent usage. Open-source VLMs often degrade in performance when given raw tool access without dedicated training, failing to select the right tools, invoke them correctly, or integrate feedback effectively.
3. Training Limitations: Existing methods often use limited tool spaces or domain-specific data, lacking a unified policy to compose perception and interaction tools for general spatial reasoning.

Methodology: PERIA

The authors introduce PERception-Interaction-reason Agent (PERIA), a tool-augmented visual agent designed to bridge the gap between having tools and using them effectively. PERIA organizes visual problem-solving into a unified three-stage process: Perceive (gather global spatial evidence), Interact (perform fine-grained local analysis and verification), and Reason (synthesize observations to produce an answer).

1. Tool Sandbox

PERIA operates within a modular tool sandbox containing 18 tools divided into two complementary families:

• Vision Perception Tools: Designed to expose textual, symbolic, and spatial evidence. Examples include text_spotting, map_text_ocr, grounding_dino (for object detection), and auto_segment. These tools transform images into structured observations (coordinates, text, bounding boxes).
• Vision Interaction Tools: Designed to manipulate the visual context, trace paths, and verify spatial relations. Examples include image_crop, draw_path, bounding_box, and image_highlight. These tools allow the agent to zoom in, annotate, or trace routes on the image.

2. Training Pipeline

The training of PERIA involves three distinct stages:

• Stage A: Tool Sandbox Construction: The environment is built with function-based utilities and lightweight tool agents (e.g., PaddleOCR-VL, Grounding DINO, SAM3) to support flexible tool invocation.
• Stage B: Supervised Fine-Tuning (SFT) Trajectory Synthesis:
  • A proprietary model (GPT-5) is used to generate high-quality, executable tool-use trajectories via an explore-and-exploit strategy.
  • The process involves expanding reasoning-action prefixes until a correct answer is found or a turn budget is reached.
  • An expert open-weight model (Qwen3-VL-235B) diversifies the intermediate reasoning styles (thoughts) while keeping tool calls and observations fixed, creating a diverse dataset for SFT.
• Stage C: Reinforcement Learning with OR-GIGPO:
  • To optimize multi-step tool use, the authors propose Observation-Relaxed Group-in-Group Policy Optimization (OR-GIGPO).
  • Motivation: Standard Group-in-Group Policy Optimization (GIGPO) relies on exact state matching. In visual tool use, semantically equivalent tool calls may return slightly different observations (e.g., OCR text variations), making exact matching too restrictive.
  • Mechanism: OR-GIGPO replaces exact state matching with observation-relaxed matching. It groups rollouts by task input and uses a relaxed anchor group for step-level advantage calculation, matching observations that are semantically similar (using string similarity metrics) rather than identical.
  • Credit Assignment: The algorithm combines episode-level advantages (trajectory quality) and step-level advantages (local tool-decision quality) to assign credit across multi-step trajectories.
  • Reward Design: A composite reward function includes penalties for repetition, format rewards for valid tool-use tags, and correctness rewards (binary for QA, continuous NDTW-based for route tracing).

Key Contributions

1. Unified Framework: Introduction of PERIA, a general framework for tool-augmented visual agents that integrates perception and interaction tools for reasoning, planning, and verification across diverse spatial tasks.
2. Diverse Toolset & Trajectory Synthesis: Development of a specialized perception-interaction toolset and the synthesis of a high-quality, diverse trajectory dataset for supervised fine-tuning, enabling reliable tool invocation.
3. OR-GIGPO Algorithm: Proposal of Observation-Relaxed Group-in-Group Policy Optimization, a novel reinforcement learning method that handles the variability of visual tool observations, enabling effective fine-grained credit assignment for multi-step tool use.
4. Empirical Validation: Demonstration that PERIA-8B significantly outperforms strong baselines and approaches the performance of much larger proprietary models.

Experimental Results

The authors evaluated PERIA-8B on 13 benchmarks spanning 8 datasets, covering both in-distribution (MapTrace, ReasonMap, Visual Probing) and out-of-distribution tasks (Ball Tracking, Paper Folding, Cube 3-View, etc.).

• Performance Gains: PERIA-8B improves over the Qwen3-8B backbone by 10.0% on in-distribution benchmarks and 4.4% on out-of-distribution benchmarks.
• Comparison to SOTA: It outperforms previous state-of-the-art baselines of similar size (e.g., VTool-R1, Mini-o3) by 7.0%–14.8%.
• Scaling: PERIA-8B achieves performance comparable to significantly larger models, such as Qwen3-VL-235B-A22B-Thinking and GPT-5.
• Ablation Studies:
  • OR-GIGPO: Removing RL (w/o RL) causes a substantial drop (avg -7.2% on in-distribution), and OR-GIGPO outperforms other RL algorithms (GRPO, DAPO) by significant margins.
  • Tool Access: Removing tools (w/o Tool) leads to imbalanced performance and catastrophic forgetting in certain spatial domains.
  • SFT: Training directly with RL without SFT (w/o SFT) results in poor performance, highlighting the necessity of supervised trajectory synthesis.
  • Tool Types: Both perception and interaction tools are necessary; removing either family degrades performance.

Significance and Claims

The paper claims that trained visual tool use is a critical direction for building capable spatial reasoning agents. The key insight is that the gap between "having tools" and "using tools" is a policy-learning problem. By combining a diverse toolset with a unified training recipe (SFT + OR-GIGPO), PERIA enables agents to:

• Acquire global spatial evidence.
• Perform fine-grained local verification.
• Reason over accumulated observations to solve complex spatial tasks.

The authors emphasize that their approach allows smaller models (8B) to rival much larger proprietary models, suggesting that effective tool-use training can provide gains beyond simple backbone scaling. They acknowledge limitations, such as the focus on 8B models due to computational constraints and the current 10-turn interaction limit, and suggest future work in extending to 3D reasoning, embodied tasks, and larger trajectory datasets.