💬 NLP

AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning

AdaReasoner is a family of multimodal models that achieves state-of-the-art visual reasoning and generalization to unseen tools by learning tool orchestration as a general skill through a scalable data pipeline, Tool-GRPO reinforcement learning, and an adaptive mechanism for dynamic tool regulation.

Original authors: Mingyang Song, Haoyu Sun, Jiawei Gu, Linjie Li, Luxin Xu, Ranjay Krishna, Yu Cheng

Published 2026-01-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mingyang Song, Haoyu Sun, Jiawei Gu, Linjie Li, Luxin Xu, Ranjay Krishna, Yu Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but slightly clumsy assistant who is trying to solve a complex puzzle, like navigating a maze or fixing a broken picture. Sometimes, this assistant gets stuck because they can't see the details clearly, or they forget the rules of the maze.

AdaReasoner is a new way of training this assistant so they stop trying to do everything in their head and start knowing exactly when to ask for help, which tool to ask for, and how to use it.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Do-It-All" Trap

Current AI models are like students who are told to solve a math problem using only their brain. If the problem requires a calculator, the student might guess the answer or try to do the math in their head and get it wrong. They don't know when to grab the calculator, or they might grab the wrong one (like a ruler instead of a calculator).

2. The Solution: A "Tool Orchestra"

The researchers created AdaReasoner, which treats tools (like a magnifying glass, a map, or a calculator) like instruments in an orchestra. The AI isn't just a soloist anymore; it's a conductor.

It knows when to play: It learns that for some tasks, it needs to zoom in (a "magnifying glass" tool), but for others, it needs to draw a line (a "ruler" tool).
It knows when to stop: If a tool doesn't help, it learns to put it down and try a different approach.
It adapts to new instruments: Even if you give the AI a brand-new tool it has never seen before, it can figure out how to use it just by reading the instruction manual (the tool's description).

3. How They Trained It (The Three-Step Recipe)

To teach the AI these skills, the researchers used a three-step training process:

Step 1: The "Scripted Rehearsal" (Tool Cold Start)
Imagine a drama teacher giving the actor a perfect script for a play. The AI is shown high-quality examples of how to solve problems step-by-step, using tools correctly. This teaches it the basic "moves" and how to hold the tools.
- Analogy: It's like showing a chef exactly how to chop an onion before letting them cook.
Step 2: The "Trial and Error" Game (Tool GRPO)
Once the AI knows the basics, they let it play a game where it gets points for solving the puzzle.
- If it uses the right tool at the right time, it gets a big reward.
- If it uses a tool that doesn't help, or guesses without checking, it gets a smaller reward or a penalty.
- Analogy: This is like a video game where the AI learns that using a "jetpack" to cross a river is great, but using a "jetpack" to open a door is a waste of time. It learns to optimize its strategy.
Step 3: The "Secret Code" Challenge (Adaptive Learning)
To make sure the AI isn't just memorizing the names of the tools (like memorizing that "Tool A" is always for measuring), the researchers changed the names and descriptions of the tools randomly during training.
- Analogy: Imagine teaching someone to drive a car, but every day you change the name of the gas pedal from "Gas" to "Go," "Fuel," or "X7a2." The student has to learn what the pedal does based on its function, not its name. This ensures the AI can handle any tool, even ones it has never seen before.

4. The Results: Small Brain, Big Skills

The most exciting part of the paper is that they took a relatively small AI model (a "7B" model, which is like a smart college student) and gave it these tool skills.

The Result: This small model became so good at using tools that it beat much larger, "closed-source" models (like GPT-5 and Claude Sonnet 4) on difficult visual puzzles.
The Analogy: It's like giving a bicycle a set of high-tech gears and a GPS. Suddenly, the bicycle can race against a Ferrari because the bicycle knows exactly how to navigate the terrain, while the Ferrari is just driving blindly.

5. Real-World Examples from the Paper

The paper shows the AI doing things like:

Navigating a Maze: Instead of guessing, it uses a tool to find the start and end points, then uses a "pathfinding" tool to draw the perfect route, avoiding holes.
Fixing a Jigsaw Puzzle: It uses a tool to find the missing black spot in a picture, then tries inserting different puzzle pieces until one fits perfectly.
Reading a Website: It uses a "crop" tool to zoom in on a specific button and an "OCR" tool (like a digital eye) to read the text on it, figuring out exactly how to buy something.

Summary

AdaReasoner teaches AI models to stop trying to be perfect at everything internally. Instead, it teaches them to be smart managers who know when to call in an expert tool to do the heavy lifting. By learning to adapt to new tools and tasks on the fly, even a small AI can solve complex visual problems better than much larger, more expensive systems.

Technical Summary: DAREASONER (AdaReasoner)

Problem Statement

Multimodal Large Language Models (MLLMs) often struggle with complex visual reasoning tasks that require fine-grained perception, multi-step planning, and intermediate verification. While humans rely on external tools to solve problems beyond their immediate cognitive or perceptual capabilities, current MLLMs lack the ability to adaptively use tools. Existing approaches typically rely on rigid, pre-defined invocation patterns or are constrained to single-tool trajectories. Consequently, they fail to learn when to use a tool, which tool to select, or how to coordinate multiple tools dynamically, especially when faced with unseen tools or novel task distributions. The core challenge is to enable MLLMs to treat tool selection and orchestration as a general reasoning skill rather than a task-specific behavior.

Methodology

The authors introduce AdaReasoner, a framework designed to enable adaptive, multi-turn tool planning. The methodology is built upon three core innovations:

1. High-Quality Trajectory Data Curation (Tool Cold Start - TC)

To establish a robust foundation, the authors propose a data curation pipeline that synthesizes high-fidelity, multi-turn reasoning trajectories. This process involves:

Abstract Trajectory Design: Manually designing optimal problem-solving blueprints for tasks like Visual Spatial Planning (VSP), Jigsaw puzzles, and GUI QA.
Complex Scenario Integration: Deliberately incorporating reflection and backtracking (where the model corrects sub-optimal outcomes) and explicit tool failure scenarios (where tools return errors, forcing the model to rely on intrinsic capabilities).
CoT Generation: Using powerful LLMs to generate Chain-of-Thought (CoT) reasoning that connects tool calls, teaching the model not just what to call, but why and how to reason between steps.

2. Tool-GRPO (Tool Group Relative Policy Optimization - TG)

The framework extends the Group Relative Policy Optimization (GRPO) algorithm to handle multi-turn tool-calling trajectories. The reward function is designed to optimize long-horizon strategic planning:

Multi-turn Reward Accumulation: The total reward ( $R_{total}$ ) combines a Format Reward (strict adherence to syntax), a Tool Reward (hierarchical scoring based on structure, name, and parameter correctness), and an Accuracy Reward (based on the final answer).
Adaptive Reward for Tool Use: An asymmetric reward structure encourages concise reasoning. If the final prediction is correct, the model is rewarded regardless of tool usage. If incorrect, the reward depends on the quality of tool usage, granting partial credit for informative tool-based reasoning while penalizing unsupported guessing. This treats tools as a fallback mechanism under uncertainty rather than a mandatory step.

3. Adaptive Learning (ADL) for Generalization

To prevent overfitting to specific tool identifiers and enhance generalization to unseen tools, the authors introduce an adaptive learning mechanism:

Token-Level Randomization: Tool names and parameter names are replaced with random alphanumeric strings (e.g., GetWeather $\rightarrow$ Func_X7a2). This forces the model to infer functionality solely from semantic descriptions and context.
Semantic-Level Paraphrasing: Tool and argument descriptions are rephased using an LLM (Gemini 2.5 Flash) to alter syntactic structure while preserving functional meaning, ensuring the model learns robust semantic understanding rather than specific phrasing.

Key Contributions

A Comprehensive Method for Tool-Augmented Models: The paper proposes a unified pipeline integrating a data curation recipe for multi-turn planning, an RL framework (Tool-GRPO) for interaction, and an adaptive learning method to enhance generalizability.
AdaReasoner Model Family: The introduction of a new family of state-of-the-art models capable of complex tool planning. These models exhibit self-adaptive behaviors, learning to autonomously adopt beneficial tools, discard irrelevant ones, and modulate usage frequency based on task demands without explicit supervision for these specific behaviors.
Performance and Generalization: The work demonstrates that smaller open-source models (e.g., 7B parameters) can achieve performance competitive with or superior to leading proprietary systems (e.g., GPT-5, Claude Sonnet 4) on structured reasoning tasks. It establishes that effective tool orchestration can shift the performance bottleneck from model scale to tool utility.

Experimental Results

The authors evaluated AdaReasoner on diverse benchmarks including Visual Spatial Planning (VSP, VSPO), Jigsaw puzzles (Jigsaw-COCO, BLINK-J), and GUI QA (GUIChat, WebMMU).

Performance Gains: The 7B variant of AdaReasoner achieved an average improvement of +24.9% over its base model across benchmarks. Specifically, it reached 97.64% on VSP and 96.60% on Jigsaw, outperforming GPT-5 (80.10% on Jigsaw) and Claude Sonnet 4.
Adaptive Tool Usage: In ablation studies, the model demonstrated the ability to:
- Adopt Beneficial Tools: Increase invocation frequency of useful tools (e.g., A* for pathfinding) when introduced during inference.
- Discard Irrelevant Tools: Suppress the use of tools that do not contribute to the task (e.g., A* for verification tasks) after RL training.
- Modulate Frequency: Dynamically adjust the number of tool calls per sample based on task complexity.
Generalization: When evaluated on unseen tool definitions (via randomization) and unseen tasks (zero-shot transfer), AdaReasoner maintained high accuracy and tool usage success rates, whereas other methods suffered significant performance degradation.
Scale Independence: The tool-augmented approach allowed both 3B and 7B models to achieve near-perfect accuracy on VSP, suggesting that tool utility can overcome intrinsic scale limitations.

Significance

The paper argues that the primary bottleneck in multimodal reasoning is shifting from the model's internal reasoning capacity to the effectiveness of its tool planning. By treating tool usage as a generalizable reasoning skill rather than a fixed capability, AdaReasoner enables smaller, open-source models to autonomously curate tool-use strategies. This approach allows MLLMs to handle complex, multi-step visual problems with a level of adaptability and robustness previously seen only in larger proprietary systems, effectively bridging the capability gap between open-source and closed-source models in structured reasoning domains.