💬 NLP

AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

AutoTool is a training framework that enhances LLM agents' adaptability to evolving toolsets by employing a dual-phase optimization pipeline and a large-scale dataset to enable dynamic tool selection, resulting in superior performance across diverse reasoning tasks compared to existing methods.

Original authors: Jiaru Zou, Ling Yang, Yunzhe Qi, Sirui Chen, Mengting Ai, Ke Shen, Jingrui He, Mengdi Wang

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaru Zou, Ling Yang, Yunzhe Qi, Sirui Chen, Mengting Ai, Ke Shen, Jingrui He, Mengdi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but slightly rigid assistant (an AI) who is great at solving puzzles but gets stuck when the tools in its toolbox change.

The Problem: The "Fixed Toolbox" Trap
Most current AI assistants are trained like a chef who only knows how to cook with the exact 10 knives and 5 pots sitting on their counter. If you give them a new type of pan or a different kind of knife they've never seen before, they freeze. They are trained to use specific tools for specific tasks, but they can't adapt if the "kitchen" evolves or if you hand them a brand-new gadget.

The Solution: AutoTool
The paper introduces AutoTool, a new training method that turns the AI into a master "tool shopper" rather than just a tool user. Instead of memorizing a fixed list of tools, AutoTool teaches the AI to:

Look at the problem.
Scan a massive, ever-changing shelf of thousands of tools.
Pick the right one on the fly, even if it's a tool the AI has never seen before.

How It Works: The Two-Step Training Camp
The researchers trained the AI using a two-phase "boot camp" approach:

Phase 1: Learning to Think and Act (Stabilization)
First, they taught the AI how to think through a problem step-by-step (like a long chain of thoughts) and how to smoothly switch between thinking and using a tool. Imagine teaching a student how to write an essay and how to use a calculator without getting confused about when to do which. This phase makes the AI's reasoning stable and coherent.
Phase 2: Learning to Choose (Refinement)
This is the secret sauce. Once the AI knows how to use tools, they taught it how to pick the best one.
- The Analogy: Imagine a game where the AI has to pick a tool to solve a math problem. In the past, it might just guess. With AutoTool, the AI is shown many different attempts. It learns that picking the "Code Interpreter" for a math problem is like picking the "Right Key" for a lock, while picking a "Search Engine" is like picking the "Wrong Key."
- The Magic Trick (Embeddings): Instead of memorizing tool names (like "Tool A" or "Tool B"), the AI learns the vibe or essence of the tools. It's like learning that a "hammer" feels heavy and is for hitting, rather than just remembering the word "hammer." This allows the AI to look at a brand-new, strange tool and say, "This feels like a hammer, so I'll use it to hit this nail," even if it's never seen that specific hammer before.

The "Library" They Built
To teach this, the researchers didn't just use a few examples. They built a massive library of 200,000 training examples involving over 1,000 different tools (like code runners, search engines, and image readers) across 100+ tasks.

Crucial Detail: They trained the AI on 460 tools, but then tested it on a library of 1,346 tools. This means 65% of the tools the AI used during the test were completely new to it. It had to figure them out on the spot.

The Results: A Smarter, More Flexible Assistant
When they tested AutoTool against other advanced AI models:

It got better at Math and Science (solving complex problems).
It got better at Search (finding the right answers on the web).
It got better at Coding (writing computer programs).
It got better at Multimodal tasks (understanding images and charts).

The Bottom Line
AutoTool proves that an AI doesn't need to be trained on every single tool in existence to be useful. By teaching it how to choose and giving it a way to understand the "essence" of tools, it can walk into a room full of gadgets it has never seen and still pick the right one to solve the problem. It's the difference between a robot that only knows how to use a specific screwdriver and a human who can look at a toolbox, figure out which tool fits the screw, and get the job done.

Technical Summary: AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

1. Problem Statement

Recent advances in Agentic Reinforcement Learning (RL) have enabled Large Language Models (LLMs) to generate long chain-of-thought (CoT) trajectories interleaved with external tool use. However, existing approaches typically operate under the assumption of a fixed inventory of tools predefined for specific tasks. While effective in closed environments, these methods fail to capture realistic scenarios where:

Agents must select appropriate tools from complex, domain-diverse toolsets.
New, unseen tools are introduced during inference that were not present during training.

Current models risk overfitting to static tool lists and lack the ability to generalize to evolving tool environments. The core challenge addressed by this work is: How can LLM agents dynamically select tools from evolving toolsets during agentic reasoning while preserving strong reasoning capabilities?

2. Methodology: AutoTool Framework

The authors propose AutoTool, a training framework designed to equip LLM agents with dynamic tool-selection capabilities. The framework integrates reasoning and tool use into a unified trajectory and employs a dual-phase optimization pipeline.

2.1. Tool-Aware Trajectory Generation

The framework conceptualizes a reasoning trajectory $\tau$ as an interplay of three components:

Inner-reasoning ( $\tau^{reason}$ ): Standard CoT internal reasoning.
Tool-selection ( $\tau^{select}$ ): The agent reasons over the toolset $\mathcal{T}$ to select a tool $t_k$ .
Tool-integration ( $\tau^{integrate}$ ): Executing the selected tool and feeding feedback back into the generation.

To handle evolving toolsets, AutoTool utilizes Embedding-Anchored Tool Selection:

Tool Embeddings: Each tool $t_k$ is paired with a feature description $\mu(t_k)$ (name, functionality, schema) and encoded into a latent embedding $e_{t_k}$ using the LLM's internal embedding layer.
Selection Mechanism: Instead of generating tool names directly (which fails on unseen tools), the agent generates a selection rationale $s_i$ and a predicted embedding anchor $e'_i$ . The tool is selected by sampling from a softmax distribution based on the Frobenius norm distance between $e'_i$ and the candidate tool embeddings $e_{t_k}$ . This allows the agent to select new tools via representation alignment rather than memorized identifiers.

2.2. Dual-Phase Optimization Pipeline

The training process consists of two distinct phases:

Phase I: Trajectory Stabilization

Goal: Equip the agent with stable long-form CoT reasoning and valid tool-integration patterns.
Method: The agent is initialized from a base reference model ( $\pi_{\theta_{ref}}$ ) and trained via Supervised Fine-Tuning (SFT) followed by RL-based policy optimization (e.g., GRPO). This ensures the model can generate coherent trajectories involving reasoning, selection, and execution steps.

Phase II: Tool-Selection Refinement

Goal: Explicitly optimize the tool-selection process to handle dynamic toolsets.
Method: The optimization focuses specifically on tool-selection steps, masking internal reasoning and integration steps. The authors frame tool selection as a Plackett–Luce (PL) Ranking problem.
- Theoretical Bridge: The authors prove that optimizing the KL-regularized RL objective is equivalent to matching the policy's induced PL ranking distribution over candidate trajectories.
- Objective: The policy is optimized to align with a reward-guided ranking where trajectories with better tool-selection rationales and correct final answers are ranked higher.
- Loss Function: A tractable Cross-Entropy (CE) loss is used to train the policy $\pi_\theta$ towards the optimal policy $\pi^*$ , minimizing the difference between the model's induced ranking and the reward-based ranking.

3. Data Curation

To support dynamic tool environments, the authors constructed a 200k-scale dataset with explicit tool-selection rationales.

Scope: Covers 1,346 tools (Code, Search, Multimodal) and 120+ task types across mathematics, science, code generation, and multimodal understanding.
Pipeline:
1. Collection: Assembled tools and tasks.
2. Rationale Generation: Used an expert reasoning model (DeepSeek-R1) to generate explicit justifications for why a specific tool was chosen at each step, ensuring alignment between context and selection.
3. Augmentation: Filtered invalid rationales and inserted valid ones into trajectories to create high-quality training data.
Evolving Setting: Only 460 tools (34.2%) were used during training. The remaining 886 tools were kept unseen and reserved exclusively for inference to evaluate generalization.

4. Experimental Results

Experiments were conducted on ten diverse benchmarks using Qwen3-8B and Qwen2.5-VL-7B as base models.

4.1. Performance Gains

AutoTool consistently outperformed advanced LLM agents and specialized tool-integration methods:

Math & Science Reasoning: +6.4% average gain.
Search-based QA: +4.5% average gain.
Code Generation: +7.7% average gain.
Multimodal Understanding: +6.9% average gain.
Comparison: AutoTool (8B/7B parameters) achieved performance comparable to or exceeding much larger models (e.g., Qwen2.5-72B, GPT-4o) and specialized RL baselines (e.g., ReTool, Search-R1) across all domains, demonstrating balanced robustness.

4.2. Generalization to Unseen Tools

Unseen Tool Performance: When evaluated on a fully unseen toolset (886 tools) during inference, AutoTool maintained strong performance, with a marginal drop of less than 2% compared to using the full toolset.
Hit Rate: Higher tool-selection hit rates (selecting the correct tool) directly correlated with improved downstream accuracy.
Ablation: Models trained with tool-selection rationales significantly outperformed those without, increasing successful tool executions by 13.5%.

5. Significance and Claims

The paper claims that AutoTool represents a significant step toward extensible agentic systems capable of operating in evolving environments.

Beyond Static Tools: Unlike prior work that assumes fixed tool inventories, AutoTool enables agents to adaptively choose tools from dynamic, open-ended sets.
Generalization: By anchoring selection in embedding space and optimizing via PL ranking, the framework achieves strong generalization to tools never seen during training.
Efficiency: The method achieves these results with fewer parameters (8B/7B) than many competing state-of-the-art agents, highlighting the efficiency of explicit tool-selection optimization.
Practical Impact: The framework addresses the gap between closed-loop training and real-world deployment where toolsets are fluid and continuously expanding.

The authors conclude that explicit optimization of tool selection, rather than just tool usage, is critical for building agents that can effectively reason and act in complex, changing environments.