🤖 AI

Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning

The paper proposes EAPO, an efficient agentic policy optimization framework that mitigates tool abuse in reinforcement learning by learning selective tool use through tool-free trajectories, difficulty-aware reward shaping, and confidence-aware token reweighting, thereby improving reasoning accuracy while significantly reducing unnecessary tool calls across multiple models and benchmarks.

Original authors: Liuji Chen, Dianxing Tang, Xing Shi, Dingshuo Chen, Qiang Liu, Shu Wu, Liang Wang

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liuji Chen, Dianxing Tang, Xing Shi, Dingshuo Chen, Qiang Liu, Shu Wu, Liang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student who has just learned how to use a super-powered library (the internet) and a high-tech calculator (Python code). You want them to solve math problems and answer trivia questions.

At first, the student is a bit over-enthusiastic. Even when you ask, "What is 1 + 1?", they immediately run to the library to find a book about addition, or they fire up the calculator to crunch the numbers. They get the right answer, but they wasted time, energy, and money doing something they could have done in their head. This is what the paper calls "Tool Abuse."

The researchers behind this paper, EAPO, wanted to teach this student a better lesson: "Learn when not to act."

Here is how they did it, broken down into simple concepts:

1. The Problem: The "Over-Reliant" Student

In the past, AI models trained with Reinforcement Learning (RL) were rewarded simply for getting the right answer. If using a tool helped get the answer, the model learned to use it every time, even for silly, easy questions. It's like a chef who uses a food processor to chop a single clove of garlic because they know it works, even though a knife would be faster and free.

2. The Solution: The "Efficient" Training Camp (EAPO)

The authors created a new training method called EAPO (Efficient Agentic Policy Optimization). Think of it as a three-step coaching strategy:

Step A: The "No-Tools" Drill (Efficiency-Aware Rollout)

Usually, when training these AI students, they are allowed to use tools on every practice question. EAPO changes the rules. For every batch of practice questions, the coach forces the student to solve some of them without using any tools at all.

The Analogy: Imagine a driving instructor who sometimes says, "Okay, today you must drive this easy route without using your GPS."
The Result: This forces the AI to realize, "Hey, I actually know this answer! I don't need the GPS." It creates a clear comparison: "I solved this without tools, and I solved that with tools. Which was better?"

Step B: The "Difficulty" Scorecard (Difficulty-Aware Reward Shaping)

The coach doesn't punish the student for using tools on hard questions. They only penalize the student for using tools on easy questions where they should have known the answer.

The Analogy: If you ask a math whiz to solve "1+1" and they use a calculator, the coach says, "That's a waste of time, you lose points." But if you ask them to solve a complex physics equation and they use a calculator, the coach says, "Great job! That tool was necessary."
The Result: The AI learns to be smart about when to reach for the tool, rather than just using it less often overall.

Step C: The "Confidence" Spotlight (Confidence-Aware Reweighting)

The coach pays extra attention to the moments the student is unsure.

The Analogy: If the student is confident but gets the answer wrong, the coach says, "You were too sure of yourself; let's look at that again." If the student is unsure but gets the answer right, the coach says, "You took a risk and it paid off; remember that feeling."
The Result: The AI learns to trust its own "gut feeling" (internal reasoning) more when it's right, and be more careful when it's guessing.

3. The Results: Smarter and Faster

The paper tested this method on three different AI models (Qwen and Llama) across nine different types of challenges, from hard math problems to complex trivia.

Better Accuracy: The models got more questions right overall.
Fewer Tools: They used the tools significantly less often (about 18% to 24% fewer calls).
The Trade-off: They achieved this without slowing down or getting worse at the hard stuff. They just stopped using the tools for the easy stuff.

Summary

The paper argues that a truly smart AI agent shouldn't just know how to use tools; it should know when to put them down. By forcing the AI to practice solving problems without tools and by only punishing tool use when it's unnecessary, EAPO teaches the AI to be efficient, saving time and resources while actually getting better at solving problems.

Technical Summary: Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning

1. Problem Definition: Tool Abuse in Agentic RL

Recent advances in Large Language Models (LLMs) have spurred the development of Agentic Reinforcement Learning (RL) frameworks that enable models to interact with external tools (e.g., search engines, Python interpreters) to solve complex reasoning tasks. While these frameworks improve problem-solving capabilities, they introduce a critical issue known as tool abuse.

In standard Agentic RL, models are rewarded primarily for task success. Consequently, agents often learn to treat tool invocation as a default reasoning strategy, even for queries that can be solved efficiently through internal reasoning (e.g., simple arithmetic or basic factual questions). This behavior leads to:

Unnecessary Computation: Redundant tool calls increase latency and deployment costs, particularly when external services are billed per use.
Inefficient Exploration: Uniform penalties or hard limits on tool usage (e.g., in OTC or standard GRPO) often fail to distinguish between necessary tool use for difficult queries and unnecessary use for easy ones. Such uniform constraints may suppress useful exploration on genuinely challenging tasks or fail to penalize abuse on trivial ones.
Lack of Training Signals: Standard rollout procedures typically sample trajectories under tool-available settings. Without explicit tool-free trajectories, the training process lacks direct evidence to determine whether a query is solvable by the model's intrinsic reasoning alone, making it difficult to distinguish necessary tool use from abuse.

2. Methodology: Efficient Agentic Policy Optimization (EAPO)

To address these challenges, the authors propose EAPO, a multi-tool Agentic RL framework designed to optimize the trade-off between reasoning effectiveness and tool-use efficiency. EAPO introduces three core mechanisms:

2.1 Efficiency-Aware Rollout

Unlike standard rollouts where all trajectories may invoke tools, EAPO ensures that for each query, the rollout group of $M$ trajectories includes at least $n$ tool-free trajectories.

Implementation: During training, the prompt explicitly prohibits tool usage for a subset of trajectories, and the rollout module disables tool execution for these samples.
Mechanism: This creates an explicit contrast between tool-augmented and tool-free solutions. It allows the model to learn whether tool invocation yields genuine performance improvements for a specific query.
Efficiency Coefficient: An efficiency coefficient $\varepsilon(T_i) = \exp(-\beta(c_i - c_{min}))$ is calculated for each trajectory, where $c_i$ is the number of tool calls and $c_{min}$ is the minimum tool calls among correct trajectories. This penalizes trajectories with redundant tool calls relative to the most efficient correct solution in the group.

2.2 Difficulty-Aware Reward Shaping

To avoid overly aggressive penalties that might hinder learning on difficult tasks, EAPO incorporates the inherent difficulty of the query into the reward signal.

Query Difficulty ( $d(q)$ ): Defined as $1 - \frac{1}{M}\sum I(T_i)$ , where $I(T_i)$ indicates trajectory correctness. If most trajectories in a group are incorrect, the query is deemed difficult.
Reward Adjustment: The adjusted reward $\tilde{R}_i$ $\tilde{R}_{i}$ is computed as:
$\tilde{R}_i = [d(q) + (1 - d(q))\varepsilon(T_i)] R_i$
- For easy queries (low $d(q)$ ), the reward is heavily modulated by the efficiency coefficient $\varepsilon(T_i)$ , strongly penalizing unnecessary tool use.
- For hard queries (high $d(q)$ ), the reward relies more on the base correctness reward $R_i$ , preserving the incentive to explore tool-assisted solutions even if they involve more calls.

2.3 Confidence-Aware Advantage Reweighting

EAPO refines policy learning by adjusting the weight of tokens based on the model's generation confidence ( $C_t = \log p_t(y_t)$ ).

Strategy:
- Correct Answers: Tokens with low confidence (below the $\rho$ -quantile) are up-weighted ( $W_{pos}$ ), emphasizing the learning signal from uncertain but correct reasoning steps.
- Incorrect Answers: Tokens with high confidence (above the $1-\rho$ -quantile) are down-weighted ( $W_{neg}$ ), reducing the impact of overconfident errors.
Goal: This reweighting facilitates the learning of a more effective policy by focusing on high-information-density tokens and mitigating the noise from confident mistakes.

3. Key Contributions

Efficiency-Aware Rollout Mechanism: A novel training strategy that introduces tool-free trajectories to enable the model to distinguish between necessary tool use and excessive dependence.
EAPO Framework: A joint optimization framework combining difficulty-aware reward shaping and confidence-aware token reweighting to simultaneously improve reasoning performance and tool-use efficiency.
Empirical Validation: Extensive experiments across nine benchmarks (mathematical and knowledge-intensive) and multiple model families (Qwen2.5-3B/7B, Llama3.1-8B) demonstrating consistent improvements in the accuracy-efficiency trade-off.

4. Experimental Results

The authors evaluated EAPO against baselines including Base (no tools), TIR (prompt-based tool use), GRPO, Reinforce++, ToolStar, ARPO, and AEPO.

Performance Gains: EAPO consistently achieved the best or second-best performance across all benchmarks.
- Compared to GRPO, EAPO improved average performance by 10.45% (Qwen2.5-3B), 7.27% (Qwen2.5-7B), and 9.69% (Llama3.1-8B).
- On mathematical reasoning (Pass@1) and knowledge-intensive tasks (F1), EAPO outperformed strong baselines like ARPO and AEPO.
Efficiency Gains: EAPO significantly reduced unnecessary tool usage without sacrificing accuracy.
- Compared to GRPO, EAPO reduced average tool calls by 18.33% (Qwen2.5-3B), 18.33% (Qwen2.5-7B), and 24.59% (Llama3.1-8B).
- The reduction was more pronounced on 7B models than 3B models, suggesting smaller models have less redundant tool use to eliminate.
Ablation Studies: Removing the cold-start SFT caused the largest performance drop. Removing reward shaping led to a sharp increase in tool calls on QA tasks. Removing advantage reweighting made the policy more conservative.
Hyperparameter Sensitivity: The framework is robust to the tool penalty parameter $\beta$ and confidence reweighting weights, provided they are within reasonable ranges.

5. Significance and Claims

The paper claims that EAPO demonstrates that agentic models can learn when not to use tools without compromising the benefits of tool-integrated reasoning.

Shift in Paradigm: The work moves beyond simply reducing tool frequency to identifying and penalizing unnecessary tool use based on query difficulty and intrinsic capability.
Practical Deployment: By improving the accuracy-efficiency trade-off, EAPO makes agentic RL more suitable for practical deployment where tool usage incurs non-trivial computational and financial costs.
Limitations: The authors note that current benchmarks focus heavily on hard problems. Assessing tool abuse effectively requires an "easy-to-hard" dataset that simultaneously evaluates effectiveness and efficiency, which is an area for future evaluation paradigms.

In summary, EAPO provides a structured approach to teaching agents to balance internal reasoning with external tool invocation, ensuring tools are used as a supplement to reasoning rather than a default crutch.