🤖 machine learning

On Effectiveness and Efficiency of Agentic Tool-calling and RL Training

This paper investigates the effectiveness and efficiency of agentic tool-calling by revealing how minor implementation choices significantly skew evaluation results and by introducing techniques to accelerate reinforcement learning training without compromising performance.

Original authors: Tong Liu, Cheng Qian, Matej Cief, Yuan He, Daniele Dan, Nikolaos Aletras, Gabriella Kazai

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tong Liu, Cheng Qian, Matej Cief, Yuan He, Daniele Dan, Nikolaos Aletras, Gabriella Kazai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant (a Large Language Model) that knows a lot of facts from its training. But to get things done in the real world, it needs to use tools—like calling a weather API, searching the web, or checking a calendar. This paper is about two big questions: How well do we know if the robot is actually good at using these tools? and How can we teach it to use them without wasting a fortune on electricity and computer time?

Here is the breakdown using simple analogies:

Part 1: The "Effectiveness" Problem (Is the Test Fair?)

The authors found that measuring how good a robot is at using tools is surprisingly fragile. It's like judging a chef's cooking skills, but the score changes wildly depending on tiny, often unspoken details in the kitchen.

The "Random Seed" Roulette: Imagine flipping a coin to decide the order of ingredients. The paper found that if you change this random starting point (the "seed"), the robot's score can jump around significantly, especially in long, multi-step tasks. It's like a student getting a different grade on the same test just because they sat in a different seat.
The "Storytelling" Format: How you tell the robot about the past matters. The paper compared two ways of showing the robot a conversation history:
- Method A (Native): Showing the history as a natural back-and-forth chat.
- Method B (Context): Dumping the whole history into one giant block of text.
- The Result: The robot performed 6–8% better with Method A. It turns out the robot understands a natural conversation flow much better than a giant wall of text, even if the information is the same.
The "Thinking" History: Should the robot see its own previous thoughts (like "I need to check the weather first")? The paper found that keeping these thought traces visible helps the robot stay on track, improving scores by about 3–5%.
The "Teacher's Note" (System Prompt): Sometimes, just adding a tiny sentence to the robot's instructions (e.g., "Remember to act as a user in multi-turn chats") boosted performance as much as months of extra training. This suggests that many "improvements" seen in other papers might just be because the teacher gave better instructions, not because the student learned better.

The Big Takeaway: If you don't standardize these tiny details (how you format the chat, which random seed you use, what instructions you give), comparing different robots is like comparing apples to oranges. The leaderboard rankings might be misleading.

Part 2: The "Efficiency" Problem (Stop Wasting Time)

Once we know how to test fairly, the authors looked at how we train these robots using Reinforcement Learning (RL). They found the current training process is incredibly wasteful, like a student studying for a test by re-reading pages they already know perfectly.

They identified two main sources of waste:

1. The "Boring Homework" Problem (Zero-Variance Prompts)

The Issue: During training, the robot is given many practice problems. For about 80% of them, the robot either gets them 100% right immediately or fails completely in a way that teaches nothing. These are "zero-variance" prompts—they provide no learning signal. It's like a teacher making a student solve 1+1=2 a thousand times; it doesn't help them learn.
The Fix: The authors created a "skip list." If the robot gets a specific problem right three times in a row, the system stops wasting time generating new attempts for it and moves to a harder problem. This saves a massive amount of computer time.

2. The "Over-Engineering" Problem (High Update Costs)

The Issue: To learn, the robot usually tries a problem multiple times (say, 8 times) to see which answer is best. The computer then spends a huge amount of energy updating the robot's brain based on all 8 attempts. The paper found that the "brain update" part takes 3 to 5 times longer than the actual "trying" part.
The Fix: Instead of using all 8 attempts to update the brain, the system picks only the most useful ones—specifically, the best answer and the worst answer. This creates the biggest contrast for learning. By ignoring the "middle of the road" attempts, they cut the update time significantly without hurting the robot's learning.

The Final Result

By fixing the "waste" in the training process, the authors made the robot learn 1.7 to 2.6 times faster (in terms of real-world clock time) without making the robot any dumber. In fact, the robot got better at using tools in multi-step conversations, beating several other famous models.

In summary: The paper argues that we need to stop treating robot benchmarks like a casual game and start treating them like a rigorous science experiment (fixing the "Effectiveness"). Simultaneously, we need to stop training robots like they are paying for every second of their thinking time and start being smarter about what we actually learn from (fixing the "Efficiency").

Technical Summary: On Effectiveness and Efficiency of Agentic Tool-calling and RL Training

1. Problem Statement

The paper addresses two critical, complementary challenges in the development of Large Language Model (LLM) agents equipped with tool-calling capabilities: evaluation effectiveness and training efficiency.

Effectiveness (Evaluation Fragility): While tool-calling is central to modern agents, the reliability of benchmark results is questionable. The authors argue that reported performance is often highly sensitive to seemingly minor, undocumented implementation choices (e.g., random seeds, prompt templates, history handling). This fragility undermines reproducibility and makes cross-paper comparisons unreliable, potentially causing the field to chase superficial gains while overlooking robust methods.
Efficiency (Computational Waste): Training tool-calling agents via Reinforcement Learning (RL), particularly using algorithms like PPO and GRPO, is computationally expensive. The authors identify two specific sources of waste:
1. Rollout Waste: A significant portion of prompts (up to 80%) produce "zero-variance" rollouts where all sampled trajectories receive identical rewards, yielding no gradient signal for learning.
2. Update Cost: The policy update stage in tool-calling incurs a disproportionately high computational cost compared to rollout generation, driven by long context sequences (tool schemas, multi-turn history, I/O).

2. Methodology

2.1. Analysis of Evaluation Effectiveness

Using the BFCL (Berkeley Function Calling Leaderboard) benchmark as a case study, the authors systematically varied specific components of the evaluation pipeline to measure their impact on performance:

Random Seeds: Evaluated performance across 10 seeds for five models (Qwen and Llama series) to quantify variance.
Multi-turn Templates: Compared "native" serialization (role-content messages formatted by the chat template) against "context" concatenation (injecting full history into a single user turn).
Thinking History: Analyzed the impact of retaining intermediate reasoning traces (Chain-of-Thought) versus omitting them in multi-turn interactions.
System Prompts: Tested the sensitivity of results to minor manual modifications in system instructions tailored for multi-turn interactions.
Training Data Format: Conducted controlled experiments comparing models trained on pure single-turn data versus pure multi-turn data to isolate the contribution of supervision format.

2.2. Proposed Efficiency Techniques

Guided by the identification of computational bottlenecks, the authors propose two lightweight acceleration techniques for RL-based tool-calling training:

Online Pre-rollout Filtering:
- Mechanism: Before generating expensive rollouts, the system checks a lightweight cache of per-prompt "all-correct" streaks. If a prompt has been solved correctly in the previous $k$ epochs (typically $k=1$ or $2$), it is skipped for the current epoch.
- Rationale: Prompts that are uniformly solved tend to remain solved for subsequent epochs. This dynamic filtering avoids redundant sampling of uninformative prompts without requiring static pre-training filtering.
Variance-aware Rollout Down-sampling:
- Mechanism: For each prompt, the model generates $n$ rollouts as usual. However, during the policy update step, only a subset of $m$ rollouts ( $m < n$ ) is backpropagated. The subset is selected to maximize reward variance (specifically, selecting the $m'$ lowest-reward and $m-m'$ highest-reward rollouts).
- Rationale: This retains the most contrastive learning signals (high vs. low reward) while reducing the computational load of the policy update stage, which is the dominant cost in tool-calling scenarios.

3. Key Results

3.1. Evaluation Sensitivity Findings

Seed Variance: Single-turn performance is stable across seeds, but multi-turn evaluation shows significant variance (up to ~3% deviation) due to error compounding over turns.
Template Impact: Using the "native" multi-turn template yields a consistent 6–8% performance gain over the "context" concatenation approach.
Thinking History: Retaining thinking history improves multi-turn performance by 2–5% for reasoning models.
System Prompts: Minor, targeted changes to system prompts can produce gains comparable to or larger than those attributed to RL training, highlighting the risk of confounding variables.
Data Format: In controlled settings, training on pure multi-turn data did not reliably improve multi-turn performance and sometimes degraded it compared to single-turn training. This suggests that trajectory quality and noise accumulation are more critical than the mere presence of multi-turn supervision.

3.2. Efficiency and Performance Gains

Zero-Variance Prevalence: Approximately 80% of prompts initially produce zero-variance rollouts, confirming significant rollout waste.
Training Speedup: The proposed efficiency techniques achieve substantial wall-clock speedups without degrading final performance:
- 1.7× speedup in single-turn settings.
- 2.6× speedup in multi-turn settings.
Benchmark Performance:
- The RL-trained model (Qwen3-4B) achieved an average score of 39.4 on BFCL multi-turn, outperforming the base model and several open-source baselines.
- On ACEBench, the model improved by 12.1 absolute accuracy points over the base model, outperforming Nova1-Lite and other open-source models.
Generalization: Downstream evaluations (HellaSwag, MMLU, TruthfulQA, Winogrande) showed that the RL training did not meaningfully degrade general capabilities.

4. Significance and Claims

The paper claims to provide the first systematic analysis of the fragility of tool-calling evaluation pipelines, arguing that the community must adopt more rigorous standardization (e.g., reporting seeds, standardizing templates and prompts) to ensure meaningful comparisons.

Regarding efficiency, the authors demonstrate that RL training for tool-calling can be made significantly faster by addressing specific computational bottlenecks inherent to the task (long contexts and high rates of uninformative prompts). They assert that their techniques offer a practical path to reducing the environmental and financial footprint of developing capable agents without sacrificing performance.

The work does not propose new tool-use datasets or agent architectures but focuses on improving the methodology of evaluation and the efficiency of the training process. The authors note that while more efficient training lowers barriers to deployment, it necessitates responsible safety evaluation and monitoring to prevent potential misuse.