💬 NLP

CacheRL:Multi-Turn Tool-Calling Agents via Cached Rollouts and Hybrid Reward

CacheRL is a system that trains small agent foundation models to achieve near-frontier multi-step tool-calling accuracy with 100 times less compute by leveraging hybrid reasoning traces, a three-tier fuzzy cache to eliminate live execution costs, and cache-tier-aware rewards to ensure robust learning from noisy environments.

Original authors: Md Amirul Islam, Sumiran Thakur, Huancheng Chen, Su Min Park, Jiayun Wang, Gyuhak Kim

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Md Amirul Islam, Sumiran Thakur, Huancheng Chen, Su Min Park, Jiayun Wang, Gyuhak Kim

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a small, smart assistant (a 4-billion-parameter AI model) how to solve complex problems by using a massive toolbox of 1,185 different tools (like weather APIs, code executors, and databases). Usually, you'd need a giant, expensive super-computer brain (like GPT-5) to do this, but that's too costly and slow for everyday use.

The paper introduces CacheRL, a clever system that teaches these small models to act like the big ones, but at a fraction of the cost. Here's how it works, broken down into simple concepts:

1. The Problem: The "Live" Training Trap

Normally, to train an AI to use tools, you have to let it actually use them in the real world during training.

The Analogy: Imagine teaching a student to cook by having them actually buy ingredients, cook, and clean up every single time they practice. It would take forever, cost a fortune in groceries, and you'd waste a lot of food if they made a mistake.
The Reality: For AI, "live" tool use means paying for thousands of API calls, waiting seconds for responses, and risking errors. It's too expensive to do this enough times to learn well.

2. The Solution: The "Cached" Kitchen (CacheAgentLoop)

The researchers built a system called CacheAgentLoop. Instead of letting the AI cook in a real kitchen, they gave it a "simulated kitchen" where the ingredients and results are pre-stored in a giant, smart pantry.

How it works: When the AI says, "I need to check the weather in Tokyo," the system looks it up in its pantry.
- Exact Match: If the pantry has the exact answer, it hands it over instantly.
- Fuzzy Match: If the pantry has a similar answer (e.g., "Tokyo, Japan" vs. "Tokyo, 2024"), it gives the closest one.
- Best Effort: If it's totally new, it gives a generic placeholder.
The Magic: This cuts the cost of training by 100 times. It's like practicing cooking with a photo of the ingredients and a pre-written recipe card instead of buying real food every time.

3. The "Why" vs. The "What" (Hybrid Thinking Trajectories)

Just showing the AI what tool to use isn't enough; it needs to understand why.

The Analogy: Imagine a master chef (GPT-5) writing a cookbook. A bad cookbook just lists steps: "Add salt. Add pepper." A good cookbook explains the logic: "Add salt to draw out moisture, then pepper to enhance the flavor."
The Innovation: The researchers used a giant AI (GPT-5) to rewrite thousands of existing tool-use examples. They added "thinking blocks" (like a chef's internal monologue) to explain the reasoning behind every tool choice. They then taught the small model using these "thinking" examples.
The Result: The small model learned not just the mechanics of calling a tool, but the strategy behind it.

4. The "Fair Judge" (Cache-Tier-Aware Reward)

Here is the tricky part: Since the AI is practicing with a "simulated pantry" (the cache), sometimes the data it gets is slightly wrong or generic.

The Problem: If the AI gets a generic answer from the pantry and fails the task, a standard teacher might punish the AI for being "wrong." But the AI didn't make a mistake; the pantry was imperfect!
The Fix: The researchers created a "Fair Judge."
- If the pantry gave a perfect answer, the judge grades the AI strictly on the final result.
- If the pantry gave a rough or generic answer, the judge ignores the final result and only grades the AI on whether it chose the right tool and thought correctly.
Why it matters: This prevents the AI from getting confused or discouraged by the imperfections of the simulation.

5. The Results: Small but Mighty

After training with this system, the small 4-billion-parameter model achieved 92% accuracy on multi-step tool tasks.

The Comparison: This is almost as good as the giant GPT-5 model (which hit 94%), but the small model is 100 times cheaper to train and run.
The Surprise: The researchers found that the quality of the data (the "thinking" examples and the fair judging) mattered much more than the complex math of the training algorithm itself. If you remove the "thinking" examples, performance crashes by 41%. If you remove the "fair judge," it drops by 17%.

Summary

CacheRL is like a masterclass for small AI assistants. It teaches them by:

Giving them a simulated practice environment (the cache) so they don't have to pay real-world costs.
Providing step-by-step reasoning guides (the hybrid trajectories) so they understand the logic, not just the steps.
Using a smart grading system (the fair judge) that knows when the practice data is imperfect and doesn't punish the student for it.

The result is a small, efficient AI that can handle complex, multi-step tasks almost as well as the giants, making powerful AI agents accessible for real-world use.

` tags) or "user-facing content" (which triggers the generation of new causal reasoning).

Causal Reasoning Generation: For user-facing content, GPT-5 generates explanations detailing why a specific tool was chosen, how previous results inform the current action, and what outcomes are expected.
Preservation of Authenticity: All original tool calls, arguments, and API responses (including errors) are preserved to ensure the model learns real invocation patterns across 1,185 unique tools.

2. CacheAgentLoop (Scalable RL Training)

To eliminate live execution costs, the authors introduce a three-tier fuzzy cache system integrated into the rollout loop:

Three-Tier Cache:
- Exact: Hash-based lookup for perfect matches.
- Fuzzy: Jaccard similarity for similar tool calls.
- Best-Effort: Returns a generic result for the tool if arguments don't match.
Token-Level Masking: The system maintains a binary mask where model-generated tokens (reasoning, tool calls) are marked 1 and injected cached results are marked 0. This ensures gradients flow only through model decisions, preventing the model from being trained on environment responses it did not generate.
Structure Preservation: The loop pre-closes thinking blocks in injected results to guide the model toward producing answers rather than entering endless reasoning loops.

3. Cache-Tier-Aware Reward Design

To solve the attribution problem where models are penalized for cache limitations, the authors propose a dynamic reward weighting mechanism:

Hybrid Reward: $R = \alpha(\tau) \cdot R_{answer} + (1 - \alpha(\tau)) \cdot R_{process}$ .
Dynamic Weighting ( $\alpha$ ):
- Exact Matches: High weight on answer quality (LLM-as-judge).
- Fuzzy/Best-Effort Matches: Shift weight toward process metrics (deterministic checks for valid tool names, thinking blocks, diversity).
This ensures the model is not penalized for incorrect answers derived from imperfect cached data, while still receiving strong learning signals when data quality is high.

4. Iterative Training Pipeline

The system employs a cycle of Supervised Fine-Tuning (SFT), rejection sampling (filtering for high-quality trajectories), and Group Relative Policy Optimization (GRPO). High-scoring trajectories from RL are fed back into SFT, creating a virtuous cycle that refines the model's capabilities.

Key Contributions

First Cached RL System for Small Agents: Enables efficient RL training of 4B-parameter models for multi-turn tool calling without live execution, reducing costs by 100× compared to live execution.
Hybrid Thinking Trajectory Pipeline: A scalable method to augment trajectories with GPT-5 reasoning, teaching small models the causal logic behind tool usage across 1,185 tools.
Cache-Tier-Aware Reward: A novel dynamic weighting scheme that adjusts reward attribution based on cache fidelity, solving the noise problem in imperfect environments.
Performance at Small Scale: Demonstrates that a 4B model can achieve 92% process accuracy on multi-step tool calling, approaching GPT-5's 94%, while requiring 100× less compute.

Results

Performance: CacheRL improves the validation reward of Qwen3-4B-Thinking from 0.43 (SFT baseline) to 0.78. On public benchmarks, the model achieves 92% process accuracy, competitive with frontier models.
Ablation Studies:
- Data Quality: Removing high-quality knowledge transfer (GPT-5 reasoning) causes a 41.2% performance drop.
- Reward Design: Removing cache-tier-aware weighting causes a 17.2% drop, highlighting the importance of not penalizing models for cache limitations.
- RL Impact: While GRPO training provides stability, the performance gain beyond a strong SFT baseline is minimal (only ~3% improvement). This suggests that data quality and reward design are more critical than complex optimization algorithms for building practical small agent models.
- Scale: Expanding the dataset from 1.5K to 44K samples improved generalization, preventing overfitting and stabilizing training over 200+ steps.

Significance and Claims

The paper claims that CacheRL provides a practical blueprint for building deployable agent foundation models. Its primary significance lies in demonstrating that systematic knowledge transfer and intelligent training design are more effective than sophisticated RL algorithms for small models.

Key insights include:

Data Quality Dominates: The quality of the training data (specifically the reasoning traces) is the primary driver of performance, far outweighing the benefits of RL optimization itself.
RL for Stability, Not Gains: In the context of cached rollouts, RL primarily stabilizes training rather than unlocking new capabilities beyond a strong SFT baseline.
Feasibility of Small Agents: Small models (4B) can serve as robust, general-purpose agent foundation models if trained with the right combination of knowledge distillation and cache-aware RL, challenging the notion that massive scale is the only path to complex tool use.

The authors conclude that while cache-based RL has limitations (e.g., staleness, dependency on teacher model quality), it successfully bridges the gap between knowledge transfer and practical deployment, offering a cost-effective path to ubiquitous AI agents.