Technical Summary: PEARL (Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use)

Problem Statement

While Large Language Models (LLMs) have demonstrated significant potential when augmented with external tools, they face substantial hurdles in complex, multi-turn tool invocation scenarios. Current approaches often suffer from:

• Weak Planning: Agents frequently adopt myopic, one-step strategies, failing to formulate or adhere to coherent, long-horizon plans.
• Execution Errors: Models exhibit "tool hallucination" (invoking non-existent tools), generate erroneous parameters, and struggle to manage dependencies between sequential tool calls.
• Poor Adaptability: When execution errors occur, agents often fail to diagnose the issue or dynamically adjust their strategy, leading to repetitive failure loops.

Existing methods, largely reliant on supervised fine-tuning (SFT) or imitation learning, are brittle when faced with novel scenarios or require complex error recovery. There is a critical need for a framework that decouples robust execution from strategic planning and utilizes adaptive learning to optimize both.

Methodology: The PEARL Framework

PEARL (Plan Exploration and Adaptive Reinforcement Learning) addresses these challenges through a principled, two-stage architecture that separates Strategic Plan Generation from Informed Execution.

1. Task Formulation

The agent's goal is to solve a complex query $q$ using a set of tools $\mathcal{T}$ by generating an ordered sequence of sub-tasks (a plan) $P = ((s_1, \tau_1), \dots, (s_n, \tau_n))$, where $s_i$ is a natural language goal and $\tau_i$ is the selected tool.

2. Stage 1: Strategic Plan Generation (Online RL)

A dedicated Planner model ($\pi_\theta$) is trained to generate the entire multi-step plan before execution begins.

• Optimization Algorithm: The Planner is optimized using Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO). GRPO enhances training stability by normalizing advantage estimates across a group of candidate plans generated for the same input.
• Reward Design: A novel, planning-centric reward function is employed to provide dense, step-by-step feedback.
  • The reward compares the generated plan against a ground-truth plan.
  • It assigns a normalized reward ($1/m$) for each correctly selected tool at step $i$, where $m$ is the number of steps in the optimal plan.
  • This design directly optimizes the structural and semantic correctness of the tool chain, solving the credit assignment problem in long-horizon planning.

3. Stage 2: Informed Execution (Offline + Online)

An Executor model ($\pi_\phi$) is responsible for carrying out the plan generated by the Planner.

• Offline Tool Exploration: Before any task execution, the agent engages in a proactive, trial-and-error phase with each available tool. It systematically attempts various parameter configurations to learn valid usage patterns, argument constraints, and failure conditions. This builds a "learned user manual" that grounds the executor in reality, drastically reducing hallucinations and parameter errors.
• Online Step-by-Step Execution: The Executor receives the Planner's output and executes it iteratively. It leverages the knowledge acquired during the offline phase to construct valid tool calls with correct parameters, utilizing the history of previous execution results to inform subsequent steps.

Key Contributions

1. Novel Framework: PEARL integrates offline tool exploration for robust execution with online reinforcement learning for adaptive, strategic planning, effectively decoupling the two complex problems.
2. Planning-Centric Reward Mechanism: The authors design a specific reward function that enables the direct optimization of multi-step tool plans via RL. This guides the agent toward optimal strategies by providing dense feedback on plan quality rather than relying solely on sparse final task success signals.
3. State-of-the-Art Performance: The framework establishes new performance benchmarks, demonstrating that specialized architectural design can outperform simple model scaling.

Experimental Results

The authors evaluated PEARL on two benchmarks: ToolHop (multi-hop tool reasoning) and T-Eval (tool utilization capabilities).

• ToolHop Performance: PEARL-7B achieved a 56.5% Success Rate (SR), significantly outperforming the specifically tuned Qwen2.5-7B-SFT baseline (46.2%) and larger proprietary models like GPT-4o (49.0%). Crucially, it maintained a low Invocation Error Rate (IER) of 3.8%, validating the efficacy of the offline exploration phase.
• T-Eval Generalization: On the generalization benchmark, PEARL-7B achieved a 77.0% SR, surpassing leading proprietary models (GPT-4o at 76.0%, Claude 3.5 Sonnet at 71.0%) and open-source models. Its IER dropped to 1.0%, indicating high reliability on unseen tasks.
• Planner Generalization: Experiments showed that plans generated by the PEARL-Planner significantly improved the performance of other powerful models (e.g., GPT-4o and Qwen2.5-72B) when used as executors, confirming the transferability of the learned planning strategies.
• Ablation Studies:
  • Removing the planning-centric reward caused SR to plummet from 56.5% to 23.4%, highlighting its critical role in credit assignment.
  • Removing offline exploration caused IER to spike from 3.8% to 24.1%, proving that pre-emptive tool learning is essential for reliable execution.

Significance and Claims

The paper claims that PEARL represents a key advance in addressing the complex planning challenges of tool use. By systematically separating the learning of tool mechanics (offline) from the learning of strategic orchestration (online RL), the framework produces agents that are both robust (low error rates) and strategic (high success rates in multi-step tasks).

The authors assert that their work contributes to the development of more reliable and autonomous LLM-based agents. The results suggest that specialized training architectures for planning and execution are more effective for complex tool-use tasks than simply scaling up model parameters or relying on standard supervised fine-tuning. The high generalization capability of the PEARL Planner further validates the quality of the learned strategies, suggesting a path toward agents capable of handling increasingly complex, real-world digital interactions.