APEX-Searcher: Augmenting LLMs' Search Capabilities through Agentic Planning and Execution

1. Problem Statement

While Retrieval-Augmented Generation (RAG) effectively mitigates hallucinations and provides access to external knowledge for Large Language Models (LLMs), standard RAG frameworks struggle with complex multi-hop questions. These questions require synthesizing information from multiple, interdependent pieces of evidence.

• Limitations of Current Approaches: Existing solutions primarily rely on Iterative RAG, which increases the number of retrieval rounds or refines queries reactively. However, these methods often lack a global strategic plan. They operate at a "tactical" level (reacting to the last piece of information) rather than a "strategic" level (decomposing the problem beforehand).
• Core Gap: Without a coherent task decomposition strategy, iterative methods suffer from inefficient search paths, over-planning, repetitive queries, and disruption by irrelevant documents. The paper argues that enhancing the LLM's intrinsic task planning ability is more critical than simply increasing retrieval iterations.

2. Methodology: APEX-Searcher Framework

The authors propose APEX-Searcher (Agentic Planning and EXploration Searcher), a hierarchical framework that decouples the reasoning process into two distinct, synergistic phases:

Phase 1: RL-Driven Agentic Planning (High-Level)

• Objective: To decompose a complex multi-hop query ($Q$) into a logical, executable sequence of simpler sub-questions ($S = \{s_1, s_2, ..., s_n\}$).
• Mechanism: A Planning Agent (an LLM) is trained using Reinforcement Learning (RL).
• Algorithm: The authors employ Group Relative Policy Optimization (GRPO). Unlike PPO, GRPO does not require an auxiliary value function; instead, it uses the average reward of multiple sampled outputs as a dynamic baseline.
• Reward Function: The reward ($R_{plan}$) is calculated at the end of a decomposition episode by comparing the generated plan against a human-annotated "gold" plan. The metric is an F1-score based on semantic similarity (using sentence-transformers) and optimal bipartite matching (Hungarian algorithm) to handle step ordering and paraphrasing.
• Output: A structured plan where sub-questions may reference previous answers (e.g., "What is the capital of #1?").

Phase 2: SFT-Guided Agentic Exploration (Low-Level)

• Objective: To execute the generated plan by performing iterative retrieval and reasoning for each sub-question.
• Mechanism: An Execution Agent is trained via Supervised Fine-Tuning (SFT) on a specialized dataset of multi-turn retrieval instructions.
• Process:
  1. Reference Resolution: Placeholders in sub-questions are resolved using the accumulated knowledge base ($K_{acc}$).
  2. Knowledge Sufficiency Check: The agent decides if the current context is sufficient to answer the sub-question or if further retrieval is needed.
  3. Adaptive Multi-Hop Loop: If retrieval is needed, the agent generates dynamic queries, retrieves documents, filters duplicates, and updates the context. This loop continues until a maximum hop limit ($H_{max}$) is reached or the agent decides to stop.
  4. Synthesis: Once all sub-questions are answered, the agent synthesizes the final answer and a confidence score.

3. Key Contributions

1. Novel Framework: Introduction of APEX-Searcher, which explicitly separates strategic planning (decomposition) from tactical execution (retrieval), addressing the lack of global reasoning in existing iterative RAG.
2. RL for Planning: Application of GRPO with task decomposition rewards to significantly improve the LLM's ability to break down complex problems into logical sub-tasks.
3. SFT for Exploration: Construction of a high-quality, multi-turn retrieval dataset to train the Execution Agent via SFT, optimizing the model's ability to manage context and perform dynamic query generation.
4. Comprehensive Evaluation: Demonstration of state-of-the-art performance across multiple challenging benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle).

4. Experimental Results

The framework was evaluated on Qwen2.5-3B and Qwen2.5-7B models against baselines including Standard RAG, Iterative RAG (IRCoT), and various Agentic RAG methods (Search-o1, Search-R1, etc.).

• Performance Gains:
  • APEX-Searcher achieved the highest Exact Match (EM) scores across all benchmarks.
  • On the 7B model, it achieved an average improvement of 0.230 over Standard RAG and significantly outperformed the strongest baseline (e.g., +8.2% to +13.1% EM on specific datasets).
  • On the 3B model, the improvement was even more dramatic, showing a 149% relative increase over the baseline when all components were enabled.
• Ablation Studies:
  • Synergy: The best performance was achieved only when both Planning (RL) and Exploration (SFT) were combined.
  • Planning Necessity: Using the Planning module without RL optimization yielded limited gains or even degraded performance on smaller models (due to forced decomposition without guidance).
  • RL Impact: Adding RL to the planning module significantly improved plan quality (e.g., 3B model improved from 12.63 to 18.10 Eval score).
  • SFT Impact: SFT was crucial for "execution capability," allowing models to better utilize retrieved context, especially for smaller models.
• Parameter Sensitivity:
  • Retrieval Count: Accuracy improved as retrieved documents increased from 1 to 3, but declined beyond 3 due to noise.
  • Reasoning Hops: Accuracy showed a positive correlation with the maximum allowed reasoning hops (up to 5).

5. Significance and Conclusion

The paper establishes that strategic planning is a critical, often overlooked component in solving complex multi-hop reasoning tasks. By decoupling the "what to search" (Planning) from the "how to search" (Exploration), APEX-Searcher provides a robust scaffold for LLMs.

• Theoretical Insight: It shifts the paradigm from reactive, iterative retrieval to proactive, plan-driven exploration.
• Practical Impact: The framework demonstrates that smaller models (3B) can achieve performance comparable to or exceeding larger baselines when equipped with the correct planning and exploration training strategies.
• Future Work: The authors suggest exploring multi-round RL during the exploration phase and expanding the retrieval scope from local databases to web search.

In summary, APEX-Searcher represents a significant advancement in Agentic RAG, proving that equipping LLMs with explicit, RL-optimized planning capabilities is essential for mastering complex information-seeking tasks.