Technical Summary: DecoupleSearch

Problem Statement

Retrieval-Augmented Generation (RAG) systems have become essential for grounding Large Language Models (LLMs) in external knowledge to mitigate hallucinations. However, standard RAG often relies on static workflows that struggle with complex, multi-step reasoning tasks. Agentic RAG addresses this by introducing autonomous agents that iteratively plan and search. Despite its potential, Agentic RAG faces three critical challenges:

1. Interdependence of Planning and Search: The success of the reasoning process relies on both high-quality planning (determining what information is needed) and accurate search (retrieving that information). Errors in either stage propagate, and optimizing one without the other is difficult.
2. Lack of Intermediate Supervision: Most RAG datasets provide only final answers, offering no explicit feedback on the quality of intermediate reasoning steps (e.g., the validity of a generated plan or the relevance of a retrieved document).
3. Exponential Search Space: The candidate space for planning and searching grows exponentially, making it computationally expensive to identify optimal reasoning paths through brute-force exploration.

Methodology: DecoupleSearch

The authors propose DecoupleSearch, a framework that decouples the planning and search processes using dual value models (a planning value model and a search value model) to enable independent optimization. The methodology consists of three main phases:

1. MCTS Annotation (Training Data Construction)

To address the lack of intermediate supervision, the authors employ Monte Carlo Tree Search (MCTS) to construct reasoning trees for training queries.

• Process: The LLM acts as an agent to generate plans and search queries, interacting with a retriever to expand the tree.
• Evaluation: During simulation, the LLM acts as a judge to assign reward scores ($-1$ to $1$) to both planning and search steps. Terminal nodes receive rewards based on the correctness of the final answer.
• Backpropagation: Rewards are back-propagated through the tree to update node values and visit counts, creating a dataset of high-quality (correct) and low-quality (incorrect) reasoning paths.

2. Model Training

The framework utilizes a shared backbone policy model ($\pi_\theta$) initialized from a pre-trained LLM.

• Architecture: Two auxiliary linear layers with Tanh activation are added to the backbone to derive the Planning Value Model ($V_\phi$) and Search Value Model ($V_\psi$). This design ensures parameter efficiency and joint optimization.
• Loss Function: A multi-task loss is applied:
  • Policy Loss: Negative log-likelihood on correct solution paths to guide token prediction.
  • Value Loss: Mean Squared Error (MSE) between the predicted values and the MCTS-derived rewards for both planning and search steps across all paths.

3. Inference: Hierarchical Beam Search

During inference, the system employs a Hierarchical Beam Search algorithm to navigate the exponential search space efficiently.

• Decoupled Exploration: At each step, the policy model generates multiple candidate plans. The Planning Value Model evaluates and prunes these, retaining only the most promising ones.
• Search Grounding: Based on the selected plans, the policy generates multiple search queries. The Search Value Model evaluates the retrieved documents, pruning low-value results.
• Iteration: This iterative refinement continues until a maximum depth is reached or no nodes can be expanded. The final answer is selected based on the highest value assigned by the planning model.

Key Contributions

1. Decoupled Framework: Introduction of DecoupleSearch, which separates planning and search optimization via dual value models, allowing for independent refinement of reasoning logic and retrieval grounding.
2. MCTS-Driven Supervision: A novel approach to generating intermediate supervision signals for Agentic RAG by using MCTS to annotate reasoning trees with step-level rewards, overcoming the limitation of final-answer-only datasets.
3. Efficient Search Strategy: The proposal of Hierarchical Beam Search, which leverages the dual value models to prune the exponential candidate space, balancing exploration and computational cost.
4. Empirical Validation: Extensive experiments demonstrating the method's effectiveness across varying policy model sizes (7B and 14B parameters) and diverse QA datasets.

Experimental Results

The method was evaluated on five datasets: HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle (multi-hop), and TriviaQA (single-hop). The policy models used were Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct.

• Performance Gains: DecoupleSearch achieved state-of-the-art results across all datasets. With the 7B model, it showed a 25.8% relative average improvement over the best-performing baseline.
• Comparison: It outperformed vanilla prompting methods (Direct, CoT), advanced RAG methods (Iterative, Judge-then-retrieve), and other Agentic RAG baselines (RAgent, Search-o1, GreedyAgent).
• Scaling: While larger models (14B) generally performed better, Hierarchical Beam Search allowed the 7B model to achieve performance comparable to the 14B model, highlighting the efficacy of inference-time scaling.
• Ablation Studies:
  • Removing either planning or search expansion degraded performance, with planning expansion being more critical.
  • The value models proved essential; replacing value-based ranking with random selection resulted in significant performance drops, particularly in search expansion.
  • Optimal performance was observed with a planning expansion size of approximately 3, while larger search expansion sizes generally yielded better results.

Significance and Claims

The paper claims that DecoupleSearch addresses the fundamental limitations of current Agentic RAG systems by treating planning and search as distinct but coupled optimization problems. By decoupling these processes and utilizing MCTS to generate high-quality intermediate supervision, the framework enables LLMs to navigate complex reasoning tasks more effectively.

The authors emphasize that their approach allows smaller models to compete with larger ones through structured inference strategies (Hierarchical Beam Search) rather than relying solely on parameter scaling. The work provides a robust methodology for improving the reliability and accuracy of autonomous agents in knowledge-intensive tasks, specifically by mitigating the "hallucination" problem through rigorous search grounding and by solving the "planning" problem through value-guided exploration.

The authors acknowledge limitations, noting the computational cost of MCTS annotation and the current focus on retaining a single best path per step, leaving the exploration of multi-path retention for future work.