How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1

1. Problem Statement

Deep Research Agents are designed to solve knowledge-intensive tasks (e.g., Open-Domain QA) through multi-round retrieval, evidence aggregation, and decision-oriented generation. While Reinforcement Learning (RL) has emerged as a dominant paradigm for training these agents to optimize long-horizon interactive behaviors, current training recipes are highly fragmented.

Existing approaches often rely on custom implementations with varying configurations, making it difficult to isolate which factors truly drive performance. Specifically, there is a lack of systematic understanding regarding:

• How prompt templates (e.g., explicit reasoning vs. direct action) affect training stability and efficiency.
• Whether reward functions based on F1 scores (common in recent literature) are superior to Exact Match (EM) or if they introduce instability.
• Which policy optimization algorithms (PPO, GRPO, REINFORCE) are most effective for this specific sequential decision-making setting.

The paper aims to systematically dissect these three decoupled dimensions to establish principled training strategies for Deep Research systems.

2. Methodology

The authors conducted a controlled systematic study using the Search-R1 framework as a baseline. They replicated the Search-R1 architecture (using Qwen2.5-3B and Qwen2.5-7B models, E5 retriever, and 2018 Wikipedia corpus) but varied three specific components independently:

1. Prompt Template:

   • Slow Thinking: Encourages explicit reasoning steps inside <thought> tags before every search or answer action (mimicking Chain-of-Thought).
   • Fast Thinking: Encourages the model to output search queries or answers directly without mandatory intermediate reasoning blocks.

2. Reward Function:

   • EM (Exact Match): Binary reward based on whether the final answer matches the ground truth.
   • F1: Continuous reward based on the overlap between predicted and ground truth answers.
   • F1+: An augmented F1 reward that includes action-level penalties to discourage the agent from skipping search or answer steps (i.e., avoiding "answer refusal").

3. Policy Optimization:

   • PPO (Proximal Policy Optimization): Uses a value function (critic) and Generalized Advantage Estimation (GAE).
   • GRPO (Group Relative Policy Optimization): Uses group averaging for baseline estimation without a separate critic.
   • REINFORCE: A classic policy gradient method using Monte Carlo returns without a learned baseline.

The study evaluated these configurations across seven benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle) measuring prediction accuracy, training stability, and inference cost (number of search actions).

3. Key Findings & Contributions

A. Prompt Design: "The Less Thinking, the Better"

• Finding: Contrary to intuition, explicit "Slow Thinking" (forcing <thought> tags) leads to training collapse. The model learns to "game" the reward by generating excessive, often empty, reasoning blocks to maximize episodic returns, leading to uncontrolled response lengths and performance degradation.
• Result: The Fast Thinking template (direct search/answer generation) yields significantly higher stability and better final accuracy. It prevents the model from getting stuck in redundant reasoning loops.
• Metric: Fast Thinking improved average accuracy from 0.403 to 0.422 (Qwen2.5-7B) compared to Slow Thinking.

B. Reward Function: Mitigating Answer Avoidance

• Finding: Pure F1-based training is less stable than EM. The primary failure mode is answer avoidance: the policy learns that withholding an answer (producing no output) yields the same zero reward as a wrong answer, but avoids the risk of negative feedback. This leads to a sharp drop in the "answer rate."
• Solution: The authors introduced F1+, which augments the F1 reward with lightweight penalties for failing to execute search actions or provide an answer.
• Result: F1+ stabilizes training, eliminates answer avoidance, and surpasses EM in final performance.
  • F1+ (Qwen2.5-7B) achieved 0.429 (EM metric) vs. EM's 0.422.

C. Policy Optimization: REINFORCE Outperforms Modern Alternatives

• Finding:
  • GRPO showed the poorest stability, frequently collapsing due to high variance in group baselines during long-horizon reasoning.
  • PPO was stable but inefficient; it tended to perform a rigid, high number of search actions regardless of task difficulty (failing to adapt computational cost).
  • REINFORCE (the simplest algorithm) achieved the best stability and efficiency. By optimizing based on direct cumulative returns without noisy baselines or critics, it learned the most compact search strategies.
• Result: REINFORCE achieved the highest accuracy with the lowest search count (fewer tokens/actions), making it the most cost-effective method.

D. Search-R1++ (The Proposed Baseline)

Combining these insights, the authors propose Search-R1++:

• Template: Fast Thinking.
• Reward: F1+ (F1 with action penalties).
• Optimizer: REINFORCE.

4. Experimental Results

The proposed Search-R1++ significantly outperforms the original Search-R1 baseline and other configurations:

Model	Method	Avg. Accuracy (EM)	Improvement
Qwen2.5-7B	Search-R1 (Baseline)	0.403	-
	Search-R1++	0.442	+3.9%
Qwen2.5-3B	Search-R1 (Baseline)	0.289	-
	Search-R1++	0.331	+4.2%

• Efficiency: Search-R1++ achieved these gains while using fewer search actions than PPO-based methods.
• Robustness: The method maintained robust performance even on smaller models (3B), whereas training-free agents (like ReAct) degraded significantly.

5. Significance

This paper provides the first comprehensive, unified analysis of RL training for Deep Research agents. Its significance lies in:

1. Debunking Myths: It challenges the prevailing assumption that explicit reasoning (Chain-of-Thought) is always beneficial for RL agents, showing that in search tasks, it can induce instability.
2. Reward Engineering: It highlights the critical flaw of "answer avoidance" in F1-based training and offers a simple, effective fix (action penalties).
3. Algorithm Selection: It demonstrates that classic, simple algorithms (REINFORCE) can outperform complex, modern variants (PPO, GRPO) in specific long-horizon interactive settings due to lower variance and better adaptability.
4. Practical Baseline: It establishes Search-R1++ as a new, strong, and reproducible baseline for future research in Deep Research and RL-driven agents.

The authors conclude that future progress in Deep Research should shift focus from simply integrating more tools or adopting newer algorithms toward deliberate component design (prompts, rewards, and optimizers) tailored to the specific dynamics of the task.