SynPlanResearch-R1: Encouraging Tool Exploration for Deep Research with Synthetic Plans

The paper introduces SynPlanResearch-R1, a framework that synthesizes tool-use trajectories to encourage deeper exploration during supervised fine-tuning, thereby overcoming the limitations of reinforcement learning with verifiable rewards and significantly improving research agent performance across multiple benchmarks.

The Gist

Here is an explanation of the paper SynPlanResearch-R1 using simple language and creative analogies.

The Problem: The "Impatient Detective"

Imagine you hire a brilliant but inexperienced detective (an AI agent) to solve a complex mystery, like "Who stole the crown jewels?"

You give this detective a toolbox: they can search the internet (ask the police) or visit a crime scene (read a specific webpage).

However, when you let this detective loose, they have two bad habits:

1. They give up too soon: They ask one question, get a vague answer, and immediately shout, "I think it was the butler!" before gathering enough evidence.
2. They only use one tool: They love asking the police (searching) but are terrified of visiting the crime scene (reading webpages), even when the answer is hidden in a specific document.

The researchers found that if you just tell the detective, "Try harder to get the right answer," they don't get better. They just keep making the same mistakes because they are stuck in a rut. This is what happens when AI tries to learn complex research tasks using standard reinforcement learning alone.

The Solution: The "Scripted Rehearsal"

The authors propose a new method called SynPlanResearch-R1. Instead of just telling the detective to "go solve it," they create a rehearsal script before the real performance.

Here is how it works, step-by-step:

1. The "Randomized Map" (Synthetic Plans)

Imagine you are training a hiker to find a hidden treasure. Instead of letting them wander aimlessly, you give them a map with a random route drawn on it.

• The map says: "First, ask the village elder (Search). Then, visit the old library (Crawl). Then, ask the village elder again."
• The AI (the hiker) is forced to follow this map. It can't just skip to the end. It has to practice the full journey, even if the map looks a bit weird.

2. The "Nudge" (Injected Cues)

Sometimes, the AI looks at the map and thinks, "I know how to do this, I'll just do my own thing."
To stop this, the researchers add gentle whispers (cues) at every step.

• Instead of just thinking: "I need info."
• The AI thinks: "I need info... and the map says I should go to the library next."
  This keeps the AI on the path of deep exploration without forcing it to be a robot.

3. The "Editor" (Thought Rewriting)

Because the AI is following a script and listening to whispers, its internal monologue sounds a bit robotic and unnatural.

• Before: "Okay, the map says library. I am going to the library. The map is right."
• After Editing: "The search results were vague, so I need to dig deeper into the specific documents."
  The researchers use a super-smart AI editor to rewrite these thoughts, making them sound like a natural, smart human detective while keeping the actions (the map steps) exactly the same.

4. The "Real Exam" (Reinforcement Learning)

Now that the detective has practiced with these high-quality, deep-exploration scripts, they are ready for the real test.

• They are put in a real environment where they get points only if they find the correct answer.
• Because they started with a strong "habit" of exploring deeply (thanks to the rehearsal), they don't get stuck in the "give up early" rut. They naturally keep digging until they find the truth.

Why This Matters

Think of it like training a marathon runner.

• Old Way: You tell the runner, "Run until you win." They might sprint the first mile, get tired, and quit.
• New Way (SynPlanResearch-R1): You first make them run a specific, long training route with a coach guiding them at every mile marker. They learn how to pace themselves and how to push through the hard parts.
• Result: When they finally race for real, they have the stamina and the strategy to win, whereas the others quit early.

The Results

The paper tested this on seven different "mystery" challenges (like complex trivia and open-web research).

• The new method made the AI significantly smarter.
• It stopped the AI from giving up too soon.
• It forced the AI to use all its tools (searching and reading) effectively.

In short: The paper teaches AI how to be a better researcher by giving it a "training camp" where it practices deep, thorough investigation before it ever tries to solve a real problem on its own.

Technical Summary

Here is a detailed technical summary of the paper "SynPlanResearch-R1: Encouraging Tool Exploration for Deep Research with Synthetic Plans".

1. Problem Definition

Research Agents are Large Language Models (LLMs) designed to autonomously gather information from the web using tools (e.g., search, crawling) to answer complex, multi-hop queries. While Reinforcement Learning with Verifiable Rewards (RLVR) has shown promise in training these agents, the authors identify a critical bottleneck: poor exploration behavior.

Specifically, agents trained with standard RLVR often exhibit:

• Premature Termination: The agent stops reasoning and outputs an answer after too few tool calls, failing to gather sufficient evidence.
• Biased Tool Usage: The agent relies heavily on familiar tools (e.g., web search) and neglects others (e.g., crawling specific webpages), leading to shallow evidence gathering.
• The Initialization Trap: Because RLVR is an on-policy method, it relies on the agent's own rollouts to learn. If the initial policy (cold-start) is weak or biased, the agent gets trapped in suboptimal local optima and cannot discover deeper, more effective exploration strategies.

2. Methodology: SynPlanResearch-R1

The authors propose SynPlanResearch-R1, a two-stage framework designed to shape the agent's exploration behavior during the cold-start phase before RL training begins. The core innovation is a plan-guided synthetic data synthesis pipeline.

A. Plan-Guided Data Synthesis (Cold-Start SFT)

Instead of using standard expert trajectories, the authors generate synthetic training data that forces the model to explore deeper:

1. Tool-Plan Construction: A generator creates randomized tool-use plans (sequences of actions like web search $\to$ crawl webpage $\to$ web search). The length of these plans is sampled from a range (e.g., 3 to 8 steps).
2. Cue-Injected Thoughts: To ensure the Large Reasoning Model (LRM) follows these plans without breaking the natural ReAct (Reasoning + Acting) flow, the authors inject soft cues at the beginning of each thought step.
   • Example: If the plan requires a crawl, the cue might be: "There are several promising links from the search results. Perhaps I should inspect one of them..."
   • These cues act as soft constraints, guiding the LRM toward the intended action while preserving natural language generation.
3. Trajectory Generation & Filtering: The LRM generates full ReAct trajectories based on these plans. Only trajectories that pass two checks are retained:
   • Format Validity: Correct ReAct structure (Thought, Action, Observation, Answer).
   • Answer Correctness: The final answer matches the ground truth.
4. Thought Rewriting: Since injected cues can sound unnatural, a high-quality rewriting model (e.g., Claude) paraphrases the thoughts to be fluent and concise while retaining the directive intent.

B. Reinforcement Learning (RL) with Stabilization

The model is first fine-tuned (SFT) on the synthesized dataset to create a strong initialization policy ($\pi_{sft}$). This policy is then optimized using GRPO (Group Relative Policy Optimization).

• Reward Shaping: Rewards are based on answer accuracy (F1 score) and format validity.
• Stabilization Tricks:
  • Masking Void Trajectories: Rollouts that exceed token/turn limits are excluded from the policy loss (to prevent gradient noise) but included in the advantage calculation (to maintain group statistics).
  • Immediate Termination on Schema Errors: If a tool call violates the JSON schema, generation stops immediately to prevent the model from learning to recover from malformed outputs, which can destabilize training for smaller models.

3. Key Contributions

1. SynPlanResearch-R1 Framework: A novel data synthesis approach that uses randomized tool plans and soft cues to generate diverse, deep exploration trajectories for cold-start SFT.
2. Addressing the Exploration Bottleneck: The paper demonstrates that the primary limitation of RLVR in research agents is not the RL algorithm itself, but the lack of a strong exploration prior in the initial SFT phase.
3. Stabilization Strategies: Practical techniques (masking void turns, strict JSON error handling) that enable stable training of multi-turn tool-using agents, particularly on smaller model scales (4B/8B parameters).
4. Comprehensive Analysis: Detailed ablation studies and training dynamics analysis showing how the method improves policy entropy and tool diversity.

4. Experimental Results

The method was evaluated on seven benchmarks spanning multi-hop QA (HotpotQA, 2WikiMultihopQA, MuSiQue, Bamboogle) and advanced open-web research (GPQA, WebWalkerQA, GAIA).

• Performance Gains:
  • Qwen3-8B: Improved performance by up to 6.0% over SOTA baselines.
  • Qwen3-4B: Improved performance by up to 5.8% over SOTA baselines.
  • Specifically, on Multi-Hop QA, it achieved a 5.1% gain, and on Advanced QA, an 8.7% gain (for 8B models).
• Comparison: It significantly outperformed baselines like Search-R1, SimpleDeepSearcher, and standard Rejection Sampling.
• Tool Usage Analysis:
  • Agents trained with SynPlanResearch-R1 made more tool calls on average compared to baselines.
  • There was a strong positive correlation between the number of tool calls and task accuracy.
  • The agents utilized web crawling much more frequently (a critical capability for deep research) compared to baselines that relied almost exclusively on search.
• Training Dynamics:
  • Higher Entropy: The proposed method maintained higher policy entropy throughout training, indicating a more diverse exploration strategy that avoids premature convergence.
  • Long-term Reward: While initial rewards were slightly lower due to increased exploration, the method eventually surpassed all baselines in final reward, proving that the initial exploration cost leads to superior long-term performance.

5. Significance

This paper fundamentally shifts the focus of training Research Agents from "better RL algorithms" to "better initialization via synthetic data."

• Paradigm Shift: It proves that simply applying RL to a weakly initialized agent yields diminishing returns due to exploration bias. By engineering the cold-start data to explicitly encourage diverse and deep tool usage, the ceiling for subsequent RL optimization is significantly raised.
• Scalability: The approach is effective across different model sizes (4B to 8B), making deep research capabilities accessible to smaller, more efficient models.
• Practical Impact: The stabilization techniques (handling void turns and JSON errors) provide a robust recipe for training complex, multi-turn agents in real-world environments where tool failures are common.

In conclusion, SynPlanResearch-R1 demonstrates that controlling the exploration prior during the supervised fine-tuning phase is the most critical lever for training robust, deep-researching AI agents.