Technical Summary: SearchMaster

Problem Statement

Training Large Language Model (LLM) based search agents requires high-quality data featuring genuine multi-hop retrieval tasks and trajectories that effectively utilize search tools. Existing pipelines typically rely on expensive human annotation, expert demonstrations, or stronger teacher models to generate this data. While self-play offers a potential solution for generating data autonomously, current search-agent self-play methods suffer from distinct failure modes:

1. Pseudo Multi-Hop Questions: Models may generate questions that appear complex but are answerable from a single document or the seed text alone, failing to require cross-document evidence.
2. Misleading Difficulty Signals: Relying solely on rollout success rates to estimate task difficulty often fails to distinguish between shallow lookup tasks and those requiring deep, multi-step search.
3. Over-Opening Drift: Models may over-rely on the open tool, revisiting already-seen documents rather than acquiring new, targeted evidence, leading to long but shallow browsing trajectories.

The core challenge is not merely generating more data, but stabilizing the self-play loop to ensure training signals reinforce evidence-grounded tasks, multi-step search, and efficient tool use.

Methodology: SearchMaster Framework

SearchMaster is a grounded and regulated self-play framework that trains a single LLM policy ($\pi_\theta$) acting as both a Proposer (task generator) and a Solver (task responder) within a local search environment. A frozen copy of the initial model serves as a Verifier. The framework employs three specific mechanisms to address the failure modes of standard self-play:

1. Evidence-Chain Generator (ECG)

To prevent pseudo multi-hop questions, the Proposer is required to explicitly construct a cross-document evidence chain during exploration.

• Process: Starting from a seed document, the Proposer iteratively selects entities/facts, searches for connected documents, and extends the chain ($c_j = e_{j,1} \to \dots \to e_{j,n}$).
• Constraint: The final question ($q_j$) is derived strictly from this full chain, with explicit requirements that the answer cannot be found in a single document.
• Filtering: Candidates undergo four filters (tool-use, format, validity, and parametric-knowledge) to ensure the question is valid, supported by the chain, and cannot be answered without tools.

2. Search-Depth Reward (SDR)

To address the limitation of success-rate-based difficulty estimation, SearchMaster rewards tasks based on the search depth of successful rollouts rather than just the binary success outcome.

• Metric: Task difficulty is scored by the minimum search depth ($h_{min}$) among all successful Solver rollouts. If a task can be solved with a shallow search, it receives a lower reward, preventing the model from learning "shortcut" tasks.
• Function: The reward saturates at a target depth ($H_{sdr}$), ensuring tasks requiring non-trivial search depth are favored without rewarding arbitrarily long searches.

3. Over-Opening Penalty (OOP)

To regulate tool usage and prevent redundant document opening, an OOP is applied to both Proposer and Solver rewards.

• Mechanism: The penalty is calculated based on the ratio of open actions to search actions ($\rho_{oop}$). It penalizes trajectories where the number of opens significantly exceeds the number of searches, discouraging the revisiting of already-seen documents while allowing legitimate multi-document exploration.
• Application: The penalty is subtracted from the base reward for both roles, as they share the same policy.

Optimization

The framework uses Group Relative Policy Optimization (GRPO).

• Data Flow: For each seed document, the Proposer generates $K_P$ candidate tasks. Passing tasks are solved by the Solver ($K_S$ rollouts).
• Selection: Only the Solver rollouts for the highest-reward passing task are retained for the Solver group; all Proposer rollouts are retained for the Proposer group.
• Update: The shared policy is updated jointly using token-level advantages derived from the normalized rewards, which include the SDR and OOP components.

Key Contributions

1. Formulation of Grounded Self-Play: The paper formulates search-agent self-play as a problem of grounding and regulating self-generated supervision, turning a potentially noisy loop into a reliable training mechanism.
2. Three-Part Control Mechanism: The introduction of ECG, SDR, and OOP explicitly addresses the specific failure modes of search self-play (pseudo multi-hop, shallow difficulty estimation, and tool misuse).
3. Human-Free Training: The framework demonstrates that effective search-agent training data can be generated and utilized without human-labeled QA pairs or expert demonstrations, relying solely on a local search environment.

Experimental Results

The authors evaluated SearchMaster using a Qwen3.5-9B backbone across six deep-search benchmarks.

• Performance Gain: The framework improved the average accuracy from 38.19% to 51.52% (+13.3 points).
• Specific Benchmark: On the BrowseComp-Plus benchmark, the model achieved a 30.1-point gain, rising from 30.12% to 60.24%.
• Data Efficiency: These results were achieved using only 1,280 qualified seed documents generated over 20 iterations, without external human data.

Significance and Claims

The paper claims that SearchMaster provides a scalable solution for training search agents by stabilizing the self-play loop. By making evidence construction, task difficulty (via search depth), and tool-use balance explicit parts of the training objective, the framework enables the "self-evolution" of search agents. The results suggest that grounded and regulated self-play can produce high-quality training data that rivals methods dependent on human annotation or stronger teacher models, significantly improving the model's ability to perform genuine multi-hop retrieval and efficient tool use.