Technical Summary: Step-Level Self-Distilled Policy Optimization for Deep Search Agents

Problem Statement

Deep search agents operate over long-horizon trajectories involving dozens of steps (e.g., issuing searches, browsing pages, and refining reasoning). Standard Reinforcement Learning (RL) approaches for these agents, such as Group Relative Policy Optimization (GRPO), typically provide only a single binary outcome reward per trajectory. This sparse supervision creates a severe credit assignment bottleneck: the model receives no guidance on which specific intermediate steps led to success or failure.

While On-Policy Self-Distillation (OPSD) has been successful in single-turn reasoning tasks (like math and code) by using the model's own logits as dense token-level teachers, extending this to multi-turn search agents introduces fundamental challenges:

1. Extreme Information Asymmetry: In search tasks, a "teacher" model with access to privileged information (e.g., curated evidence and the correct answer) collapses the search process significantly (e.g., 3.5 tool calls vs. 17.7 for the student). Simple distillation causes the student to mimic this brevity, learning to abandon tool use prematurely rather than developing robust search strategies.
2. Lack of Natural Privileged Information: Unlike math or code, open-ended web search lacks ready-made reference solutions or execution feedback that can serve as a teacher prefix without revealing the entire answer path.

Methodology: SSPO

The authors propose Step-Level Self-Distilled Policy Optimization (SSPO), a framework designed to resolve the tension between the need for dense supervision and the risks of information asymmetry. The method consists of two core components:

1. Evidence Anchors (Privileged Information)

To provide the teacher with necessary context without revealing the full solution path, the authors construct Evidence Anchors. These are concise, step-level evidence snippets extracted from the web that capture key reasoning conditions required to answer a question.

• Construction: A state-of-the-art LLM identifies evidence supporting the ground-truth answer for each QA pair.
• Usage: These anchors serve as a prefix for the teacher model only when the student generates an incorrect trajectory. The teacher is also provided with the student's incorrect answer to prevent it from repeating the same mistakes.

2. Step-Level Advantage Weighting

Instead of minimizing the divergence between the teacher and student distributions (which leads to the "shortcut" problem), SSPO converts the teacher-student disagreement into step-level advantage weights within the GRPO objective.

• Mechanism: For each step $\tau$ in a trajectory, the method computes the log joint-likelihood ratio ($\Delta^{step}_{\tau}$) between the teacher (with privileged context) and the student.
• Application: This signal is applied exclusively to incorrect trajectories.
  • Decoupling Direction and Magnitude: The outcome reward determines the direction of the policy update (reinforce or suppress the trajectory), while the teacher signal modulates the magnitude of the update at each step.
  • Weighting Logic:
    • If the student is overconfident on a step rejected by the teacher ($\Delta^{step}_{\tau} < 0$), the penalty is amplified.
    • If the student is uncertain on a step endorsed by the teacher ($\Delta^{step}_{\tau} > 0$), the penalty is reduced, preserving valid intermediate reasoning.
    • Correct trajectories are left untouched to preserve behavioral diversity.
• Granularity: The supervision is aligned with the natural unit of search actions (Thought + Tool Call) rather than individual tokens, ensuring the signal matches the information-seeking process.

Key Contributions

1. Identification of Assumption Breakdown: The paper demonstrates that extending self-distillation to multi-turn search breaks the assumptions valid for single-turn tasks (modest teacher-student gaps and existing privileged information), necessitating a fundamentally different design.
2. Evidence Anchors: Introduction of a novel form of privileged information—compact, step-level evidence snippets—that aligns with the structure of search actions without revealing the full answer path.
3. SSPO Algorithm: A policy optimization method that uses self-distillation signals as advantage weights rather than optimization targets. This approach injects fine-grained process supervision while avoiding privileged-information leakage and the suppression of diversity in correct trajectories.

Experimental Results

The method was evaluated on Qwen3-8B across three information-seeking benchmarks: BrowseComp, GAIA, and FRAMES.

• Performance: SSPO consistently outperforms standard GRPO. On average, SSPO achieves a +4.8 improvement over the cold-start baseline, compared to GRPO's +2.4.
• Sample Efficiency: SSPO trained for 100 steps surpasses GRPO trained for 200 steps, demonstrating superior sample efficiency.
• Efficiency: The method adds only approximately 5% computational overhead per step (due to a single additional forward pass for the teacher).
• Training Dynamics: Analysis shows SSPO maintains higher rewards and more stable gradient norms throughout training compared to GRPO, which exhibits early saturation. Entropy curves remain aligned, suggesting gains come from more effective policy updates rather than increased exploration.
• Ablation Studies:
  • Direct distillation (minimizing divergence) collapses tool use and underperforms GRPO.
  • Token-level weighting is insufficient; step-level weighting is required to align with information-seeking actions.
  • Evidence Anchors are the primary driver of process supervision, while incorrect-answer feedback serves a complementary role in penalizing terminal failures.

Significance and Claims

The paper claims that SSPO successfully resolves the credit assignment problem in deep search agents by aligning privileged supervision with the natural structure of search steps. By decoupling the update direction (determined by outcome rewards) from the update magnitude (modulated by step-level self-distillation), the method guides agents toward more targeted and evidence-grounded information-seeking.

The authors emphasize that this approach allows agents to learn better search strategies without inheriting the information asymmetry of the teacher, achieving superior performance with minimal computational overhead. The results suggest that fine-grained process supervision, when correctly structured, is critical for scaling deep search agents beyond the limitations of sparse outcome rewards.