Technical Summary: Self-Distilled Agentic Reinforcement Learning (SDAR)

Problem Statement

Reinforcement Learning (RL) has become a central paradigm for post-training Large Language Model (LLM) agents, particularly for long-horizon, multi-turn interactions. However, standard RL relies on trajectory-level reward signals, which provide only coarse supervision for complex, multi-step tasks. To address this, On-Policy Self-Distillation (OPSD) has been proposed to introduce dense, token-level guidance from a "teacher" branch augmented with privileged context (e.g., retrieved skills).

Despite its theoretical appeal, transferring OPSD to multi-turn agents presents two critical challenges identified in this work:

1. Multi-turn Instability: As the student agent drifts from the teacher-supported trajectory, token-level supervision becomes increasingly unreliable. This leads to compounding errors, surging per-turn KL divergence, and catastrophic degradation in task performance.
2. Asymmetric Trust in Privileged Guidance: In skill-conditioned OPSD, the teacher is the same policy augmented with privileged context (like retrieved skills), not an inherently stronger model. Consequently, a "negative gap" (where the teacher assigns lower probability to a token than the student) does not always imply the token should be suppressed. It may instead result from imperfect skill retrieval, irrelevant skills, or the teacher's failure to ground skills into reliable preferences. Treating all teacher signals equally leads to instability, especially when negative-gap tokens constitute a significant portion of the training data (observed as >50% in preliminary studies).

Existing hybrid approaches (e.g., simply adding OPSD loss to RL) or rigid curriculum methods fail to resolve these issues, often suffering from unbounded distillation gradients that overwhelm the RL signal or relying on hand-crafted schedules that do not adapt to token-level dynamics.

Methodology: SDAR

The authors propose Self-Distilled Agentic Reinforcement Learning (SDAR), a framework that treats OPSD as a carefully controlled auxiliary objective while maintaining RL as the primary optimization backbone. The core innovation is a token-level adaptive gating mechanism that modulates the distillation signal based on the reliability of the teacher's guidance.

1. Optimization Framework

The total training objective combines the standard Group Relative Policy Optimization (GRPO) loss with a gated self-distillation loss:
$$\mathcal{L}(\theta) = \mathcal{L}_{GRPO}(\theta) + \lambda_{SDAR} \cdot \mathcal{L}_{SDAR}(\theta)$$

• $\mathcal{L}_{GRPO}$: The primary RL loss, driven by environment rewards and verifier feedback, remains untouched to preserve unbiased policy optimization.
• $\mathcal{L}_{SDAR}$: The auxiliary distillation loss, which is selectively applied.

2. Token-Level Gating Mechanism

Instead of applying distillation uniformly, SDAR computes a gate $g_t \in [0, 1]$ for each token $t$ to control the intensity of the teacher's influence.

• Teacher-Student Gap ($\Delta_t$): Defined as the log-probability difference between the privileged teacher and the student: $\Delta_t = \log \pi_T(y_t | s^+_t) - \log \pi_\theta(y_t | s_t)$.
• Sigmoid Gating: The gap is passed through a sigmoid function with a sharpness parameter $\beta$:
  $$g_t = \sigma(\beta \cdot \Delta_t)$$
  This creates a smooth, differentiable weight that:
  • Amplifies distillation for positive-gap tokens (where the teacher endorses the student's choice), treating them as high-confidence guidance.
  • Attenuates distillation for negative-gap tokens (where the teacher disagrees), preventing the model from blindly following potentially noisy or irrelevant skill-based rejections.
• Stop-Gradient: The gate $g_t$ is detached from the gradient computation. This ensures the gate acts purely as a confidence weight, preventing self-referential optimization loops and ensuring stable, first-order updates.

3. Skill Retrieval and Context

SDAR utilizes a "Self-Teacher" branch that conditions on privileged context $c^+$ (e.g., retrieved skills from a SkillBank). The framework evaluates robustness across four retrieval strategies: UCB (Upper Confidence Bound), Keyword Matching, Full Retrieval, and Random Retrieval. The gating mechanism is designed to filter out noise from low-quality or irrelevant retrieved skills.

Key Contributions

1. Identification of Asymmetric Trust: The paper establishes that in multi-turn agent training, privileged guidance is inherently asymmetric. Positive endorsements from a teacher with privileged context are reliable, but negative rejections may stem from context instability rather than token quality.
2. Adaptive Token-Level Gating: SDAR introduces a novel gating strategy that allows each token to autonomously regulate its distillation intensity. This replaces rigid, hand-crafted schedules with a dynamic, self-paced curriculum operating at the finest granularity.
3. Stability and Robustness: By decoupling the distillation signal from the primary RL objective and applying soft attenuation to negative signals, SDAR avoids the catastrophic instability observed in naive GRPO+OPSD combinations.
4. Internalization of Knowledge: Unlike methods that rely on external skills at inference time, SDAR successfully internalizes privileged knowledge into the policy parameters, enabling strong performance even without retrieved skills during testing.

Experimental Results

The authors evaluated SDAR on three diverse benchmarks: ALFWorld (embodied AI), Search-QA (search-augmented reasoning), and WebShop (e-commerce interaction). Experiments were conducted across the Qwen2.5 (3B, 7B) and Qwen3 (1.7B) model families.

• Performance Gains: SDAR substantially outperforms the baseline GRPO.
  • ALFWorld: +9.4% improvement on Qwen2.5-3B.
  • Search-QA: +7.0% improvement.
  • WebShop: +10.2% improvement in Accuracy (Qwen2.5-7B).
• Comparison to Hybrids: SDAR consistently outperforms hybrid baselines like Skill-SD and RLSD across all model scales. Notably, it avoids the performance collapse seen in naive GRPO+OPSD (which degrades significantly on smaller models like Qwen3-1.7B).
• Robustness to Retrieval Quality: A critical finding is that SDAR degrades gracefully with retrieval quality. Even with Random Retrieval (zero task awareness), SDAR outperforms the pure GRPO baseline. This confirms that the gating mechanism effectively filters out noise from low-quality skills, extracting only beneficial signals.
• Inference Independence: SDAR achieves state-of-the-art results without requiring skill retrieval at inference time, demonstrating true knowledge internalization. For instance, on ALFWorld with Qwen2.5-3B, SDAR (84.4%) outperforms Skill-GRPO* (80.5%), which relies on skills at test time.

Significance and Claims

The paper claims that SDAR offers a principled solution to the instability of applying self-distillation in multi-turn agentic RL. By treating distillation as a gated auxiliary objective rather than a primary driver or a rigid constraint, SDAR reconciles the coarse, environment-driven optimization of RL with the dense, token-level guidance of privileged context.

The authors emphasize that their approach does not require a separate, stronger teacher model; instead, it leverages the same policy augmented with privileged context, making it scalable and practical. The gating mechanism's ability to handle asymmetric guidance—trusting positive signals while conservatively attenuating negative ones—is presented as the key to stabilizing multi-turn training and enabling robust generalization across different model sizes and task domains. The work suggests that for multi-turn agents, RL should remain the primary backbone, with distillation serving as a dynamic, token-level regulator rather than a static supervisor.