Technical Summary: PCSD – Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Problem Statement

Large Language Model (LLM) agents face significant challenges in complex, multi-turn interactive tasks due to reward sparsity. In environments like ALFWorld or WebShop, a trajectory may span dozens of turns and hundreds of tokens, yet receive only a single scalar outcome signal (success or failure) at the end. This delayed feedback makes credit assignment—determining which specific intermediate actions or tokens contributed to the final outcome—extremely difficult.

While On-Policy Self-Distillation (OPSD) offers a solution by using a privileged teacher (a frozen, skill-augmented copy of the student) to provide dense token-level supervision, it introduces a new reliability problem. The teacher's guidance is not uniformly trustworthy; factors like imperfect retrieval, noisy context, or task ambiguity can cause the teacher to favor suboptimal tokens at specific positions. Existing credibility-aware distillation methods struggle to balance two extremes:

1. Token-level methods: Rely on isolated discrepancies at a single position, making them sensitive to sampling noise and transient fluctuations.
2. Step-level methods: Aggregate signals over an entire reasoning step, assigning a shared weight to all tokens. This improves robustness but obscures critical positional variations in teacher reliability.

Methodology: PCSD

The authors propose Persistent Consistency Self-Distillation (PCSD), a framework that estimates teacher credibility based on the local persistence of teacher-favoring signals. The core insight is that informative teacher advantages should persist across a local neighborhood, whereas isolated spikes likely reflect noise. PCSD operates through three complementary mechanisms to generate continuous, token-level distillation weights ($w_{k,i}$):

1. Persistent Consistency Assessment

Instead of relying on a single pointwise log-probability gap ($\delta_{k,i}$) between the teacher and student, PCSD aggregates these gaps over a forward-looking window using exponential decay. This assigns higher importance to positions closer to the current token while incorporating evidence from the immediate local region, creating a "persistent-consistency estimate" ($\bar{\delta}^{(N)}_{k,i}$).

2. Adaptive Aggregation Mechanism

Recognizing that gap patterns vary across a response, PCSD employs a soft adaptive windowing strategy. It computes the local variance of the gap sequence within a maximum analysis window ($N_{max}$).

• Low Variance: If the local gap is stable, the mechanism favors a short-window estimate to preserve fine-grained positional resolution.
• High Variance: If the local gap is noisy, it shifts weight toward a long-window estimate to provide stronger smoothing.
  This results in an adaptive estimate ($\bar{\delta}^{adaptive}_{k,i}$) that dynamically balances precision and robustness without discrete window switching.

3. Trend Modulation and Continuous Gating

To prevent transient signals from dominating, PCSD applies one-sided trend modulation. It calculates the local slope of the gap sequence; if the teacher's support is declining (negative slope), the weight is attenuated. This is combined with a sigmoid gating function that maps the adaptive estimate to a bounded weight ($w_{k,i}$).

• Trend Factor ($\eta_{k,i}$): Reduces reliance on signals where support is decreasing.
• Sigmoid Gating: Transforms the credibility estimate into a smooth, continuous weight.

Training Objective

The final objective combines the standard GRPO (Group Relative Policy Optimization) loss, which uses sparse trajectory-level rewards, with the PCSD distillation loss:
$$\mathcal{L}_{total} = \mathcal{L}_{GRPO} + \lambda_{PCSD} \mathcal{L}_{PCSD}$$
where $\mathcal{L}_{PCSD}$ is a weighted negative log-likelihood that encourages the student to align with the teacher only at positions where persistent, stable support is detected.

Key Contributions

1. PCSD Framework: A token-level on-policy self-distillation method that estimates teacher credibility from the local persistence of signals, effectively combining fine-grained discrimination with noise robustness.
2. Adaptive Aggregation: A mechanism that adjusts its effective window size based on local gap statistics and uses exponential decay to preserve positional locality.
3. Trend-Aware Modulation: The introduction of one-sided trend modulation and continuous sigmoid gating to filter out declining or unstable teacher support.
4. Empirical Validation: Extensive experiments demonstrating that PCSD outperforms outcome-only GRPO and existing weighting schemes (like RLSD and SDAR) across different model scales and tasks.

Experimental Results

The authors evaluated PCSD on ALFWorld (text-based embodied AI) and WebShop (e-commerce navigation) using Qwen2.5-3B-Instruct and Qwen3-1.7B-Instruct backbones.

• ALFWorld Performance: PCSD achieved the highest Overall success rates among all baselines.
  • With Qwen2.5-3B: 90.6% (exceeding GRPO by 15.6 points and SDAR by 6.2 points).
  • With Qwen3-1.7B: 59.4% (exceeding GRPO by 13.3 points and SDAR by 5.5 points).
  • Notable improvements were observed in subtasks like Heat, Cool, and Pick2.
• WebShop Performance: PCSD achieved the highest Score (85.0) with Qwen2.5-3B and tied for the highest Accuracy (67.2%) with SDAR on both backbones.
• Generalization: On an unseen split of ALFWorld, PCSD maintained robust performance (86.7% success rate), significantly outperforming GRPO (70.9%) and SDAR (72.7%).
• Ablation Studies: Removing adaptive aggregation, trend modulation, or exponential decay consistently degraded performance, validating the necessity of each component. The distillation coefficient $\lambda_{PCSD}$ was found to be non-monotonic, with 0.01 yielding optimal results.

Significance and Claims

The paper claims that Persistent Consistency is a critical factor in effective self-distillation for agentic RL. By moving beyond isolated token discrepancies or coarse step-level averages, PCSD successfully identifies which teacher signals are trustworthy. The authors emphasize that their method achieves these results without requiring inference-time skill retrieval, relying solely on the trained student policy's ability to leverage persistent teacher guidance.

The work addresses the fundamental trade-off between token-level precision and aggregation-based robustness, offering a solution that allows dense teacher guidance to complement sparse environmental feedback effectively. The authors note that while their current approach uses fixed hyperparameters to isolate the effect of the weighting mechanism, future work could explore learning these parameters dynamically from trajectory statistics and teacher uncertainty.