Technical Summary: Co-Evolving Skill Generation and Policy Optimization (SAPO)

1. Problem Statement

The paper addresses a critical limitation in existing skill-augmented reinforcement learning (RL) methods for language agents. While prior approaches (e.g., SkillRL, Skill0, D2Skill) utilize reusable procedural knowledge (skills) to improve agent performance, they suffer from three primary issues:

1. Unvalidated Skill Injection: Existing methods often insert newly generated skills into a retrievable skill bank immediately, relying on the implicit assumption that skills produced by strong proprietary Large Language Models (LLMs) are reliable. Preliminary experiments in the paper reveal this assumption is flawed: even skills generated by frontier models (e.g., GPT-5.4) exhibit highly mixed utility, with many providing little benefit or actively degrading performance.
2. Delayed and Non-Contextual Feedback: Current methods track skill utility only after storage, using delayed rollout feedback. Furthermore, these feedback signals typically reflect the aggregate effect of multiple retrieved skills rather than the marginal contribution of a single skill. This makes it difficult to distinguish whether a skill is useful in isolation or merely redundant given the current retrieval context.
3. High Cost and Dependency: Relying on proprietary frontier LLMs for continuous skill generation during online training incurs significant API costs. Additionally, existing methods lack a mechanism to prune outdated skills or rerank candidates efficiently without expensive counterfactual rollouts.

2. Methodology: Skill-Augmented Policy Optimization (SAPO)

The authors propose SAPO, an online RL framework that integrates skill induction, validation, and maintenance into a unified loop. The core innovation is pre-storage skill validation using the standard rollout budget, eliminating the need for additional counterfactual rollouts.

2.1 Online Rollouts and Matched Group Validation

SAPO operates on induction units (queries or groups of semantically similar queries). For each unit, it splits the rollout budget ($G$) into two matched groups under the same task and retrieved skill context:

• Base Rollouts ($Y_{base}$): Generated using the current policy $\pi_\theta$ conditioned only on the task and the set of currently retrieved existing skills ($S_{i,j}$).
• Skill-Augmented Rollouts ($Y_{skill}$): Generated using the same policy and context, but with one candidate skill ($\hat{s}_i$) induced from the base trajectories added to the context.

The candidate skill is induced by the policy itself based on the failure patterns observed in the base rollouts.

2.2 Marginal Utility Estimation

The framework estimates the context-dependent marginal utility of the candidate skill by calculating the reward gap between the two matched groups:
$$u(x_{i,j}, \hat{s}_i) = \text{AvgReward}(Y_{skill}) - \text{AvgReward}(Y_{base})$$
This metric isolates the specific value added by the candidate skill beyond the existing retrieved skills. If the utility is positive and the skill is distinct from existing long-term skills, it is promoted to the long-term bank ($B_{long}$); otherwise, it is discarded.

2.3 Policy as Skill Generator and Scorer

SAPO leverages the utility signal to train the policy $\pi_\theta$ in two capacities, reducing reliance on proprietary models:

1. Skill Generator: The policy is trained with an asymmetric utility-weighted objective (inspired by W-REINFORCE). It is reinforced to generate skills with positive marginal utility and suppressed for negative utility. This allows the open-weight policy to learn to generate high-quality skills over time.
2. Skill Scorer: The probability of the policy generating a specific skill sequence serves as a learned quality score.
   • Training: The policy is distilled to predict the utility of a skill based on the task and context alone (omitting rollout evidence) to enable efficient inference.
   • Maintenance & Retrieval: This likelihood score is used to prune outdated skills from the long-term bank and to rerank candidate skills during retrieval-time selection, ensuring that only the most relevant and non-redundant skills are used.

3. Key Contributions

1. Identification of Mixed Utility: The paper empirically demonstrates that skills generated by frontier LLMs are not inherently reliable, with average marginal utility often near zero, and that unvalidated injection harms learning.
2. SAPO Framework: A novel online RL framework that validates skills before storage using a matched-group rollout strategy, estimating marginal utility without extra rollout overhead.
3. Self-Supervised Skill Generation: The method trains the agent policy itself to act as a skill generator and scorer, effectively replacing costly proprietary API calls for skill induction and enabling context-aware skill maintenance.

4. Experimental Results

The authors evaluated SAPO on ALFWorld (embodied household tasks), WebShop (web navigation), and a suite of Search-Augmented Question Answering benchmarks (single-hop and multi-hop).

• Performance: SAPO consistently outperformed prior skill-augmented RL methods (SkillRL, Skill0, D2Skill) and strong baselines (ReAct, Reflexion, RLOO, GRPO). On ALFWorld, SAPO achieved a success rate of 92.2% (All) and 78.1% (Success), surpassing the previous best (SkillRL at 85.2% and 72.7% respectively).
• Training Dynamics: SAPO demonstrated faster convergence and higher final validation performance compared to SkillRL, particularly in the early stages where utility-based filtering provides reliable guidance.
• Skill Quality: Analysis showed that while the SAPO policy (using a smaller open-weight model) started with lower utility skills than GPT-5.4, it rapidly learned to generate skills with comparable or superior utility, proving the effectiveness of utility-weighted training.
• Ablation Studies: Removing any of the three core components (Validation, Generator Training, or Scoring) resulted in performance degradation, confirming that skill validation, policy-based generation, and likelihood-based maintenance are all essential.

5. Significance and Claims

The paper claims that SAPO addresses the "black box" nature of skill generation in current RL agents. By explicitly validating the marginal utility of a skill within its specific retrieval context, the framework prevents low-quality or redundant skills from polluting the agent's memory.

Crucially, the authors emphasize that SAPO achieves these gains without additional rollout costs (by reusing the standard budget for validation) and without repeated calls to proprietary LLMs (by training the policy to generate and score skills). This makes the approach scalable and cost-effective for long-term agent evolution, shifting the paradigm from "generate and hope" to "validate and evolve."