Technical Summary: Skill-R1: Agent Skill Evolution via Reinforcement Learning

Problem Statement

Agentic Large Language Models (LLMs) increasingly rely on skills—reusable natural-language procedures that guide planning, tool use, and action verification. However, current approaches to improving these skills face significant limitations:

1. Cost and Infeasibility: Optimizing skills often requires prompt engineering or fine-tuning the task LLM itself. The latter is computationally expensive, model-specific, and frequently impossible for closed-source or proprietary models.
2. Inefficiency of Current Methods: Many systems treat skill optimization as a one-shot prompt-editing problem or rely on single-round self-refinement. These strategies often fail to align with downstream rewards and do not leverage the recurrent nature of skill improvement.
3. Credit Assignment Complexity: Effective skill optimization is a coupled, bi-level problem. A useful skill must improve the quality of rollouts (execution) under current conditioning, while a useful revision must successfully translate observed successes and failures into a better skill for the next round.

Existing methods struggle to address this as a recurrent decision problem, often lacking a principled objective for directional evolution across multiple generations.

Methodology: Skill-R1

The authors propose Skill-R1, a reinforcement learning framework designed for instance-level recurrent skill optimization from verifiable rewards. The core innovation is the decoupling of task execution from skill improvement.

Core Architecture

• Frozen Task LLM: The primary task-solving model ($\pi_{task}$) remains frozen. It generates rollout trajectories conditioned on the current skill. This ensures compatibility with both open- and closed-source models and eliminates the need for gradients through the task model.
• Lightweight Skill Generator: A separate, trainable policy ($\pi_\theta$) acts as a skill editor. It takes the task context, prior rollouts, and their verified outcomes as input to produce revised skills.
• Multi-Generation Loop: The process operates over multiple generations ($g$). In each generation, the current skill induces a group of rollouts from the frozen task model. A verifier scores these rollouts, and the history of trials, errors, and rewards is appended to guide the next skill revision.

Bi-Level Group-Relative Policy Optimization (GRPO)

To train the skill generator, the authors introduce a bi-level GRPO objective that assigns credit at two levels:

1. Intra-Generation Advantage ($A_{intra}$): Compares rollouts sampled under the same skill within a single generation. This isolates rollout quality from skill quality, similar to standard GRPO.
2. Inter-Generation Advantage ($A_{inter}$): Measures the improvement in the average verified performance of the rollout population relative to the previous generation. This rewards revisions that shift the distribution toward higher rewards.

The final advantage signal is a weighted combination:
$$A(y^{(i)}_g) = A_{intra}(y^{(i)}_g) + \lambda A_{inter}(g)$$
where $\lambda$ interpolates between pure within-generation optimization and cross-generation improvement. The skill generator is optimized to maximize this bi-level advantage, encouraging the production of skills that not only perform well immediately but also evolve directionally over time.

Key Contributions

1. Formulation: The paper formulates recurrent instance-level skill evolution as a bi-level group-relative policy optimization problem, explicitly coupling execution quality within generations with skill improvement across generations.
2. Framework: It proposes Skill-R1, a model-agnostic framework that trains a lightweight skill generator while keeping the task LLM frozen, making it applicable to proprietary models.
3. Algorithm: It introduces a recurrent multi-generation rollout process paired with a bi-level GRPO objective that combines intra- and inter-generation advantages to guide directional evolution.
4. Empirical Validation: The framework is evaluated on agent tasks with verifiable rewards, demonstrating superior performance over no-skill baselines and standard GRPO.

Experimental Results

The authors evaluated Skill-R1 on two benchmarks requiring multi-step reasoning and tool use: GAIA (heterogeneous sources) and WebWalker (multi-hop web navigation).

• GAIA Benchmark:

  • Baseline: GPT-4o-mini without skills achieved only 6.1% accuracy.
  • Vanilla GRPO: Training an editor without bi-level decomposition improved accuracy to 29.7%.
  • Skill-R1 (Inference): Using the multi-generation framework without gradient-based training on the editor achieved 30.9%, suggesting the rollout structure itself provides benefits.
  • Skill-R1 (GRPO): The full system with trained editors achieved 41.8% accuracy, a +12.1 point gain over Vanilla GRPO. The gains were most pronounced on complex Level 3 tasks (38.5% vs. 0.0% for the no-skill baseline).

• WebWalker Benchmark:

  • Baseline: 2.0% accuracy.
  • Vanilla GRPO: 22.0% accuracy.
  • Skill-R1 (GRPO): Achieved 26.0% accuracy, with significant improvements in medium-difficulty and multi-source tasks (29.4% and 28.6% respectively).

• Analysis of Evolution:

  • Generational Progress: Performance gains were largely monotonic, with the most significant jumps occurring between Generations 1 and 3. Later generations (4 and 5) primarily improved the consistency and reliability of rollouts rather than discovering entirely new capabilities.
  • Training vs. Inference: The trained editor (GRPO) consistently outperformed the inference-only editor, particularly on WebWalker. Qualitative analysis showed the trained editor preserved core executable structures and added safeguards, whereas the inference-only editor tended to overfit to spurious details, leading to brittle artifacts.

Significance and Claims

The paper claims that Skill-R1 offers a practical and efficient alternative to model-level adaptation. By treating skill optimization as a recurrent, bi-level problem, the framework enables:

• Black-box Compatibility: It works with closed-source models where fine-tuning is impossible.
• Directional Evolution: It moves beyond heuristic self-refinement to a principled, reward-driven process that improves skills over time.
• Robustness: The recurrent structure allows the system to refine skills based on aggregate evidence (successes and failures) rather than isolated attempts.

The authors conclude that skill evolution is a viable mechanism for improving agent behavior, particularly for complex, multi-step tasks where single-pass refinement methods struggle. The results suggest that a small number of editing rounds can capture the majority of capability improvements, with subsequent iterations enhancing reliability.