🤖 AI

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

ReSkill is a novel agentic RL framework that reconciles skill creation with policy optimization by embedding assertion-driven skill revision, within-group rollout sampling, and Thompson Sampling into the GRPO algorithm, enabling the automatic co-evolution of reusable strategies and policies to achieve superior generalization on unseen tasks.

Original authors: Zelin He, Haotian Lin, Boran Han, Wei Zhu, Haoyang Fang, Bernie Wang, Xuan Zhu, Runze Li, Matthew Reimherr

Published 2026-06-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zelin He, Haotian Lin, Boran Han, Wei Zhu, Haoyang Fang, Bernie Wang, Xuan Zhu, Runze Li, Matthew Reimherr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching a Robot to Learn While It Learns

Imagine you are teaching a robot to play a complex video game.

The Old Way: You teach the robot the rules, let it play, and when it fails, you manually write down a new rule (a "skill") for it to follow next time. But here's the problem: the robot is constantly changing its own playing style as it gets better. If you write a new rule based on how it played yesterday, that rule might actually confuse it today because it has already evolved. It's like trying to teach a toddler how to tie their shoes using a manual written for a grown-up; the toddler has outgrown the old instructions, but the new ones haven't been tested yet.
The RESKILL Way: Instead of writing rules in a separate notebook, RESKILL turns the rule-writing process into part of the game itself. It creates a "rule-making factory" that runs inside the training loop. The robot tries out new rules while it is still learning, sees if they help, and keeps the good ones while tossing the bad ones, all in real-time.

The Core Problem: The "Mismatch"

The paper argues that current methods suffer from a Skill-Policy Conflict.

The Policy: This is the robot's brain (its strategy for playing). It changes constantly as it learns.
The Skills: These are reusable tricks or shortcuts (like "always check the fridge first" or "search before guessing").
The Conflict: Existing methods create skills separately from the brain's training. They might create a skill that worked for the "old" brain but clashes with the "new" brain. It's like a coach giving a player a new playbook while the player is in the middle of a game, without checking if the player actually understands the new moves.

How RESKILL Works: The "Taste-Test" Kitchen

RESKILL solves this by integrating skill creation directly into the training process using three main mechanisms, which the authors call Reconciling Skill Creation with Policy Optimization.

1. The "Group Taste-Test" (Within-Group Sampling)

Imagine a cooking competition where a chef (the AI) has to make a dish.

Standard Method: The chef makes 10 dishes using the same recipe. Then, a critic writes a new recipe. The chef tries the new recipe later.
RESKILL Method: The chef is given a group of 10 orders. For 5 orders, they use the Old Recipe. For the other 5 orders, they use a New Recipe (created on the fly).
Why it works: Because the chef is in the exact same mood and state for all 10 orders, the only difference is the recipe. The system can instantly see: "Did the New Recipe help the chef get a higher score right now?" This allows for a fair, controlled comparison without needing extra time or resources.

2. The "Self-Healing" Rulebook (Assertion-Driven Creator)

When the chef fails a dish, a human usually has to step in and say, "You forgot to salt the soup." RESKILL automates this.

It keeps a Reservoir (a bucket) of every time the chef succeeded and every time they failed.
It uses a "detective" (an LLM) to look at the bucket and ask: "What went wrong? Did we forget to check the oven? Did we search for the wrong ingredient?"
Based on these patterns, the system automatically writes a new "Skill" (a rule like: If you see a pot, check the temperature before stirring).
Crucially: It doesn't just guess. It tests this new rule immediately against the old one using the "Group Taste-Test" described above.

3. The "Smart Gambler" (Thompson Sampling)

How does the system decide how many times to try the New Recipe vs. the Old Recipe?

If the New Recipe looks promising, the system tries it more often.
If the New Recipe looks bad, it tries it less often.
The Twist: The robot's brain is evolving every second. A rule that worked 10 minutes ago might be stale now. RESKILL uses a mathematical trick called Thompson Sampling with Adaptive Discounting.
- Think of it like a gambler who knows that yesterday's winning lottery numbers are useless today.
- If the robot has tried the New Recipe many times recently, the system trusts the newest data and forgets the old data (discounting it).
- If the robot hasn't tried it much, it keeps the old data to avoid making a rash decision based on just one bad attempt.
- This ensures the system is always betting on the skill that works best for the robot's current brain, not its past brain.

The Results: Why It Matters

The paper tested RESKILL on several "games" (tasks):

Household Chores (ALFWorld): Moving objects around a virtual house.
Search Engines: Answering complex questions by searching the web.
Science & Coding: Solving physics problems and writing SQL code.

The Findings:

Better at the Hard Stuff: RESKILL didn't just do slightly better; it crushed the competition on unseen tasks (tasks it had never seen before). This is because the skills it learned were flexible and general, not just memorized answers.
No Extra Cost: It didn't require the robot to play the game twice as long. It did the skill-testing during the normal training steps.
Self-Correction: The system automatically "pruned" (deleted) skills that stopped working as the robot got smarter. It's a living library of skills that grows and shrinks as needed.

Summary Analogy

Imagine a Soccer Team:

Old Method: The coach watches the game, writes a new play on a whiteboard, and tells the team to try it next week. Meanwhile, the players have changed their formation, so the new play doesn't fit.
RESKILL: The coach is on the field during the game. Every time the team attacks, half the players try the old play, and half try a new play invented on the spot based on the last mistake. The coach instantly sees which one scores a goal right now and tells the team to stick with it. The team evolves its strategy and its playbook simultaneously, in perfect harmony.

In short: RESKILL stops treating "learning to play" and "learning the rules" as two separate jobs. It merges them, allowing the AI to build a better brain and a better rulebook at the same time.

Technical Summary: RESKILL – Reconciling Skill Creation with Policy Optimization in Agentic RL

Problem Statement

Agentic Reinforcement Learning (RL) enables Large Language Model (LLM) agents to improve continuously through environment rewards. However, a critical limitation exists: the resulting policies often fail to systematically accumulate reusable, transferable strategies that generalize across tasks. While modular skills (reusable snippets encoding task-specific strategies) offer a solution, existing skill-augmented RL methods typically decouple skill creation from policy optimization.

This decoupling creates a "skill-policy conflict." Methods that evolve skills externally (e.g., via human-in-the-loop testing or static distillation) risk adopting skills that do not align with the agent's evolving policy. Conversely, methods that learn skills without principled testing may accumulate ineffective knowledge. The core challenge is to create a framework where skill evolution and policy learning are reconciled, ensuring that skills are dynamically created, tested, and refined in direct response to the policy's current learning state.

Methodology: The RESKILL Framework

RESKILL (Reconciling Skill Creation with Policy Optimization) addresses this by embedding the skill creation cycle directly into the RL training loop, specifically leveraging the group-wise structure of Group Relative Policy Optimization (GRPO). The framework operates through three integrated mechanisms:

1. Within-Group Skill Testing

Instead of using dedicated, expensive testing rollouts, RESKILL exploits the fact that GRPO samples a group of $G$ trajectories for the same task under the same policy.

Mechanism: Within a single training step, different skill versions (e.g., an existing bank $S_{old}$ vs. a candidate $S_{new}$ ) are assigned to different rollouts in the same group via Thompson Sampling.
Benefit: This allows for a controlled, cost-free comparison of skill versions. Since the task and policy are identical across the group, the only variable is the skill version, enabling direct attribution of performance differences to the skills themselves.
Triple-Duty Rollouts: Each rollout simultaneously serves: (i) policy optimization via GRPO gradients, (ii) failure diagnosis for the skill creator, and (iii) bandit evaluation for skill version selection.

2. RL-in-the-Loop Skill Creation

RESKILL adapts Anthropic's "Skill Creator" pipeline but replaces human feedback with automated, reward-driven signals from the RL loop.

Experience Reservoir: A continuously refreshed reservoir $R$ collects success-failure trajectory pairs annotated with their skill version and reward.
Assertion-Driven Diagnosis: Instead of analyzing single trajectories, the system applies a dynamic set of rule-based assertions (predicates) to the entire reservoir to identify prevalent failure modes.
Conditional Revision: An LLM-based agent proposes candidate skill versions (ADD, MODIFY, DELETE operations) based on these failure profiles. Crucially, proposed triggers are validated against the reservoir to ensure they have sufficient activation coverage before entering the testing phase.

3. Adaptive Thompson Sampling for Skill Evolution

To manage the trade-off between exploring new skill versions and exploiting known effective ones as the policy evolves, RESKILL employs Thompson Sampling with adaptive discounting.

Allocation: At each step, the probability of assigning a rollout to a new skill version is determined by the posterior probability that the new version is superior ( $Pr(p_{new} > p_{old})$ ).
Adaptive Discounting: Because the policy evolves during the evaluation window, older observations become stale. RESKILL introduces a memory parameter $M$ to discount earlier evidence. The discount factor $w_t$ is adjusted based on the number of new episodes $n_t$ : if $n_t$ is small, past evidence is retained; if $n_t$ is large, past evidence is aggressively discounted to reflect the current policy's capabilities.
Accept/Reject: At the end of an evolution cycle, a new skill version is accepted only if its expected performance under the discounted posterior exceeds that of the old version.

Key Contributions

Reconciled Co-Evolution: RESKILL is the first framework to integrate skill creation directly into the policy optimization loop, eliminating the risk of skill-policy misalignment found in decoupled approaches.
Zero-Overhead Testing: By utilizing the group structure of GRPO, RESKILL enables controlled comparison of skill versions without requiring additional rollout budgets or dedicated testing phases.
Adaptive Skill Lifecycle: The framework automates the full lifecycle of skills—creation, testing, refinement, and pruning—based on real-time policy performance and reward signals.
Robustness to Policy Drift: The introduction of adaptive discounting in Thompson Sampling allows the system to handle the non-stationarity of the evolving policy, ensuring skill evaluations remain relevant.

Experimental Results

RESKILL was evaluated across five domains: embodied reasoning (ALFWorld), agentic search (NQ, HotpotQA, and five unseen datasets), science discovery (ScienceWorld), coding (InterCode-SQL), and web shopping (WANDS).

Performance Gains: RESKILL consistently outperformed baselines, including vanilla GRPO, memory-based RL (MEMRL), and existing skill-based methods (SKILLRL, INSPO).
- On ALFWorld, RESKILL achieved a 90.0% success rate (4B model) and 92.7% (8B model), surpassing the strongest baseline (SKILLRL) by significant margins.
- On Agentic Search, RESKILL improved performance by 2.5–2.3% over SKILLRL on seen splits and achieved even larger gains on unseen splits.
Generalization to Unseen Tasks: The most substantial improvements were observed on unseen and hard tasks. For example, on ALFWorld unseen tasks, RESKILL (8B) achieved 95.3%, a 12.7% gap over SKILLRL (82.6%). This suggests that reconciled evolution retains only skill versions that generalize better.
Cross-Domain Adaptation: In a test-time adaptation experiment, a policy trained on ALFWorld was frozen and applied to ScienceWorld. RESKILL rapidly adapted its skill bank to the new domain without policy updates, whereas baseline models failed to evolve.
Efficiency: RESKILL maintained competitive training and inference speeds relative to GRPO. Its conditional trigger mechanism ensured that only relevant skills were loaded, keeping context lengths compact compared to methods that load all skills unconditionally.

Significance and Claims

The paper claims that RESKILL demonstrates that skill creation and policy learning should not be treated as separate processes. By reconciling these two processes, the system develops a general capacity to follow and benefit from new skills, leading to superior performance on complex, unseen, and out-of-domain tasks.

The authors emphasize that the gains stem primarily from the reconciled co-evolution mechanism rather than the specific LLM used for skill creation (ablation studies showed performance remained high even when the skill creator was the base model itself). The framework establishes a new paradigm for agentic RL where skills are not static instructions but dynamic, evolving components that co-adapt with the agent's policy.