🤖 AI

SkillMaster: Toward Autonomous Skill Mastery in LLM Agents

SkillMaster is a novel training framework that enables LLM agents to autonomously create, refine, and select skills through trajectory-informed review, counterfactual utility evaluation, and a dual-advantage reinforcement learning algorithm, thereby achieving significant performance improvements and self-improvement capabilities on complex tasks without external guidance.

Original authors: Min Yang, Jinghua Piao, Xu Xia, Xiaochong Lan, Jiaju Chen, Yongshun Gong, Yong Li

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Min Yang, Jinghua Piao, Xu Xia, Xiaochong Lan, Jiaju Chen, Yongshun Gong, Yong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: From a Librarian to a Self-Taught Expert

Imagine you are teaching a robot (an AI agent) how to do chores, like cleaning a house or shopping online.

The Old Way (Current Technology):
Think of the robot as a student with a very strict, external Librarian.

The robot tries to do a task.
If it fails, the Librarian (an outside computer program) looks at what happened, writes a new "rule" or "skill" on a piece of paper, and puts it in a book.
The robot doesn't know how to write these rules; it just has to look them up in the book when it needs them.
The Problem: The robot is passive. It can't decide if a rule is good or bad, and it can't fix a rule that is slightly wrong. It just follows the Librarian's instructions.

The New Way (SKILLMASTER):
SKILLMASTER turns the robot into a Self-Taught Expert who manages its own notebook.

The robot tries a task.
Afterward, it looks at its own performance and asks: "Did I do well? Did I miss a step? Is there a better way to do this?"
It then uses a special tool to write a new rule, fix an old rule, or keep the current rules in its own notebook.
Crucially, it learns how to write these rules by seeing if they actually help it do better next time.

How It Works: The Three Magic Tricks

The paper introduces three specific tricks to make this self-learning possible.

1. The "Post-Game Review" (Trajectory-Informed Skill Review)

The Analogy: Imagine a basketball player watching a recording of their game immediately after playing.

Old Way: A coach watches the tape and tells the player, "Next time, shoot from the left side."
SKILLMASTER Way: The player watches the tape, realizes, "I kept missing because I wasn't checking the corner first," and writes that note into their own playbook.
In the Paper: After the AI finishes a task (like finding a tomato in a fridge), it reviews the whole story of what happened. It then uses a "tool call" (like a digital pen) to decide: Propose a new skill, Update an old one, or Keep things as they are.

2. The "Test Drive" (Counterfactual Utility Reward)

The Analogy: Imagine you invent a new way to tie your shoes. How do you know it's actually better? You don't just guess; you try it on a different pair of shoes to see if it works faster.

The Problem: If the robot changes its rulebook, how does it know the change is good? Just because it succeeded once doesn't mean the new rule is perfect.
SKILLMASTER Way: When the robot suggests a change to its skills, the system runs a "Test Drive." It takes a different but similar task (a "probe task") and tries to do it using the old rules vs. the new rules.
- If the new rules make the robot finish faster or succeed where it used to fail, the robot gets a "good job" reward.
- If the new rules make things worse, it gets a penalty.
Result: The robot learns to only keep changes that actually improve its performance on similar future tasks.

3. The "Two-Track Brain" (DualAdv-GRPO)

The Analogy: Imagine a driver who is also a mechanic.

Track A (Driving): "I need to turn the wheel left to avoid a pothole." (This needs immediate feedback).
Track B (Mechanic): "I should tighten this bolt so the car runs better next week." (This needs long-term feedback).
The Problem: If you mix these two goals together, the brain gets confused. "Should I turn left or tighten the bolt?"
SKILLMASTER Way: The system separates these two thoughts. It calculates the "score" for driving actions separately from the "score" for writing new rules. Then, it combines them carefully so the robot learns to drive and learn how to learn, without one messing up the other.

What Happened in the Experiments?

The researchers tested this on two famous "video game" worlds for AI:

ALFWorld: A simulated house where the robot has to move objects (e.g., "Put the cold tomato in the fridge").
WebShop: A simulated online store where the robot has to find and buy specific items (e.g., "Buy a blue shirt under $20").

The Results:

Better Scores: SKILLMASTER beat all other methods, including those using powerful, pre-trained "teacher" models. It improved success rates by about 8.8% to 9.3%.
Self-Improvement: The robot didn't just get lucky; it actually learned to spot its own mistakes. For example, if it kept searching the wrong drawers for food, it wrote a new rule: "Don't search low-probability zones first."
Internalizing Skills: Surprisingly, after training, the robot didn't even need to look up its "notebook" as much. It had learned the skills so well they became part of its natural thinking process, like a human who learns to ride a bike and doesn't need to think about balancing anymore.

Summary

SKILLMASTER is a framework that stops treating AI skills as static files managed by an outside computer. Instead, it gives the AI the ability to write, edit, and test its own rules based on its own experiences. It's the difference between a student who is handed a textbook and a student who writes their own textbook, tests the chapters, and keeps only the ones that help them get an 'A'.

Technical Summary: SKILLMASTER

Problem Statement

Current Large Language Model (LLM) agents struggle with autonomous skill mastery. While skills serve as effective mechanisms for improving performance on complex tasks, existing frameworks treat skills as external resources managed by external teachers, hand-designed rules, or auxiliary modules. In these setups, the agent's policy is limited to retrieving and invoking pre-defined skills; it cannot actively develop, refine, or internalize them based on its own experience.

This reliance on external management creates three primary challenges:

Lack of Internalization: Skill management is a maintenance procedure external to the agent, rather than a learnable capability within the agent's policy.
Sparse Feedback for Skill Quality: Task outcome rewards (success/failure) are too coarse to determine if a specific skill edit (proposal or update) was beneficial. A task might succeed due to luck or fail due to unrelated factors, providing no explicit signal on the quality of the skill itself.
Optimization Interference: Jointly optimizing task execution and skill management in a single policy is difficult because they have different reward structures (binary environment rewards vs. continuous skill-quality rewards), often leading to unstable training.

Methodology: SKILLMASTER

The authors propose SKILLMASTER, a training framework that integrates skill management directly into the agent's learning loop, enabling the agent to autonomously create, refine, and select skills. The framework consists of three core components:

1. Trajectory-Informed Skill Review

SKILLMASTER augments standard reinforcement learning (RL) training with an explicit Skill-Mastery Phase appended after every episode.

Acting Phase: The agent interacts with the environment, retrieving relevant skills from a skill bank ( $B$ ) to guide actions.
Skill-Mastery Phase: Upon episode termination, the agent reviews the task description, retrieved skills, trajectory, and outcome. It is instructed to output exactly one tool call: propose_skill (add new), update_skill (revise existing), or keep_skill (retain current).
Mechanism: This phase uses Tool-Integrated Reasoning (TIR), allowing the agent to treat the skill bank as a mutable object it can edit via structured function calls, rather than a static database.

2. Counterfactual Skill Utility Reward

To solve the problem of sparse feedback, the authors introduce a Downstream Utility Reward based on counterfactual evaluation.

Probe Tasks: When a skill edit is proposed, the system selects $K$ semantically related "probe tasks" (e.g., from the same task family).
Counterfactual Comparison: The system evaluates the probe tasks twice: once with the original skill bank ( $B$ ) and once with the modified bank ( $B'$ ).
Reward Signal: The reward is calculated based on two metrics:
1. Success Rate: Does the modification help previously failing tasks succeed?
2. Efficiency: Does the modification reduce the step count for tasks that were already solvable?
Formula: The utility reward ( $R_{utility}$ ) combines the average performance delta ( $\bar{\delta}$ ) across probes with a directional consistency term, ensuring edits are only rewarded if they consistently improve downstream performance.

3. DualAdv-GRPO (Decoupled Optimization)

To address the instability of jointly optimizing heterogeneous objectives (task execution vs. skill editing), the authors propose DualAdv-GRPO.

Dual-Stream Advantage Estimation: Standard Group Relative Policy Optimization (GRPO) normalizes rewards across a group of trajectories. SKILLMASTER separates this into two streams:
- Action Advantages ( $A_{act}$ ): Normalized using only environment rewards ( $r_{env}$ ).
- Skill Advantages ( $A_{skill}$ ): Normalized using only skill-mastery rewards ( $R_{skill}$ ).
Merging: These normalized advantages are merged at the token level using a tunable weight $\gamma$ . Action tokens receive $A_{act}$ , while skill-management tokens receive $\gamma \cdot A_{skill}$ . This prevents the sparse, continuous skill rewards from distorting the learning signal for immediate task actions, enabling stable joint training.

Key Contributions

SKILLMASTER Framework: A unified policy that learns both task execution and skill management end-to-end, removing the need for external teachers.
Downstream Utility Reward: A novel reward mechanism that evaluates skill edits via counterfactual comparisons on probe tasks, providing explicit, dense signals for skill quality.
DualAdv-GRPO: An optimization algorithm that decouples advantage normalization for task actions and skill decisions, stabilizing the joint training of heterogeneous objectives.
Empirical Performance: Demonstrated state-of-the-art results on ALFWorld and WebShop, showing that agents can autonomously improve their own procedural knowledge.

Experimental Results

The framework was evaluated on two benchmarks: ALFWorld (embodied household manipulation) and WebShop (online shopping).

ALFWorld: SKILLMASTER achieved an overall success rate of 98.7%, outperforming the strongest baseline (SkillRL, which uses an external teacher) by 8.8 percentage points. It achieved perfect success (100%) on four of the six task families.
WebShop: The success rate improved from 72.7% (SkillRL) to 82.0%, with the overall score rising from 85.2 to 95.0.
Skill Internalization: A critical finding is that SKILLMASTER agents can internalize learned skills. When evaluated without test-time skill retrieval, the agents maintained performance nearly identical to when retrieval was enabled (only a 0.7% drop overall), suggesting the procedural knowledge was absorbed into the policy parameters rather than relying solely on external lookup.
Ablation Studies: Removing the utility reward or the decoupled advantage normalization significantly degraded performance, confirming the necessity of both the counterfactual signal and the stable optimization strategy.

Significance and Claims

The paper claims that SKILLMASTER represents a shift from externally managed skill systems to autonomous skill mastery. By treating skill management as a learnable objective within the agent's policy, the framework enables agents to:

Identify gaps in their procedural knowledge (e.g., realizing a search strategy is inefficient).
Refine existing skills based on trajectory evidence (e.g., correcting a "heat" instruction that fails when an object is placed inside a microwave).
Transfer improvements to future tasks with minimal edits to the skill bank.

The authors position this work as a move toward self-improving agents capable of developing and adapting their own skill repertoires, rather than merely invoking static tools provided by external systems. The work acknowledges limitations, such as the computational overhead of probe evaluations and the reliance on predefined task groupings for probe selection, but asserts that the core mechanism is domain-agnostic and applicable to any environment with related tasks.