🤖 AI

From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills

This paper presents a systematic evaluation of the full lifecycle of model-generated agent skills, revealing that while they generally improve performance, their utility varies significantly across extractors and consumers due to non-uniform behaviors and negative transfer, leading to the development of a meta-skill framework that optimizes extraction to enhance quality and reduce failure.

Original authors: Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Xue Yang, Dongdong Chen, Xiaoqing Zheng, Chong Luo

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Xue Yang, Dongdong Chen, Xiaoqing Zheng, Chong Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot butler how to do chores. In the past, humans had to sit down and write a specific manual for every single task: "How to make coffee," "How to fold a shirt," "How to wash dishes." This is slow, expensive, and hard to keep up with.

Recently, scientists tried a new approach: let the robot learn by doing. The robot tries to make coffee, sometimes succeeds, sometimes spills it. Then, a "teacher" robot watches these attempts and writes a summary rule: "Skill: Coffee Making." The hope is that the robot can read this rule later and do better without needing to be retrained.

This paper, titled "From Raw Experience to Skill Consumption," is a massive reality check on that idea. The authors didn't just build one robot; they built a laboratory to test if this "learning from experience" actually works, when it fails, and why.

Here is the breakdown of their findings using simple analogies.

1. The Three-Step Recipe

The authors realized that creating a "skill" isn't just one step; it's a three-stage pipeline, like a cooking process:

Experience Generation (The Cooking): The robot tries to do a task (like making coffee). It creates a log of what happened—some attempts worked, some failed.
Skill Extraction (The Recipe Writing): A second robot (the "Extractor") reads that log and tries to write a concise rulebook (the "Skill").
Skill Consumption (The Cooking Again): The original robot reads the rulebook and tries to do the task again to see if it got better.

2. The Big Surprise: It Works, But It's Risky

The team tested this across five different "kitchens" (domains):

Embodied Planning: A robot moving around a house (ALFWorld).
Productivity: Editing spreadsheets.
Software Engineering: Fixing code bugs.
Web Search: Finding answers online.
Tool Calling: Using digital tools like a calculator or calendar.

The Verdict: On average, giving the robot a skill rulebook does help it perform better. However, it's not a magic bullet.

The "Negative Transfer" Problem: In about 25% of cases, giving the robot a rulebook actually made it worse. It's like giving a chef a recipe that says "add salt," but the chef misreads it as "add salt to the dessert," ruining the dish.
The "Strongest Chef" Myth: You might assume the smartest robot should be the best at writing the rulebooks. The study found this is false. A robot that is great at solving a problem might be terrible at explaining how it solved it. Conversely, a "weaker" robot might write a rulebook that a "stronger" robot can actually use.

3. What Makes a Good Rulebook? (The "Why")

The researchers dug deep to find out why some rulebooks worked and others failed. They tested three stages:

A. The Experience (The Ingredients)

Question: Should the robot only learn from its successes, or should it also learn from its failures?
Finding: It depends on the task.
- For spreadsheets, mostly successful attempts make the best teachers.
- For robot movement, the failures are actually more valuable because they show the robot exactly which paths are dead ends.
- Crucial Point: A pool of only failures is the worst possible teacher. You need some success stories to show the robot what "good" looks like.

B. The Extraction (The Writing Style)

Question: Does the rulebook need to be written in a specific format (like a bulleted list vs. a paragraph) or sound very professional to be useful?
Finding: No.
- Changing the font or the format didn't matter.
- Even more surprisingly, if you asked an AI to judge which rulebook "sounded" better, it was wrong 54% of the time. The rulebook that sounded the most fluent and professional often performed the worst.
- The Real Secret: The best rulebooks weren't the ones that sounded nice; they were the ones that were specific about what went wrong and how to fix it.
- Example: A bad rulebook says, "Be careful." A good rulebook says, "If the formula doesn't calculate, don't trust the computer to do it; calculate the number yourself first."

C. The Consumption (The Eating)

Question: If two robots get the exact same rulebook, will they both improve?
Finding: No.
- One robot might read the rule and suddenly become a pro.
- Another robot might read the exact same rule and get confused or start doing the wrong thing.
- The "skill" isn't just the text; it's how well the text fits the specific brain of the robot reading it.

4. The Solution: A "Meta-Skill"

Since the researchers figured out that "sounding good" is a trap, they created a new tool to fix the process.

They took their findings and turned them into a "Meta-Skill" (a set of instructions for the robot writing the rulebook). Instead of telling the robot to "write a clear and professional summary," they told it:

Identify the Failure: Explicitly state why the robot failed before.
Be Specific: Give concrete steps, not vague advice.
List the Traps: Explicitly list actions that are dangerous to take.

The Result: When they used this new "Meta-Skill" to guide the extraction process, it improved the quality of the rulebooks in every single test case and significantly reduced the number of times the robot got worse.

Summary

This paper teaches us that teaching robots by having them summarize their own experiences is a powerful idea, but it's messy.

Don't assume the smartest robot is the best teacher.
Don't trust a rulebook just because it sounds professional.
Do focus on specific lessons learned from failures and successes.
Do check if the robot actually understands the rulebook, because a good rule for one robot might be a bad rule for another.

The authors have provided a "quality control" checklist (the Meta-Skill) that ensures the rulebooks written by AI are actually useful, moving us from guessing to a science of teaching machines.

Technical Summary: From Raw Experience to Skill Consumption

Problem Statement

Language agents increasingly rely on reusing "skills"—structured procedural artifacts distilled from past experience—to adapt to new tasks without retraining. While domain-level, model-generated skills offer a scalable alternative to labor-intensive hand-crafting, the field lacks a comprehensive understanding of their efficacy. Existing research typically isolates specific stages of the skill lifecycle (e.g., only consumption or only extraction) or relies on benchmarks that do not reflect the full pipeline. Consequently, it remains unclear whether model-generated skills actually improve downstream performance, under what conditions they fail (negative transfer), and what specific properties make a skill useful.

Methodology

The authors propose a systematic, utility-grounded evaluation framework that spans the entire skill lifecycle across five diverse agentic domains: embodied planning (ALFWorld), productivity software (SpreadsheetBench), software engineering (SWE-bench-Verified), web search (SEAL-0), and tool calling (BFCL-v4).

The Skill Lifecycle Pipeline

The study follows a three-stage protocol:

Experience Generation: A target model ( $M$ ) executes tasks from a training split to produce an experience pool containing both successful and failed trajectories.
Skill Extraction: An extractor model ( $E$ ), potentially different from the target, distills the experience pool into a single, structured domain-level skill using a minimal, two-stage framework (per-trajectory analysis followed by hierarchical consolidation). This design intentionally avoids domain-specific heuristics to isolate the extractor's capability.
Skill Consumption: The same target model ( $M$ ) is evaluated on a held-out test split with the extracted skill injected into its system prompt. Performance change ( $\Delta$ ) relative to a no-skill baseline serves as the proxy for skill utility.

Metrics and Experimental Design

The study evaluates 6 target models (spanning GPT, Gemini, and Qwen families) and 5 extractor models across the five domains. To disentangle the roles of the extractor and the consumer, two metrics are introduced:

Extraction Efficacy (EE): Measures how reliably a fixed extractor produces helpful skills across different targets.
Target Evolvability (TE): Measures how much a fixed target benefits from skills extracted by different extractors from its own experience.

Key Contributions and Findings

1. Utility is Non-Uniform and Context-Dependent (RQ1)

Model-generated skills are beneficial on average but exhibit significant variance and non-trivial negative transfer.

Negative Transfer: Approximately 25% of extractor-target pairings result in performance degradation ( $\Delta < 0$ ). This risk is domain-dependent, with ALFWorld showing the highest fragility (47% negative transfer) and SpreadsheetBench the lowest (13%).
Extractor vs. Executor Capability: A model's ability to solve tasks does not predict its ability to extract skills. For instance, a lightweight model (Gemini-3.1-Flash-Lite) achieved the highest Extraction Efficacy on SpreadsheetBench, outperforming stronger baseline models.
Target Dependence: Skill utility is highly target-specific. The same skill can significantly improve one model while degrading another, indicating that consumption ability is an intrinsic property of the target model.

2. Drivers of Skill Utility Across the Lifecycle (RQ2)

The study dissects the three lifecycle stages to identify what drives utility:

Experience Generation: The composition of the experience pool (success vs. failure ratio) critically impacts skill quality. While pure failure pools are consistently the worst, the optimal success-failure mix is domain-specific. For example, ALFWorld benefits from failure-heavy pools (revealing dead-ends), whereas SpreadsheetBench favors successful trajectories.
Skill Extraction: Neither surface format (e.g., list vs. prose) nor textual plausibility predicts downstream utility. An LLM judge evaluating skill text without utility data performs no better than random chance (46% accuracy). Crucially, "better-looking" skills often perform worse.
Skill Consumption: Skill injection reshapes the target's default policy rather than triggering explicit skill calls. For models that improve (e.g., GPT-5.4), skills steer them toward more robust strategies (e.g., computing values in Python rather than relying on unstable formula evaluation). For models that regress (e.g., Qwen3.5-9B), skills may push them toward overly complex workflows that sacrifice execution robustness for structural fidelity.

3. From Diagnosis to Intervention (RQ3)

The authors translate empirical findings into a concrete improvement for skill extraction: a Meta-Skill.

Rubric Discovery: By analyzing high-gap skill pairs, the authors identified that textual plausibility is a poor predictor of utility. Instead, they validated three specific dimensions that correlate with success: Failure Mechanism Encoding (identifying why agents fail), Actionable Specificity (step-level procedures referencing domain objects), and High-Risk Action Blacklists.
Meta-Skill Implementation: These three dimensions were codified into a "validated rubric" and inserted as a meta-skill prior into the extractor's system prompt.
Results: This guided extraction consistently improved skill quality across all evaluated cells (average +1.55 pp gain) and substantially reduced negative transfer. In contrast, a naive "plausibility rubric" (focusing on clarity and tone) actually degraded performance.

Significance

This work moves the field of agent skill extraction from a heuristic, intuition-driven practice toward a principled, utility-grounded discipline. By providing the first systematic study spanning the full experience-extraction-consumption lifecycle, the authors demonstrate that:

Model-generated skills are not universally beneficial; their utility is a complex interaction between the extractor, the target, and the domain.
Surface-level qualities of skills are poor proxies for their actual utility.
Empirical analysis of failure modes and specific actionable remedies can be operationalized into a meta-skill that reliably improves extraction quality and reduces negative transfer.

The study concludes that for skill libraries to be effective in real-world deployments, extraction systems must be guided by validated, utility-grounded criteria rather than generic text quality metrics.