💻 computer science

Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds

This paper challenges the notion of steady improvement in self-evolving agent skills by demonstrating through controlled experiments that persistent skill evolution is actually a sparse, validation-filtered search process heavily dependent on feedback dynamics and specific model-benchmark interactions, rather than a consistent gain from additional refinement rounds.

Original authors: Yuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, Yangqiu Song

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, Yangqiu Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant that can do amazing things, like solve math problems or write code. But sometimes, it makes mistakes. In the world of artificial intelligence, there's a popular idea called "self-evolving." The dream is that instead of just fixing a mistake once, the robot could learn from its errors, write down a new rule for itself, and remember that rule forever. It's like a student who, after failing a math test, writes a new study guide in their notebook that helps them ace the next one without needing a teacher to rewrite their brain.

But here's the big question: Does this actually work? Does looking at every attempt (both the good ones and the bad ones) help the robot learn better? Or is it smarter to only look at the failures to see what went wrong? And is it even better to just try the same task over and over again with more computing power, rather than trying to write a permanent new rule? This paper dives into these questions to see if "self-evolving" is a magic bullet or just a lot of work for a tiny reward.

The Great Robot Skill Hunt

The researchers set up a massive experiment to see how these AI agents actually learn. They treated the AI like a video game character trying to level up its "skills." They gave the AI three different ways to learn from its gameplay:

The "Normal" View: The AI sees both its wins and its losses.
The "Fail-Only" View: The AI only sees its mistakes.
The "Success-Only" View: The AI only sees its victories.

They ran this experiment across 14 different scenarios, using three different powerful AI models and five different types of challenges, ranging from answering trivia questions to manipulating complex spreadsheets.

The Surprise: Evolution is Rare and Picky

The biggest shock? Evolution is incredibly rare. Out of 388 different attempts to create a new, better skill, only 55 actually resulted in a distinct, improved version that passed the strict tests. It's like trying to bake a perfect cake 388 times and only getting 55 cakes that are actually better than the original recipe.

Even more surprisingly, the "Success-Only" view was a total flop. In the main study, zero of the improved skills came from only looking at the wins. The AI needed to see the failures to figure out what to fix. In fact, every single one of the 11 best skills that the researchers decided to keep came from a view that included at least some failures.

The "Search" vs. The "Steady Climb"

Many people thought that if you just let the AI keep trying for more rounds, it would steadily get better, like climbing a ladder. The paper suggests this is wrong. Instead, the process is more like searching for a hidden treasure in a dark cave.

Sometimes, the AI finds a great new skill on the very first try. Other times, it wanders around for a long time, trying many ideas that don't work, before suddenly stumbling upon a great solution in the 9th round. But often, the AI just hits a wall and stops improving completely. The researchers found that waiting for more rounds doesn't guarantee a better skill; it just costs more time and money.

The "Magic Trick" Comparison

The researchers also asked: Is writing a new permanent skill actually better than just trying the task multiple times with extra computing power?

They compared the "evolved" skills against two other methods:

Parallel Sampling: Trying the task many times at once and picking the best answer.
Sequential Refinement: Trying the task, seeing the result, and trying again based on that.

The results were mixed. For a trivia question game called SearchQA, the "evolved" skill was only slightly better than just trying many times. The AI's new rule was mostly about formatting the answer nicely, which is something you can get by just guessing a few times.

However, for a spreadsheet challenge called SpreadsheetBench, the difference was huge. The evolved skill was 30.96 points better than the best "try many times" method. Why? Because the spreadsheet task required a complex, multi-step workflow (like checking a file, running a script, saving it, and verifying it). You can't just "guess" your way through that; you need a permanent, written-down rule to get it right.

The Takeaway

So, what's the verdict? Self-evolving AI skills aren't a steady, automatic upgrade. They are more like a sparse, hit-or-miss search that depends heavily on the specific AI model and the specific task.

If you want your AI to get better at complex, multi-step tasks (like handling spreadsheets), teaching it to learn from its failures and write down a permanent rule is a game-changer. But if the task is simple or just needs a better answer format, you might get just as good results by just giving the AI more time to try, try, and try again. The paper suggests we shouldn't expect magic; instead, we should view self-evolution as a careful, filtered search for the right tool for the job.

Technical Summary: Rethinking Self-Evolving Agent Skills

Problem Statement

Self-evolving skill systems aim to improve agent performance by converting execution feedback into persistent skill updates without modifying the underlying model parameters. While prior work reports gains over skill-free baselines and even human-authored skills, the field lacks a clear understanding of three critical dynamics:

Feedback Composition: It is unclear whether successful trajectories, failed trajectories, or a mixture of both are most effective for shaping skill revisions.
Evolution Trajectory: Existing evaluations often report only "before-and-after" scores, obscuring the search process, rejected candidates, and whether additional evolution rounds yield diminishing returns or steady improvement.
Value of Persistence: It remains uncertain whether the gains from persistent skill evolution can be replicated by increasing test-time computation (e.g., parallel sampling or sequential refinement) without altering the skill artifact itself.

Current systems typically evaluate their native feedback policies as a whole, failing to isolate the specific contribution of different evidence types (success vs. failure) under a fixed revision procedure.

Methodology

The authors present a controlled evaluation framework designed to isolate the effects of feedback composition and evolution dynamics across multiple models and benchmarks.

Experimental Setup

Models: The primary study utilizes three underlying models: GPT-5.5, Gemini 3.1 Pro, and DeepSeek V4-Pro. A broader analysis extends to eight models for the SearchQA benchmark.
Benchmarks: Five heterogeneous benchmarks are used: SearchQA, OfficeQA, SpreadsheetBench, LiveMath, and DocVQA.
Feedback Conditions: For each model–benchmark setting, three feedback views are compared, all starting from the same parent skill and using identical executor/optimizer configurations, revision procedures, and a 10-round budget:
- Normal: Exposure to both successful and failed trajectories.
- Fail-only: Exposure to failed trajectories only.
- Success-only: Exposure to successful trajectories only.
Evolution Protocol: In each round, the revision operator proposes a candidate skill based on the current skill and the specific feedback view. A candidate becomes the next-round skill only if its validation score does not decrease. A separate "best checkpoint" is updated only upon a strict validation improvement and a byte-distinct candidate (ensuring the skill artifact has actually changed).
Evaluation: Post-evolution, the frozen "validation-best" skills are evaluated on:
- Released Test: Standard benchmark performance.
- Robustness: Same-task variations (surface form changes, distractors, interface contract changes).
- Transfer: New tasks from the same source or different subtypes.
Test-Time Scaling Controls: For GPT-5.5, the evolved skills are compared against two controls initialized from the frozen parent skill:
- Parallel Sampling: Oracle "any-success" over $K$ attempts.
- Sequential Refinement: Conditioning subsequent attempts on the preceding response.

Key Contributions

Cross-Model Evolution Dynamics: The study traces 388 candidates across 42 feedback runs, revealing that skill evolution is characterized by sparse, non-monotonic search rather than steady improvement. It documents late discoveries, early saturation, and complete stagnation across different model families.
Controlled Feedback Comparison: By holding the evolution procedure fixed, the paper isolates the impact of feedback views. It demonstrates that all 11 selected evolved skills in the primary study came from conditions containing failed trajectories (Normal or Fail-only), while Success-only was never selected.
Evolution vs. Test-Time Computation: The paper explicitly separates persistent skill gains from per-instance inference gains. It shows that while test-time scaling can recover some gains (e.g., on SearchQA), it fails to replicate the substantial improvements seen in complex, multi-step workflows (e.g., SpreadsheetBench).

Key Results

1. Sparsity and Feedback Dependence

Sparse Success: Only 55 of 388 candidates established byte-distinct validation bests.
Selection Bias: Validation selected an evolved skill in 11 of 14 settings. All 11 selections originated from feedback views containing failures (9 from Normal, 2 from Fail-only). Success-only was never selected in the primary study.
Performance Gains: 9 of the 11 selected skills improved released-test performance. SpreadsheetBench showed the most consistent and large gains (up to +37.7 points), while SearchQA showed smaller, variable gains.
Generalization: Selected skills improved robustness and transfer in 9 settings each, with 7 improving both. However, generalization is not guaranteed; for example, GPT-5.5 on LiveMath improved robustness but lost test performance.

2. Evolution Trajectories

Non-Monotonic Search: Evolution behaves as a "validation-filtered search." Gains are not steady; runs often exhibit saturation (e.g., SpreadsheetBench Normal peaking at round 3 and declining thereafter) or late discovery (e.g., SearchQA Normal improving at round 9 after a plateau).
Round Budgets: While 69% of new bests occur in rounds 1–4, 6 of the 11 final selected skills first appeared in rounds 6–9, indicating that longer budgets are necessary to capture late-stage improvements despite diminishing average yields.

3. Test-Time Scaling vs. Persistent Skills

SearchQA: Oracle Parallel Sampling nearly recovered the evolved skill's gain (within 0.43 points), suggesting the improvement was largely due to answer-surface guidance (e.g., canonical naming) that independent samples could explore.
SpreadsheetBench: A massive gap remained (30.96 points) between the evolved skill and Parallel Sampling. The evolved skill encoded a complex, multi-step workflow (inspect, script, save, verify) that random sampling could not reproduce.
Sequential Refinement: Failed to recover gains on either benchmark, indicating that conditioning on previous responses does not inherently provide the corrective feedback needed for complex procedural repair.

4. Artifact Analysis

Success-Only Limitations: Success-only feedback often leads to overfitting on incidental patterns or missing critical corrections because it lacks negative contrast. In one case (DeepSeek–LiveMath), it caused a validation drop from 40.0 to 11.4.
Failure-Driven Specificity: Fail-only views often produced highly specific, verifier-targeted fixes (e.g., handling empty cells in spreadsheets), while Normal views combined these with successful examples to broaden the guidance.

Significance and Claims

The paper argues that persistent skill self-evolution should be understood as a sparse, validation-filtered search process rather than a mechanism for steady, incremental improvement.

Feedback is Critical: The composition of feedback (specifically the inclusion of failures) is a primary determinant of success. Success-only feedback is insufficient for robust skill evolution in the studied settings.
Search vs. Inference: The benefits of self-evolution are distinct from test-time scaling. While scaling can help with response diversity, persistent skills are necessary for encoding complex, multi-step procedural knowledge that cannot be recovered through repeated inference attempts.
Evaluation Standards: The authors claim that evaluations should move beyond simple before-and-after scores. They advocate for reporting full search trajectories, skill identities (byte-distinctness), downstream generalization (robustness/transfer), and explicit test-time computation controls to accurately assess the value of self-evolving systems.

The study concludes that while self-evolution holds promise, its returns are highly dependent on the specific model, benchmark, and feedback view, and it requires careful management of the search process to avoid stagnation or regression.