EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?
EvoClawBench is a new benchmark evaluating whether AI agents can effectively convert evidence from their own executions into reusable skills, revealing that such self-learning yields highly selective and runtime-dependent results rather than providing automatic performance improvements.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a super-smart robot assistant named "Agent." You give it a messy job, like organizing a chaotic garage or fixing a broken video game code. The robot tries to do it, finishes the attempt (which might be perfect, or it might have mistakes), and then you ask: "Hey, can the robot look at its own work, learn a trick from it, write that trick down as a 'cheat sheet,' and use that cheat sheet to do the next job faster and better?"
That's the big question the EvoClawBench paper asks. It's like testing if a student can take a test, grade their own answer key, write a study guide based on their performance, and then ace the next test using only that guide.
The Big Experiment: Three Ways to Play
The researchers set up 100 different "missions" (like coding, fixing spreadsheets, or analyzing security logs) and tested the robot in three different ways:
- The "Go-Go-Go" Mode (Baseline): The robot gets the job and just does it from scratch. No notes, no cheat sheets. Just pure effort.
- The "Pre-Game Huddle" (Preskill): Before the robot even touches the job, it tries to guess what the job needs and writes a cheat sheet first. Then, it uses that sheet to do the job.
- The "Post-Game Review" (Postskill): The robot does the job from scratch first. Then, the system grades that first attempt and gives the robot a summary of what happened (even if it failed or made errors). The robot uses this summary to write a cheat sheet, and then tries the same job again using the new sheet.
The Shocking Results: It's Not a Magic Wand
You might think, "Of course! If you write down how to do something, you'll get better at it!" But the paper suggests something much more complicated. Learning from your own runs isn't an automatic superpower. It's more like a gamble.
Here is what the numbers (and the paper's careful measurements) tell us:
The Robot's "Brain" Matters More Than the Notes:
The results depended heavily on which robot model was used.- One robot (called nanobot with a model named GPT-5.4) was already so good at the job that it scored over 96% even without any cheat sheets. Adding a cheat sheet didn't help much; it stayed around 96%.
- But another robot (nanobot with DeepSeek-V4-Pro) was decent at first, scoring 77.77%. When it tried to use a cheat sheet it wrote before starting (Preskill), its score crashed to 4.80%. When it tried to use a cheat sheet it wrote after finishing the first try (Postskill), it plummeted to 0.99%.
- Translation: Sometimes, writing down a "how-to" guide actually confuses the robot and makes it do worse!
The "Cheat Sheet" Can Be a Trap:
The paper found that these self-written skills are selective and cost-sensitive.- Cost: Writing the cheat sheet takes time and computer "brain power" (tokens). For some robots, the time spent writing the guide was so long that even if the guide helped a tiny bit, the whole process was slower and more expensive than just doing the job twice without notes.
- The "Overfitting" Problem: The paper suggests that when a robot writes a guide based on one specific attempt, it might get too specific. It might write down, "Click the red button on the left," when the next job actually has the button on the right. The guide becomes a bad habit instead of a helpful tip.
More Notes Don't Mean Better Scores:
The researchers counted how many cheat sheets the robots wrote. Some robots wrote 20 or 30 sheets! But having 30 sheets didn't guarantee a better score. In fact, some robots that wrote more sheets performed worse than the ones that wrote fewer. The quality of the note mattered, not the quantity.
What the Paper Rules Out
The paper is very clear about what it does not prove:
- It does not prove that agents can automatically get better at new, different jobs just by learning from old ones. The tests were on the same job repeated. We don't know if the cheat sheets work for totally new puzzles yet.
- It does not suggest that adding a "skill-writing" step to every robot is a guaranteed win. In many cases, it was a waste of time and money.
The Bottom Line
The paper suggests that teaching an agent to learn from its own runs is tricky. It's not a magic button that makes robots smarter. Sometimes, the "cheat sheet" the robot writes is actually a bad map that leads it into a ditch.
For now, the best approach seems to be picky: only let the robot write a cheat sheet if it's sure the guide will be useful and if the cost of writing it isn't too high. It's a tool that can work, but it's not a free lunch. The robot has to be careful not to overthink its own mistakes!
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.