🤖 AI

Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember

This paper introduces SESA, a self-evolving search agent that establishes a bidirectional loop where informative failures are distilled into an evolving skill memory to guide a solver, which in turn shapes the challenger's problem generation, thereby co-evolving task distributions and procedural knowledge to improve performance across multiple QA benchmarks.

Original authors: Zenghuang Fu, Zhaoyang Li, Qiuyuan Ai, Haoyu Wu, Minghui Wu, Chenxu Zhao, Ante Wang, Guannan He, Changwei Wang

Published 2026-08-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zenghuang Fu, Zhaoyang Li, Qiuyuan Ai, Haoyu Wu, Minghui Wu, Chenxu Zhao, Ante Wang, Guannan He, Changwei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to play a complex video game, but you don't have a rulebook or a list of levels to practice on. Instead, the robot has to invent its own levels as it goes. This is a field of science called Self-Play, where an artificial intelligence (AI) acts as both the student and the teacher. The "teacher" part of the AI creates a puzzle, and the "student" part tries to solve it. If the student gets it right, they get a point; if they get it wrong, they learn a little bit from the mistake and try again.

Usually, when these AI students make a mistake, they learn from it in the moment, but then they forget the specific lesson once they move on to the next puzzle. It's like trying to learn to ride a bike by falling off, getting up, and immediately forgetting why you fell. Another approach, called Skill Memory, is like keeping a physical notebook where you write down every trick you've learned. But often, that notebook is filled with lessons from old, fixed games that the robot didn't choose to play. The big question researchers have been asking is: What if the robot could invent its own hard levels, learn from its own specific failures, and write those lessons into a notebook that changes as it gets smarter?

This is exactly what the paper "Self-Play Meets Skill Evolution" explores. The authors introduce a new system called SESA (Self-Evolving Skill-Augmented Agent). Think of SESA as a robot that has two distinct personalities working together: a "Challenger" and a "Solver." The Challenger is the troublemaker who invents tricky questions. The Solver is the detective trying to find the answers using a search engine.

Here is the magic trick: When the Solver fails to answer a question, instead of just tossing that failure away, SESA turns that mistake into a "Skill Card." Imagine a sticky note that says, "Hey, next time you see a question about a prince born in Prague, remember to check if he has a specific family name, or you might get confused!" The Solver then reads these sticky notes before trying new questions.

But here is the clever part that makes SESA special: The Challenger (the troublemaker) cannot see these sticky notes. The Challenger only sees whether the Solver got the answer right or wrong. This creates a fair game. If the Solver gets better because of the sticky notes, the Challenger realizes, "Oh, my questions were too easy!" and starts making even harder ones. This forces the Solver to learn even more advanced skills, which leads to even harder questions, creating a loop where the AI gets smarter and smarter on its own.

The researchers found that this system works really well. They tested it on seven different types of tricky questions, from simple facts to complex puzzles that require connecting multiple pieces of information. They discovered that SESA improved the AI's accuracy by about 1.2 to 3.2 points compared to the standard method where the AI just forgets its mistakes. Even more interestingly, they found that the AI actually "learned" the lessons so well that it could solve problems correctly even without looking at the sticky notes later on. However, having the notes available still gave it a small extra boost, like having a reference sheet you don't really need but can't resist glancing at.

The paper shows that this "evolving memory" isn't just a fancy trick for the final test; it actually changes how the AI learns during training. By turning failures into reusable skills and feeding them back into the learning process, SESA creates a self-improving cycle. The authors suggest that this approach is better than just memorizing lessons from fixed datasets because it adapts to the AI's own current level of skill, constantly pushing it to the edge of what it can do. In short, SESA proves that an AI that remembers its own mistakes and uses them to build a better future is a much smarter student than one that just moves on to the next question.

Technical Summary: Self-Play Meets Skill Evolution (SESA)

Problem Statement

Current approaches to training Large Language Model (LLM) agents face a dichotomy between self-play and skill-augmented reinforcement learning (RL), both of which suffer from significant limitations:

Self-Play (e.g., SSP): Agents generate their own training problems without external datasets, allowing for adaptive difficulty. However, the learning experience is transient; failures contribute to policy gradients but are discarded afterward, leaving no explicit, reusable record of the strategies learned.
Skill-Augmented RL: Agents utilize external memory banks to store procedural knowledge (skills) derived from experience. However, these skills are typically distilled from fixed datasets or hand-designed curricula, meaning the agent remembers lessons from tasks it did not choose, rather than evolving its own curriculum.

The core problem is the lack of an agent that can simultaneously choose its own practice frontier (via self-play) and retain lessons from its own failures (via persistent skill memory) to create a coupled, co-evolving learning process.

Methodology: SESA (Self-Evolving Skill-Augmented Agent)

SESA introduces a closed-loop framework that integrates online skill evolution into tool-augmented search self-play. The system employs a challenger (proposer) and a solver (learner) with a shared, non-parametric skill memory ( $B$ ).

Core Architecture

Asymmetric Self-Play: The system uses separately parameterized policies. The challenger poses questions with verifiable targets. The solver attempts to answer using a search tool. Crucially, only the solver has access to retrieve skills from the memory bank; the challenger does not. This prevents the leakage of solution strategies into generated questions, preserving an asymmetric game.
Four-Stage Training Loop:
- Memory Priming: The skill bank is initialized with a mix of hand-written skills and deduplicated skills mined from an early bootstrap phase, providing an initial retrieval substrate.
- Frontier Shaping: To ensure learning occurs on solvable but challenging problems, the challenger's reward is shaped using a bell-shaped function. It penalizes trivial (always solvable) and unsolvable (always failing) questions, steering the challenger to generate problems near the solver's current competence boundary.
- Failure Distillation: When the solver fails on a frontier problem, the failure is not discarded. Instead, it is distilled into a human-readable "Skill Card" containing:
  - Description ( $u$ )
  - Trigger conditions ( $c$ )
  - Avoidance cues/anti-patterns ( $a$ )
  - Reusable query templates ( $z$ )
  - Maintenance metadata ( $m$ )
    These skills are deduplicated (using cosine similarity thresholds) and written back to the memory bank.
- Solver Retrieval: In subsequent rounds, the solver retrieves relevant skills from the bank to condition its search strategy. This changes the on-policy trajectory distribution, allowing the policy gradient to internalize the lessons from past failures.

Dual-Path Reuse

SESA supports two modes of deployment, allowing for the isolation of learning effects:

SESA-Off (Memory-Free): The trained solver is deployed without retrieving from the bank. Any performance gain here represents parametric carryover, proving that the skill-conditioned training altered the model's internal weights.
SESA-On (Memory-Augmented): The trained solver uses the final skill bank during inference. The difference between SESA-On and SESA-Off isolates the value of inference-time retrieval.

Key Contributions

Coupled Self-Evolution: The paper unites endogenous problem generation with persistent skill consolidation. The agent chooses its practice frontier and retains lessons from its own failures, creating a feedback loop where task generation and procedural memory co-evolve.
The SESA Loop: A novel mechanism where solver-only retrieval, frontier shaping, and online failure distillation feed reusable skills back into self-play without leaking memory to the challenger.
Dual-Path Evaluation: The authors provide a controlled evaluation protocol separating parametric carryover from inference-time retrieval gains across multiple search benchmarks and model families.

Experimental Results

The authors evaluated SESA on 3,125 held-out questions across seven open-domain and multi-hop question-answering benchmarks (including Natural Questions, HotpotQA, and Bamboogle) using various backbones (Qwen3, Qwen2.5, LLaMA-3.1, and Search-R1).

Performance Gains: Across seven benchmarks, SESA improved average accuracy over the standard Self-Play (SSP) baseline by 1.2–3.2 points across multiple backbones.
- On Qwen3-4B, SESA improved over SSP by 2.3 points.
- On Qwen3-8B, the improvement was 3.2 points.
- On LLaMA-3.1-8B, the improvement was 1.2 points.
Parametric vs. Retrieval Gains:
- SESA-Off (trained with memory, deployed without) retained 1.8–2.2 points of improvement over SSP, demonstrating that the skill-conditioned training successfully internalized capabilities into the model parameters.
- SESA-On (deployed with memory) added a further 0.5–1.0 points over SESA-Off, showing the optional value of the external bank.
Comparison with SkillRL: Under a unified evaluation protocol, SESA surpassed the SkillRL baseline (which learns from fixed distributions) by 0.9 points, indicating that coupling skill evolution to an endogenous task frontier provides additional value.
Component Ablation: Removing failure distillation caused the largest performance drop (2.7 points), confirming that persisting lessons from failures is the most critical component. Removing frontier shaping caused a 2.2-point drop, highlighting the importance of concentrating learning on the competence boundary.

Significance and Claims

The paper claims that evolving skill memory is not merely an inference-time plug-in. Instead, it fundamentally changes policy learning and the future training distribution.

Training State vs. Inference Prompt: By distilling failures into skills that condition on-policy rollouts, the benefits are internalized into the model parameters. This allows for effective memory-free deployment while retaining the option for external augmentation.
Co-Evolution: The results demonstrate that when an agent's memory changes its behavior, it alters the reward signal for the challenger, which in turn changes the distribution of future problems. This creates a self-sustaining loop where the agent continuously pushes its own competence frontier.
Generalizability: The gains persist across different model scales, families, and even search-specialized initializations, suggesting that the mechanism of "failure-to-skill-to-solver" is a robust method for improving agent performance in open-domain search tasks.