💬 NLP

OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation

OPD-Evolver introduces a slow-fast co-evolution framework that utilizes on-policy self-distillation to cultivate holistic agent evolvers capable of effectively selecting, utilizing, and maintaining memory, thereby outperforming existing memory systems and training-based methods while enabling smaller models to rival much larger counterparts.

Original authors: Guibin Zhang, Xun Xu, Yanwei Yue, Zikun Su, Wangchunshu Zhou, Xiaobin Hu, Shuicheng Yan

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guibin Zhang, Xun Xu, Yanwei Yue, Zikun Su, Wangchunshu Zhou, Xiaobin Hu, Shuicheng Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: From a "Notebook" to a "Mentor"

Imagine you are teaching a robot how to do chores.

Old Way (Memory Agents): You give the robot a giant notebook. Every time it fails to clean a room, it writes down "I failed." Every time it succeeds, it writes "I succeeded." Later, when it faces a new room, it flips through the notebook to see what happened before.
- The Problem: The robot has a notebook full of notes, but it doesn't know which notes are actually useful. It might read a note about "cleaning the kitchen" while trying to "fix a leaky faucet." It also doesn't know how to write good notes for the future; it might just scribble nonsense that confuses it later.
The New Way (OPD-Evolver): This paper introduces a robot that doesn't just have a notebook; it has a Mentor inside its head. This robot learns how to learn. It figures out which notes to keep, how to use them to act, how to write better notes for next time, and how to throw away the bad ones.

The authors call this an "Agent Evolver." It's not just a robot that remembers; it's a robot that gets smarter at managing its own experience.

The Four Superpowers

The paper argues that a truly "self-evolving" agent needs four specific skills working together. Think of these as the four legs of a table:

Experience Selection (The Filter):
- Analogy: Imagine you are looking for a recipe in a library with millions of books. A bad agent grabs a book on "How to bake a cake" when you asked for "How to fix a car." A good agent (OPD-Evolver) knows exactly which book to pull off the shelf.
- What it does: It picks the most useful memories from a noisy, messy pile of past experiences.
Experience-Grounded Execution (The Doer):
- Analogy: Once you have the right recipe book, you have to actually cook. A bad agent reads the book but forgets to turn on the oven. A good agent uses the book to guide its hands perfectly.
- What it does: It takes the selected memories and uses them to take the right actions in the real world.
Experience Writing (The Journalist):
- Analogy: After cooking, a bad agent writes in the notebook: "It was okay." A good agent writes: "The oven was too hot; next time, lower the temperature by 20 degrees."
- What it does: It turns new experiences (successes or failures) into clear, reusable knowledge that others (or itself) can use later.
Experience Management (The Librarian):
- Analogy: Over time, a library gets full of old, dusty, or wrong books. A bad agent keeps everything forever. A good agent throws away the outdated books and organizes the new ones so they are easy to find.
- What it does: It scores memories, updates them, merges duplicates, and deletes the useless ones to keep the "library" healthy.

How It Works: The "Fast" and "Slow" Loops

The paper uses a clever training method called OPD-Evolver (On-Policy Distillation). Imagine this as a two-step dance between a Student and a Teacher.

1. The Fast Loop (The Student in the Real World)

What happens: The agent goes out and does tasks (like solving math problems or navigating a video game). It interacts with its memory, tries to solve the problem, and gets a result (Success or Failure).
The Catch: At this moment, the agent is just guessing. It doesn't fully know why it succeeded or failed yet. It's just "living" the experience.

2. The Slow Loop (The Teacher in the Review Room)

What happens: After the task is done, the system looks back at what happened. It has a "privileged" view—it knows exactly which memories helped and which ones hurt.
The Magic: The "Teacher" (which has all the hindsight) teaches the "Student" (the agent) by saying:
- "You picked the wrong memory for that task. Next time, pick this one."
- "You wrote a bad note. Next time, write this instead."
- "You kept a useless note. Delete it."
The Result: The agent learns from its own past mistakes and successes, distilling that wisdom into its brain so it can do better next time without needing the teacher to hold its hand.

The Results: Small Brain, Big Wins

The researchers tested this on various challenges, from coding (SQL) to navigating video games (MiniHack).

Beating the Giants: They used a relatively small model (9 Billion parameters) and trained it with this method. Surprisingly, this small, trained agent beat much larger, "giant" models (like those with 397 Billion parameters) on many tasks.
Better than Just "Remembering": It outperformed other systems that just rely on storing memories or simple training methods.
The Takeaway: It's not about having a bigger brain; it's about having a brain that knows how to curate, use, and improve its own knowledge.

Summary in One Sentence

OPD-Evolver is a system that teaches AI agents not just to store their past, but to become masters at selecting the right lessons, acting on them, writing better notes, and cleaning up their own mental library, allowing a small robot to outsmart much larger ones.

Technical Summary: OPD-Evolver

Problem Definition

The paper addresses a critical gap in the development of self-evolving agents. While memory has become a standard substrate for agents to store trajectories, retrieve reflections, and accumulate skills, existing systems often lack the holistic competence to effectively manage the entire experience lifecycle. Current agents may retain experience or expose it to prompts, but they frequently fail to convert this experience into sustained behavioral improvement.

The authors identify that existing methods typically optimize only isolated fragments of the evolution process (e.g., retrieval, context injection, or parameter distillation) rather than the coupled capabilities required for true self-evolution. Specifically, they define a "qualified agent evolver" as an agent that must simultaneously master four coupled capabilities:

Experience Selection: Identifying useful memories from a growing, noisy repository.
Experience-Grounded Execution: Converting selected experience into effective multi-turn actions.
Experience Writing: Extracting reusable knowledge from new trajectories and feedback.
Experience Management: Scoring, consolidating, updating, and retiring memories over time.

The core challenge is that task rewards provide direct supervision for execution but not for memory selection, writing, or long-term management. Furthermore, training these coupled abilities within a single policy without mutual interference remains underexplored.

Methodology: OPD-Evolver

To address these challenges, the authors propose OPD-Evolver, a slow-fast co-evolution framework that cultivates holistic agent evolvers via on-policy self-distillation.

1. Fast Evolution Loop (Online Interaction)

The fast loop operates during test-time interaction without modifying the model parameters ( $\theta$ ). It enables the agent to evolve through online interaction with environments and a four-level memory hierarchy:

Trajectories: Episodic evidence.
Tips: Local warnings or heuristics.
Skills: Abstracted reusable procedures.
Tools: Executable command or code templates.

In this loop, the agent performs:

Retrieval & Selection: Retrieves top- $K$ candidates from each tier via embedding similarity and uses a learned selector to compress them into a compact context ( $S_t$ ).
Execution: Acts based on the task, history, and selected context.
Writing & Maintenance: Post-episode, the agent decides which memories to write to specific tiers and periodically performs repository maintenance (lookup, merge, delete) to manage the growing repository.

2. Slow Evolution Loop (On-Policy Distillation)

The slow loop trains the agent's intrinsic evolver capabilities by distilling "privileged hindsight" from the fast loop's interaction stream. Since the only trustworthy external signal is the terminal task outcome ( $R_t$ ), the framework uses outcome-calibrated memory attribution to propagate this scalar feedback to the four coupled decisions.

Outcome-Calibrated Attribution: The system estimates the value of each memory ( $m$ ) by comparing the performance of tasks where $m$ was selected versus tasks where it was retrieved but not selected. This generates a dense, hindsight supervision signal ( $V(m)$ ) indicating what should have been selected, used, written, or preserved.
Unified Hindsight Self-Distillation: A "privileged teacher" (the same policy with stop-gradient parameters) observes the full hindsight evidence (candidate values, successful trajectories, future utility of written memories, and repository diagnostics). The "student" policy is then trained to mimic the teacher's behavior on the same rollout states but without access to the privileged hindsight.
Loss Function: The training minimizes the token-level KL divergence between the student's on-policy outputs and the teacher's outputs conditioned on the privileged hindsight, jointly shaping selection, execution, writing, and management.

Key Contributions

Objective Formulation: The paper formally defines the "qualified agent evolver" through four coupled competencies (selection, execution, writing, management), arguing that these cannot be safely separated.
Practical Solution: It introduces OPD-Evolver, a framework that converts delayed task outcomes and privileged hindsight into supervision for all four lifecycle stages via on-policy self-distillation.
Experimental Validation: The authors demonstrate that OPD-Evolver-4B and OPD-Evolver-9B surpass contemporary memory systems (e.g., ReasoningBank, MemEvolve) and training-based methods (e.g., Skill0, GRPO) across diverse benchmarks. Notably, the 9B model challenges significantly larger counterparts (e.g., QWEN3.5-397B, STEP-3.5-196B) in specific subsets.

Experimental Results

The evaluation spans five benchmarks: LifelongAgentBench (DB, OS), MemoryArena (Math, Physics), AMA-Bench (CI, SU, SA), InterCode (Bash, CTF, SQL), and MiniHack.

Performance Gains: OPD-Evolver-9B achieved the best results on 9/10 subsets compared to same-backbone memory baselines. It outperformed the strongest memory baseline (ReasoningBank) by up to 11.5% and training-based methods (Skill0) by ~5.8%.
Scaling Efficiency: OPD-Evolver-9B exceeded the performance of the 196B-parameter STEP-3.5-FLASH on 9/10 subsets and surpassed the 397B-parameter QWEN3.5-397B-A17B on 6/10 subsets, demonstrating that lifecycle-level evolution can make compact agents competitive with giant models.
Ablation Studies: Removing any component (slow evolution, memory attribution, selection, writing distillation, or maintenance) led to performance degradation. The removal of memory attribution caused the largest drop, highlighting the importance of outcome-calibrated supervision.
Internalization: Analysis showed that slow-loop distillation internalizes high-value experience, allowing the agent to execute tasks with fewer steps and higher success rates even without external memory retrieval at test time.

Significance and Claims

The paper claims a shift from building agents that merely store experience to cultivating agents that can continuously transform experience into their own capacity for evolution. By unifying the experience lifecycle into a single policy trained via on-policy distillation, OPD-Evolver enables agents to learn transferable, lifecycle-level competencies. The authors suggest that this approach moves beyond simple memory augmentation toward genuinely qualified agent evolvers capable of self-improvement across heterogeneous domains.