🤖 machine learning

DREvo: Distilling Recalibrated Historical Experience for Harness Self-Evolution

DREvo is a novel harness self-evolution method that addresses the instability of existing approaches by dynamically recalibrating historical experience and distilling actionable search directions, thereby achieving superior performance and smoother evolution trajectories across reasoning and agentic benchmarks under limited budgets.

Original authors: Hanghui Guo, Weijie Shi, Zhangze Chen, Shengxiang Xu, Yishu Wang, Yimei Zhang, Wangze Ni, Jia Zhu, Shimin Di

Published 2026-07-30

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanghui Guo, Weijie Shi, Zhangze Chen, Shengxiang Xu, Yishu Wang, Yimei Zhang, Wangze Ni, Jia Zhu, Shimin Di

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to solve a puzzle. You give the robot a brain (a Large Language Model), but the brain needs a body and a set of rules to actually do the work. This "body and rules" setup is called a harness. Think of the harness as the robot's operating system: it decides how the robot looks up information, how it uses tools like a calculator or a code editor, and how it remembers what it just did. If the harness is messy or confusing, even the smartest brain will stumble.

For a long time, building a good harness was like trying to tune a race car by hand. Experts had to guess what was wrong, tweak a few wires, test it, and hope for the best. But recently, scientists discovered that robots could actually help fix their own bodies. They could try a change, see if it worked, remember the result, and try again. This is called self-evolution. The idea is that if the robot keeps a diary of all its past tries, it can learn from its mistakes and get better over time. But here's the catch: just because the robot has a diary doesn't mean it knows how to read it. Sometimes, a trick that worked yesterday might be a disaster today because the robot's situation has changed. This paper asks: How do we make sure the robot learns the right lessons from its history without getting confused by old, outdated advice?

The Problem: The Robot's Confusing Diary

Imagine you are training a dog to fetch a ball. You throw the ball, the dog runs, and sometimes it catches it, sometimes it drops it. You write down every attempt in a notebook. Now, imagine the dog gets a new collar, or the wind starts blowing differently, or you switch from a tennis ball to a squeaky toy. If you just look at your notebook and say, "Last time the dog dropped the ball when it ran left, so it should always run right," you might be wrong. That old lesson might not apply to the new situation.

This is exactly what happens with AI agents. Researchers found that when robots try to improve their own "harness" by looking at their entire history of tries, they often get stuck in a loop. They might try a fix, see it fail, try something else, see it fail again, and then suddenly go back to a method that failed weeks ago. Their performance jumps up and down like a rollercoaster instead of smoothly climbing a hill. The researchers realized the robots were suffering from two main problems:

The "Is this still true?" Problem: The robot didn't check if its old lessons were still valid. It treated every past success or failure as if it were happening right now, even though the robot's code had changed.
The "What do I do next?" Problem: Even if the robot remembered a lesson, it didn't know exactly which part of its body to fix or how to fix it. It was like being told, "You dropped the ball," without being told, "You need to tighten your grip."

The Solution: DREvo (The Smart Librarian)

To fix this, the authors created a new method called DREvo. You can think of DREvo as a super-organized librarian who manages the robot's diary. Instead of just handing the robot a stack of papers, DREvo organizes the information, checks if it's still relevant, and gives the robot a clear, specific instruction on what to do next.

DREvo does this in three fun steps:

1. The "Label Maker" (Function-Level Evidence Anchoring)
In the old way, the robot's diary was a giant, messy blob of text. DREvo cuts this blob into tiny, labeled pieces. It looks at every change the robot made and tags it to a specific "function" (like a specific tool or memory rule). It's like taking a messy recipe book and labeling every single ingredient change to the exact step it belongs to. This way, when the robot wants to learn, it knows exactly which part of its body a past lesson is talking about.

2. The "Freshness Checker" (State-Dependent Evidence Recalibration)
This is the most important part. Before the robot uses an old lesson, DREvo asks two questions: "Is this lesson still reliable?" and "Does this lesson fit the robot's current body?"

Reliability: Did this lesson work every time it was used before, or was it just a lucky guess?
Freshness: Is the robot's code still similar to when this lesson was learned? If the robot has changed its "grip" (its code structure), an old lesson about gripping might not work anymore.
DREvo gives a "score" to every lesson. If a lesson is old or the robot has changed too much, the score drops, and the robot ignores it. If the lesson is fresh and reliable, the score stays high.

3. The "Coach's Whistle" (Role-Conditioned Search Intent Distillation)
Finally, DREvo doesn't just give the robot a list of facts; it gives it a game plan. Based on the high-scoring lessons, DREvo assigns the robot a specific "role" for the next try:

Exploit: "This trick worked great before! Do it again!"
Avoid: "This trick caused a crash last time! Don't do it!"
Retest: "We aren't sure about this one. Let's try it carefully to see if it works now."
Explore: "We have no good clues. Let's try something completely new."
This turns the robot's vague "I need to get better" feeling into a clear, actionable command like, "Go fix the memory tool using the 'Exploit' strategy."

What They Found

The researchers tested DREvo on five different challenges, ranging from solving chemistry puzzles and diagnosing medical symptoms to fixing computer code and navigating a virtual terminal. They compared it against other robots that tried to evolve themselves and against robots with human-designed harnesses.

The results were quite promising. Under a limited budget of tries (meaning they didn't let the robot practice forever), DREvo found the best harnesses in every single category.

On medical diagnosis tasks, DREvo reached 89.2% accuracy, beating the second-best method by 2.4 percentage points.
On legal reasoning, it hit 49.0%, beating the next best by 4.0 percentage points.
On chemical reaction prediction, it scored 25.0%, which was 9.0 percentage points higher than the runner-up.
For agentic tasks (like fixing code or using a computer terminal), DREvo improved performance by an average of 16.2% on reasoning tasks and 13.8% on agent tasks compared to the other methods.

Perhaps most importantly, the researchers noticed that DREvo's progress was smooth. While other robots' scores jumped up and down wildly (regressing and recovering), DREvo's accuracy climbed steadily. It also managed to keep its "thinking space" (context) small, around 2.9K tokens, whereas other methods needed over 5.3K tokens and still performed worse.

The Takeaway

The paper suggests that simply giving a robot a history of its past isn't enough. To truly learn, the robot needs a system that can check if old lessons still apply and translate those lessons into clear, specific instructions. DREvo acts as that system, turning a messy history of trial and error into a smooth, reliable path to improvement. It shows that by being smart about how we use our past experiences, we can help AI agents evolve much faster and more reliably, even when we don't have unlimited time to let them practice.

Technical Summary: DREvo – Distilling Recalibrated Historical Experience for Harness Self-Evolution

Problem Statement

The performance of Large Language Model (LLM) agents is heavily dependent on their external execution framework, known as the harness, which governs context construction, memory management, tool invocation, and output parsing. While manual harness design is effective, it is labor-intensive and difficult to scale as tasks and component interactions grow more complex. Consequently, research has shifted toward harness self-evolution, where LLMs iteratively propose, evaluate, and improve harnesses using historical trial experience.

However, existing history-driven self-evolution methods (e.g., Meta-harness, AHE, ACE) suffer from non-stationary evolution trajectories. Despite accumulating historical experience, accuracy often fluctuates significantly rather than improving steadily. The authors identify two core limitations in how current methods leverage historical data:

Lack of Dynamic Reassessment ("What is still valid?"): Historical evidence is state-dependent. A modification that was effective in a previous harness state may become invalid or harmful as the harness evolves, yet existing methods treat past records as statically valid.
Lack of Explicit Search Direction ("Where to evolve next?"): Historical experience is often presented as raw logs or summaries. Existing methods rely on the proposer's implicit reasoning to decide which component to modify and how, leading to unstable search directions and inefficient use of limited evolution budgets.

Methodology: The DREvo Framework

To address these limitations, the authors propose DREvo (Distilling Recalibrated Historical Experience), a self-evolution method that transforms raw trial logs into actionable, state-aware guidance. The framework consists of three sequential components:

1. Function-Level Evidence Anchoring

Raw trial logs are heterogeneous, mixing code changes, execution trajectories, and feedback. DREvo parses these logs to create function-anchored evidence units.

Mechanism: It maps code diffs (changes between $h_{t-1}$ and $h_t$ ) to specific functional components (e.g., tool use, memory management) based on source-span boundaries.
Output: Each evidence unit $e_{t,c}$ links a specific component $c$ , its code change $\delta_{t,c}$ , and the observed outcome (global stats, task outcomes, failure patterns).
Attribution: This localizes feedback to specific components, allowing for more precise attribution of success or failure.

2. State-Dependent Evidence Recalibration

Before reusing historical evidence, DREvo dynamically recalibrates its validity based on the current harness state. It computes a calibrated weight $w_e$ for each evidence unit based on two factors:

Historical Reliability ( $r_t(e)$ ): Measures consistency and freshness.
- Outcome Consistency: The fraction of past invocations where the outcome polarity (success/failure) matched the current evidence polarity, weighted by performance change magnitude.
- Freshness: A decay factor based on the time since the evidence was last used or created.
Structural Compatibility ( $q_t(e)$ ): Measures whether the component implementation associated with the evidence remains structurally similar to the current component. This is calculated using Tree Edit Distance (TED) on Abstract Syntax Trees (ASTs), normalizing lexical details but preserving control flow and logic.
Calibration: The final weight is the harmonic mean of reliability and compatibility, ensuring that evidence is only trusted if it is both historically consistent and structurally relevant.

3. Role-Conditioned Search Intent Distillation

Instead of feeding raw evidence to the proposer, DREvo distills calibrated evidence into an explicit search intent ( $s_t$ ) that specifies what to do and where to look.

Evidence Partitioning: Retrieved evidence is categorized into three sets based on weight and polarity: Reliable Positive ( $X_t$ ), Reliable Negative ( $A_t$ ), and Low-Confidence ( $T_t$ ).
Role Activation: Based on these sets, a search role is activated:
- Exploit: If reliable positive evidence exists, the proposer is guided to inherit or refine these modifications.
- Avoid: If reliable negative evidence exists, the proposer is guided to avoid specific harmful modifications.
- Retest: If only low-confidence evidence exists, the proposer is guided to re-evaluate these directions under the current state.
- Explore: If no relevant evidence exists or previous retests failed, the proposer is allowed to freely explore new directions based on current failure patterns.
Distillation: The activated role and current failure patterns are distilled into a structured prompt ( $s_t$ ) that explicitly directs the proposer to a target component and modification direction.

Key Contributions

Problem Identification: The paper identifies performance fluctuation in harness self-evolution as a reliability challenge rooted in the failure to dynamically reassess historical validity and the lack of explicit search policies.
DREvo Framework: A novel method that anchors evidence to functions, recalibrates validity based on state compatibility and historical consistency, and distills this into role-conditioned search intents.
Empirical Validation: Extensive experiments demonstrating smoother evolution trajectories and superior performance across diverse benchmarks.

Experimental Results

The authors evaluated DREvo on five benchmarks: three Domain Reasoning tasks (USPTO, Symptom2Disease, Law) and two Agentic Tasks (Terminal-Bench 2.0, SWE-Bench Verified).

Overall Performance: DREvo achieved the highest accuracy on all five benchmarks under limited evolution budgets (20 iterations for reasoning, 10 for agentic tasks).
- Domain Reasoning: Average gains of 16.2% over baselines. Specifically, it improved USPTO by 9.0%, S2D by 2.4%, and Law by 4.0% over the second-best methods.
- Agentic Tasks: Average gains of 13.8% over baselines. Notably, on SWE-Bench Verified, DREvo evolved a terminal-oriented initial harness into a competitive code-repair harness without manual task-specific engineering.
Evolution Trajectory: Unlike baselines (e.g., Meta-harness) which showed large accuracy oscillations and context bloat, DREvo exhibited smoother improvement curves and maintained a bounded context size (~2.9K tokens).
Robustness: Under controlled component-state drift (simulating changes in the harness code), DREvo improved performance in 11 out of 16 settings, whereas "blind reuse" of evidence (without recalibration) decreased performance in most cases.
Ablation Studies: Removing any of the three components (Anchoring, Recalibration, Distillation) resulted in performance drops, confirming their complementary roles.

Significance and Claims

The paper claims that DREvo addresses the fundamental instability of history-driven search in harness self-evolution. By moving from static, implicit reuse of history to dynamic, explicit, and state-aware guidance, DREvo enables the reliable discovery of high-performing harnesses within limited computational budgets. The authors emphasize that their gains stem from the quality of search guidance (recalibration and distillation) rather than simply exposing the proposer to more historical data or context compression. The work suggests that future harness evolution requires mechanisms to continuously validate the applicability of past experiences as the system state evolves.