💻 computer science

Rethinking the Evaluation of Harness Evolution for Agents

This paper critiques current evaluation protocols for automatic harness evolution in LLM agents by demonstrating that, when compared fairly against simple test-time scaling under matched budgets and on held-out tasks, harness evolution offers no consistent performance gains and exhibits limited generalization.

Original authors: Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, Teng Xiao

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, Teng Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant (an AI agent) that needs to solve tricky computer puzzles. To help it, you give it a "harness"—a fancy set of instructions, tools, and rules that tell the robot how to talk to the computer, what buttons to press, and how to check its work.

For a while, researchers thought the best way to make these robots better was to build a "robot mechanic" that would automatically tweak and upgrade the harness over and over again. This mechanic would watch the robot fail, think "Hmm, maybe if I change this rule," try a new harness, and repeat until it found the perfect setup. They called this Automatic Harness Evolution.

But in this new paper, a team of researchers decided to take a second look at whether this "robot mechanic" is actually doing anything special, or if it's just fooling us. They set up a fair race to see if the mechanic was truly designing better tools, or if it was just getting lucky by trying a lot of guesses.

The Big Race: Evolution vs. Just Trying Harder

The researchers set up a competition on a famous puzzle set called Terminal-Bench 2.1, which contains 89 different terminal tasks. They used three of the smartest AI models available: Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini.

They compared four different strategies, all given the exact same amount of "thinking time" (compute budget):

Parallel Sampling: Imagine asking the robot to try solving the puzzle 5 different times all at once, then picking the best answer. It's like rolling a die 5 times and hoping for a six.
Sequential Refinement: Imagine the robot tries once, sees where it messed up, fixes its thinking, and tries again. It's like revising an essay draft.
Harness Evolution: This is the "robot mechanic." It tries to rewrite the robot's instruction manual based on past failures, hoping to create a better tool for everyone.
Harness Scaling: This is like the mechanic rewriting the instructions just for this one specific puzzle while the robot is trying to solve it.

The Shocking Results

The paper found that the "robot mechanic" (Harness Evolution) did not consistently win the race. In fact, it often lost to the simpler methods.

When there were no "answer keys" (Unit Tests):
If the robot had to guess if it was right or wrong on its own, the "robot mechanic" actually made things worse.

The simple "try 5 times" method (Parallel Sampling) boosted the average score from 68.2 to 72.3.
The "robot mechanic" (Harness Evolution) only managed 67.4, which was actually lower than just using the original instructions without any changes!
The authors suggest that without a clear "answer key" to tell the robot what's right, the mechanic just started making noisy, confusing changes that hurt the robot's performance.

When there were "answer keys" (Unit Tests):
Even when the robot could check its work against a perfect solution, the mechanic still didn't shine.

The "try 5 times" method reached an average score of 86.0.
The "robot mechanic" only reached 75.8.
The authors noticed something weird: The mechanic's improvements only showed up when they were allowed to pick the best of 5 tries (pass@5). When they had to get it right on the very first try (pass@1), the mechanic barely improved anything. This suggests the mechanic wasn't actually designing a better tool; it was just helping the robot make more guesses.

The "Overfitting" Trap

The biggest surprise came when the researchers tested if the "robot mechanic" could learn a lesson that applied to new puzzles. They let the mechanic practice on 45 training tasks, then tested it on 34 brand-new tasks it had never seen before.

The result? The mechanic barely moved the needle.

On the new tasks, the score only went up by 0.6 points on average.
For one model (GPT-5.4), the score didn't change at all (72.1 to 72.1).

The paper suggests that the mechanic wasn't learning general rules for being a good robot. Instead, it was "memorizing shortcuts" for the specific puzzles it practiced on. It was like a student who memorizes the answers to a practice test but fails the real exam because they didn't actually learn the subject.

So, What's the Verdict?

The paper argues that we need to stop celebrating "Automatic Harness Evolution" as a magic bullet just because it scores higher on the same puzzles it practiced on.

What it rules out: It rules out the idea that these evolution methods are currently superior to simply trying harder (scaling up test-time computation).
What it suggests: It suggests that the gains we see might just be the result of the robot trying more times, not because the harness design got smarter.
The Takeaway: The authors propose that for harness evolution to be truly useful, we need to test it on puzzles that are hard enough to require a real upgrade in tools, and we must test it on new puzzles to make sure it's not just cheating by memorizing the old ones.

In short, the "robot mechanic" is still a work in progress. Right now, it seems like it's better at tweaking instructions for a single puzzle than at inventing a universal tool that works for everyone. The paper suggests we need fairer tests before we declare this method a winner.

Technical Summary: Rethinking the Evaluation of Harness Evolution for Agents

Problem Statement

The paper addresses critical flaws in the evaluation protocols for automatic harness evolution in Large Language Model (LLM) agents. A harness defines the prompts, tools, memory, verification routines, and control logic through which an agent interacts with an environment. While prior work has demonstrated that automatic methods can evolve these harnesses to improve agent performance, the authors argue that current evaluation practices are fundamentally flawed for two reasons:

Conflation with Test-Time Scaling: Existing methods use unit test feedback to iteratively search for better harness configurations and then evaluate the final result on the same benchmark. This setup fails to distinguish whether performance gains arise from an improved harness design or simply from the additional computational budget (search) allocated during the evaluation process. The authors posit that harness evolution must be compared against simple test-time scaling baselines (e.g., parallel sampling or sequential refinement) under matched feedback and inference budgets to determine if the gains are due to design or search alone.
Risk of Overfitting: Because the search and final evaluation often occur on the same set of public benchmark tasks, reported improvements may reflect adaptation to specific task patterns rather than the discovery of generalizable harness design principles.

Methodology

The authors propose a unified evaluation framework that compares four distinct approaches under a controlled compute budget ( $K$ ) and specific feedback conditions. The methods are categorized by what they update (trajectories vs. harness) and how they allocate resources:

Parallel Sampling: A test-time scaling baseline where $K$ independent trajectories are generated using a fixed harness. The final answer is selected via self-judgment (if no unit tests) or verifier selection (if unit tests are available).
Sequential Refinement: A test-time scaling baseline where $K$ trajectories are generated sequentially. Each attempt conditions on the previous trajectory (and potentially its outcome) to refine the solution, using a fixed harness.
Harness Evolution: The method under scrutiny. It optimizes the harness itself across a batch of tasks. A meta-agent analyzes accumulated trajectories and outcomes to propose a new harness ( $h_{k+1}$ ) for the next round, aiming to improve expected success on the task distribution.
Harness Scaling: A new baseline introduced by the authors to isolate the effect of instance-specific adaptation. Unlike evolution (which seeks a reusable design across tasks), harness scaling adapts the harness for a single evaluation instance by allocating additional budget to revise the harness based on that specific task's trajectory.

Experimental Setup:

Benchmark: Terminal-Bench 2.1 (a corrected version of Terminal-Bench 2.0 with 89 terminal tasks).
Models: Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini.
Settings:
- Without Unit Tests: Agents rely on self-judgment.
- With Unit Tests: Agents use unit tests for feedback and final selection (reporting both pass@1 and pass@5).
- Generalization: Harness evolution is trained on a disjoint set of tasks (45 training, 10 validation) and evaluated on held-out test tasks (34) to measure transferability.

Key Results

1. Performance vs. Test-Time Scaling

Across both settings (with and without unit tests), automatic harness evolution does not consistently outperform simple test-time scaling baselines.

Without Unit Tests: Harness evolution underperformed parallel sampling and direct sampling on average. Notably, on GPT-5.4, iterative harness revision actually degraded performance (from 75.3 to 69.7), suggesting that self-generated feedback is too noisy to reliably guide harness revision without external correctness signals.
With Unit Tests: While all methods improved over the baseline, harness evolution failed to surpass parallel sampling (pass@1) or sequential refinement (pass@5). The gains from harness evolution were largely realized only when selecting the best candidate from multiple attempts (pass@5), indicating that the benefit stems from increased sampling attempts rather than a fundamentally superior harness design.

2. Generalization and Overfitting

When evaluated on held-out tasks disjoint from the training set, harness evolution exhibited limited generalization.

The evolved harness provided only marginal gains (average +0.6 points) over the initial harness on the test set.
In contrast, the same method showed significant gains when evaluated on the training tasks. This discrepancy suggests that current evolution algorithms tend to overfit to the specific training instances, encoding task-specific shortcuts rather than transferable design principles.

3. Nature of Edits

Analysis of the meta-agent's modifications revealed that while edits were rational (e.g., adding behavioral rules, budget trackers, or syntax checks), they primarily memorized fixes for specific failure modes rather than distilling general strategies. These changes often addressed efficiency (e.g., reducing polling) but failed to resolve deep reasoning failures, which remain bottlenecks of the underlying model rather than the harness.

Key Contributions

Critique of Evaluation Protocols: The paper identifies that current evaluations of harness evolution conflate design improvements with test-time search, necessitating a comparison against matched test-time scaling baselines.
Unified Budget Framework: The authors formalize a comparison framework that explicitly controls for feedback types and inference budgets across parallel sampling, sequential refinement, harness evolution, and harness scaling.
Introduction of Harness Scaling: The paper introduces "harness scaling" as a baseline to distinguish between dataset-guided evolution (reusable design) and instance-guided adaptation.
Empirical Evidence of Limited Gains: Through extensive experiments on Terminal-Bench 2.1, the authors demonstrate that automatic harness evolution does not consistently outperform simpler search methods and fails to generalize to unseen tasks.

Significance and Claims

The paper concludes that the current evidence supporting automatic harness evolution should be interpreted with caution. The authors claim that:

Current gains are often illusory: Reported improvements may simply reflect the benefits of allocating more compute to search (test-time scaling) rather than the efficacy of the evolution algorithm itself.
Generalization is poor: Evolved harnesses do not transfer well to new tasks, suggesting they overfit to the training distribution.
Need for Fairer Protocols: Future research requires evaluation protocols that separate optimization feedback from final measurement and rigorously compare against test-time scaling baselines.
Task Dependency: The value of harness evolution may be task-dependent, potentially requiring benchmarks where tasks are difficult enough to leave headroom for improvement and where performance is heavily dependent on harness design (e.g., specialized tools/workflows) rather than just model reasoning.

The authors do not claim that harness evolution is useless, but rather that its effectiveness is currently overstated due to flawed evaluation methodologies and that its utility may be limited to specific, high-sensitivity task domains.