Technical Summary: Evo-Bench – Can Language Models Improve Agent Harness?

Problem Statement

While Large Language Models (LLMs) have advanced autonomous agents, current evaluations are largely confined to static task solving within fixed, human-engineered "harnesses" (the code structures that orchestrate reasoning, tool use, and memory). A critical frontier is harness evolution: the capacity of an agent to autonomously optimize its own operating harness. However, systematically benchmarking this capability is hindered by three specific challenges:

1. Harness Sensitivity: Standard benchmarks often fail to isolate harness improvements from the underlying base model's strength, making it difficult to determine if performance gains stem from better code or a smarter model.
2. Cross-Split Generalization: Existing evaluations risk task-specific overfitting, where validation and evaluation splits do not exhibit aligned responsiveness to harness variations.
3. Long-Horizon Evolution: Capturing the ability to sustain multi-round iterative refinement (diagnosing failures, hypothesizing improvements, and updating code) remains difficult.

Current literature lacks a controlled environment to evaluate the intrinsic capacity of foundation models to act as long-horizon harness engineers across heterogeneous domains.

Methodology: Evo-Bench

To address these challenges, the authors introduce Evo-Bench, the first benchmark designed to evaluate models' intrinsic harness-evolving capabilities. The framework employs a novel harness-guided construction process and a controlled evaluation protocol.

1. Benchmark Construction Framework

The construction of Evo-Bench follows a two-stage process to ensure tasks are genuinely sensitive to harness improvements:

• Stage 1: Auxiliary Harness Generation: The authors collect an auxiliary task set (320 tasks) disjoint from the final benchmarks. They run independent evolution experiments on these tasks using four frontier models (GLM-5.2, Claude-Opus-4.8, Claude-Sonnet-5, GPT-5.6-Sol) to generate a diverse set of 12 representative evolved harnesses ($H_{aux}$).
• Stage 2: Harness-Guided Task Selection: Candidate tasks from five established benchmarks (BrowseComp, HLE, GDPval, APEX-Agents, Claw-Eval) are evaluated against $H_{aux}$. Tasks are filtered based on Harness Sensitivity ($Sens(x)$), defined as the Pearson correlation between a task's score and the overall quality of the harnesses. Only tasks with positive sensitivity and appropriate difficulty are selected.
• Stratified Splitting: The final dataset is split into a visible Validation Suite ($D_{val}$, 160 tasks) and a disjoint Evaluation Suite ($D_{eval}$, 448 tasks) using sensitivity-aware stratified splitting to ensure distributional consistency and prevent overfitting.

2. Evaluation Protocol

• Domains: The benchmark covers three domains: Search (web navigation), Office (document processing), and General (diverse agent tasks).
• Process: A fixed Policy Model ($\pi$) executes tasks using a mutable Policy Harness ($H_t$). An Evolver ($E$), operating via a fixed "evolve harness," iteratively inspects failure patterns, formulates hypotheses, and edits $H_t$.
• Constraints: The process is bounded by a resource budget (iterations, time, steps). The evolver can only access $D_{val}$ during the process; $D_{eval}$ is reserved for the final assessment of the frozen harness ($H_T$).
• Metrics:
  • Overall Score: Performance on the held-out $D_{eval}$.
  • Anytime Validation Score: The average of the best-so-far validation performance over the evolution budget, measuring the efficiency of the evolutionary process.

Key Contributions

1. Evo-Bench: A novel benchmark evaluating the intrinsic ability of LLMs to autonomously refine executable code harnesses across Search, Office, and General domains, moving beyond static task solving.
2. Harness-Guided Construction Framework: A methodology that uses auxiliary-task evolution to identify harness-sensitive tasks and employs sensitivity-aware stratified splitting to ensure robust cross-suite generalization.
3. Systematic Analysis of Harness Evolution: A comprehensive evaluation across nine frontier and open-weight models, characterizing performance gains, temporal anomalies (e.g., early saturation), and the transferability of evolved structures.

Experimental Results

The authors evaluated nine models (including GPT-5.6-Sol, Claude Opus 4.8, GLM-5.2, Qwen3.7-Max, and others) against a CodeAct seed harness and a human-engineered "Artificial Harness" baseline.

• Performance Gains: Top models achieved massive absolute gains. GPT-5.6-Sol and Claude Opus 4.8 improved the Overall Score by 16.6 and 16.1 points respectively, reaching 46.3 and 45.8. These scores closely approach the human-engineered baseline of 47.5.
• Domain-Specific Variance:
  • Search: Evolution was highly effective; Claude Opus 4.8 gained +34.8 points, nearly matching the human baseline. Models successfully synthesized missing web-navigation logic.
  • General: Top evolvers (e.g., GPT-5.6-Sol, Qwen3.7-Max) surpassed the human-engineered baseline, demonstrating that autonomous reasoning structures can outstrip manual designs in this domain.
  • Office: Models struggled significantly, showing marginal improvements or slight regressions. The authors attribute this to the difficulty of discovering highly specific processing workflows required for office tasks.
• Temporal Anomalies: Analysis revealed early saturation, where models rapidly discover high-quality structures but often introduce detrimental modifications in later rounds, leading to premature termination or performance dips.
• Transferability: Evolved harnesses demonstrated robust cross-policy generalization. When applied to different policy models (Qwen, DeepSeek, GLM), the synthesized harnesses consistently boosted performance, indicating they act as transferable reasoning structures rather than overfitting to a single model's flaws.
• Cost Efficiency: A Pareto frontier analysis showed a steep logarithmic trade-off between cost and performance. While top-tier results required substantial resources (e.g., >$500 for GPT-5.6-Sol), models like GLM-5.2 and Qwen3.7-Max achieved strong results for under $40.

Significance and Claims

The paper claims that Evo-Bench provides a rigorous, controlled framework to disentangle genuine harness-evolving capability from base model strength and stochastic noise. The results suggest that while current frontier models possess the capacity to act as autonomous research engineers—particularly in Search and General domains—they still lack the deep system-level optimization and persistence of human engineers, especially in complex Office workflows.

The authors position Evo-Bench as a foundational testbed to catalyze future research into self-evolving agent architectures. They argue that by strengthening the capacity to abstract causal failure patterns and execute large-scale architectural refactoring, future models hold the potential to consistently surpass human experts in harness engineering. The paper does not claim to have solved self-improvement but rather provides the first systematic metric to measure and benchmark this specific evolutionary capability.