Technical Summary: HarnessOpt-Bench

Problem Statement

As Large Language Models (LLMs) are increasingly deployed within agentic systems, their performance depends not only on model weights but also on the "harness"—the surrounding prompts, tools, control flow, memory, and orchestration code. While harness optimization (the iterative, evaluation-guided improvement of a harness) is a critical route to building capable AI systems, the community lacks a common protocol to measure how well frontier LLMs perform this task themselves.

Existing evaluations are often conflated by varying target agents, seeds, budgets, and scoring protocols, making it difficult to distinguish whether performance gains stem from the optimizer model, the coding harness it uses, or the specific task setup. Furthermore, harness optimization differs from standard coding benchmarks because evaluation is expensive and stochastic; it requires diagnosing failures from incomplete evidence, managing a limited evaluation budget, and separating real improvement from noise, rather than simply producing a single correct code step.

Methodology: HARNESSOPT-BENCH

The authors introduce HARNESSOPT-BENCH, a benchmark designed to evaluate end-to-end harness optimization under expensive and stochastic evaluation conditions. The framework enforces a controlled, reproducible environment with the following core components:

1. The Optimization Problem

• Candidate Harness ($H$): An executable codebase containing prompts, tools, and control flow. The optimizer can edit, add, or delete files subject to a fixed execution interface and immutable paths.
• Invariants ($\theta$): The models, environment, and verifier are fixed. The optimizer cannot change the target model or the evaluation logic, only the harness code.
• Budget ($B$): Search is constrained by a fixed budget of evaluation calls (capped at 100 per partition) and total target-model tokens. The optimizer must decide what to evaluate and at what fidelity.
• Disclosure Policy:
  • Development ($D_{dev}$): Reveals case inputs, per-case outcomes, and execution traces to support diagnosis.
  • Validation ($D_{val}$): Reveals only aggregate scores to support selection.
  • Test ($D_{test}$): Remains completely inaccessible during the search process.

2. Trusted Execution Environment

To prevent overfitting and ensure fairness, the benchmark runs within a trusted execution environment (building on VERO [23]):

• Isolation: Each target agent rollout occurs in an ephemeral, unprivileged sandbox.
• Access Control: The optimizer can write only the target agent's harness. It can read development traces and validation metrics but cannot access the test partition or modify the verifier.
• Audit: Every candidate version is preserved for audit, and resource use is metered.

3. Evaluation Metric

The primary metric is Normalized Gain ($g$), defined as the improvement of the nominated candidate over the pinned seed on the held-out test partition, normalized by the available headroom:
$$g = \frac{E_\theta(H^+) - E_\theta(H_0)}{1 - E_\theta(H_0)}$$
where $H_0$ is the seed harness and $E_\theta$ is the expected score. Negative values indicate the candidate performed worse than the seed.

4. Experimental Design

• Tasks: Four downstream tasks (OfficeQA, BrowseComp-Plus, Terminal-Bench, GAIA) with pinned seeds. Three seeds are competent but basic; one (GAIA) is a non-functional stub.
• Optimizers: Five frontier LLMs (Claude Opus-5, Claude Sonnet-5, GPT-5.6-Sol, GPT-5.6-Terra, Kimi-K3) were evaluated.
• Harness Configurations: Each model was tested under two conditions:
  1. Shared Harness: All models used a common coding harness (opencode).
  2. Native Harness: Models used their respective native coding tools (e.g., claude-code, codex, kimi-cli).
• Protocol: Each configuration ran twice. Results are averaged over $K=3$ independent evaluation rounds per test case to mitigate stochastic noise.

Key Contributions

1. A Controlled Benchmark: HARNESSOPT-BENCH decouples the optimizer model from the coding harness and the target task, allowing for the isolation of model capability in harness optimization.
2. Trusted Protocol: The introduction of a trusted execution boundary that enforces budgets, blocks access to held-out data, and preserves candidates for audit, ensuring that improvements reflect generalization rather than overfitting to visible scores.
3. Empirical Evaluation: A comprehensive evaluation of 5 frontier models across 4 tasks and 111 scored runs, comparing shared vs. native harnesses.
4. Analysis of Search Behavior: Instrumented search trajectories to correlate specific optimizer behaviors (e.g., breadth of intervention, trace reading) with final performance.

Results

1. Model vs. Harness Sensitivity

• Model Choice Dominates: The choice of optimizer model has a significantly larger effect on performance than the choice of coding harness. Changing the optimizer model moved the normalized gain by an average of 0.142, whereas changing the harness moved it by 0.079.
• No Consistent Native Advantage: Native harnesses (e.g., claude-code for Claude models) did not provide a consistent advantage over the shared harness. In 11 out of 20 model-task pairs, the shared harness outperformed the native one; in 9, the native won. The aggregate difference was near zero, suggesting that sensitivity to the harness is heterogeneous rather than uniform.

2. Capability Gaps and Progression

• Model Releases: The benchmark successfully distinguishes between successive model releases. On OfficeQA, GPT-5.6 releases showed monotonic gain increases, while Claude Opus releases showed a non-monotonic but significant spread.
• Task Variance: Gains vary substantially across tasks. The strongest configurations captured roughly two-thirds of the available headroom on OfficeQA but were unresolved from zero on BrowseComp-Plus and Terminal-Bench.

3. Search Behavior Insights

• Breadth of Intervention: There is a strong positive correlation ($\rho$ from +0.34 to +0.88) between the fraction of harness "levers" (e.g., prompt, retry policy, tool schema) touched during search and the final normalized gain. Broader exploration is associated with better outcomes.
• Trace Reading: Contrary to intuition, detailed inspection of failure traces was rarely used (only 16 times across 111 cells) and was negatively associated with gain. Optimizers relied primarily on per-case score summaries.
• Budget Constraints: The case-pass allowance (4 full passes per partition) was the binding constraint for most optimizers, not the evaluation call cap.
• Overfitting: Visible validation scores were often optimistic compared to the final held-out test scores, confirming the necessity of the held-out partition.

Significance and Claims

The paper establishes harness optimization as a measurable and discriminative capability for frontier LLMs. The authors claim that:

• Current models can perform harness optimization, but their performance is uneven and highly dependent on the specific task and seed regime.
• The capability to optimize an agent is distinct from the capability to write code; the former requires reasoning about stochastic systems and managing limited resources.
• HARNESSOPT-BENCH transforms harness engineering from an optimizer-specific demonstration into a reproducible evaluation target. This allows the community to measure and develop systems that can reliably make other agents better.

The authors remain modest in their claims, noting that the benchmark is "hack-resistant, not hack-proof" and that the results are specific to the current distribution of tasks and seeds. They emphasize that the next frontier is not merely better agents, but models that can reliably make agents better.