Technical Summary: Self-Harness

Problem Statement

The performance of Large Language Model (LLM)-based agents is determined not only by their base model weights but also by their harness—the surrounding system layer comprising system prompts, tools, memory, verification rules, and runtime policies. While human experts currently engineer these harnesses, this paradigm faces scalability issues as LLMs become increasingly diverse and rapidly evolving. A harness optimized for one model often proves suboptimal for another due to distinct behavioral patterns, tool-use habits, and error modes. Existing automated approaches often rely on stronger external agents to guide improvements, which is costly and potentially mismatched to the target model's specific failure modes. The paper addresses the challenge of enabling an LLM-based agent to improve its own operating harness without human engineering or external guidance.

Methodology: The Self-Harness Loop

The authors propose Self-Harness, an iterative, self-improvement framework where a fixed model refines its own non-parametric scaffolding. The process operates as a closed loop consisting of three distinct stages:

1. Weakness Mining

The agent, operating under the current harness ($h_t$), executes a set of tasks from a held-in split ($D_{in}$). The system collects execution traces and clusters failed traces based on verifier-grounded failure signatures.

• Attribution: Instead of treating failures as isolated events, the system analyzes the trace to identify the terminal verifier-level cause ($c_i$), the causal status of the agent behavior ($q_i$), and the abstract agent mechanism ($m_i$).
• Clustering: Failures are grouped deterministically if they share the same signature, ensuring that the subsequent proposal targets reusable failure mechanisms rather than superficial symptoms.
• Output: An evidence bundle ($B_t$) summarizing dominant failure patterns, ordered by support and actionability.

2. Harness Proposal

The same fixed model ($M$) is invoked in a "proposer" role, using the current harness ($h_t$) and the evidence bundle ($B_t$) as context.

• Constraints: The proposer generates $K$ diverse yet minimal candidate edits ($\Delta_j$). Each edit must target a specific failure mechanism and map to a concrete, editable surface of the harness (e.g., specific instructions, tool policies, or verification rules).
• Diversity vs. Minimality: The system encourages exploring different failure mechanisms or harness surfaces across proposals, but enforces that individual edits remain targeted, avoiding broad rewrites of the control architecture.
• Output: A set of candidate harness variants $h^{(j)}_t = \Delta_j(h_t)$, each accompanied by an audit record.

3. Proposal Validation

Candidate edits undergo rigorous regression testing before acceptance.

• Evaluation: Both the current harness ($h_t$) and candidate harnesses ($h^{(j)}_t$) are evaluated on the held-in split ($D_{in}$) and a held-out split ($D_{ho}$).
• Acceptance Rule: A candidate is promoted only if it improves performance on at least one split without degrading the other ($\Delta_{in} \ge 0$, $\Delta_{ho} \ge 0$, and $\max(\Delta_{in}, \Delta_{ho}) > 0$).
• Merging: If multiple candidates pass, their edits are merged into the next iteration's harness ($h_{t+1}$). Rejected candidates are logged but do not alter the active harness.

Key Contributions

1. Self-Harness Paradigm: A novel framework enabling an LLM to design and refine its own operating harness, eliminating dependence on human engineers or stronger external agents.
2. Operationalized Iterative Loop: A concrete implementation that transforms behavioral evidence into model-specific harness updates through a cycle of weakness mining, minimal proposal generation, and strict regression testing.
3. Empirical Validation: Demonstration that Self-Harness can evolve a minimal initial harness into model-specific configurations that significantly outperform the baseline across diverse model families.

Experimental Results

The authors evaluated Self-Harness on Terminal-Bench-2.0 using three distinct base models: MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5. All experiments started from a minimal DeepAgent-based harness.

Performance Gains (Held-Out Pass Rates):

• MiniMax M2.5: Increased from 40.5% to 61.9% (+21.4 percentage points).
• Qwen3.5-35B-A3B: Increased from 23.8% to 38.1% (+14.3 percentage points; a 138% relative improvement).
• GLM-5: Increased from 42.9% to 57.1% (+14.2 percentage points).

Qualitative Findings:

• Model-Specific Adaptation: The system did not simply add generic instructions. Instead, it produced distinct, concrete changes tailored to each model's weaknesses:
  • MiniMax M2.5: Focused on creating required output files earlier, handling structured tool outputs carefully, and stopping unproductive tool-use loops.
  • Qwen3.5: Focused on pre-checking dependencies, avoiding repeated failed commands, breaking exploration cycles, and recovering from tool errors.
  • GLM-5: Focused on preserving environment settings across shell commands and accelerating the transition from exploration to implementation.
• Generalization: Improvements were observed on held-out tasks, indicating that the harness edits addressed reusable execution mechanisms rather than overfitting to specific training failures.

Significance and Claims

The paper claims that Self-Harness demonstrates a path toward self-creation in agent systems, where the agent participates in reshaping the very constraints that govern its behavior.

• Empirical State Transition: The authors posit that harness improvement should be treated as an empirical state transition. A useful edit must be grounded in behavioral evidence, specify the behavior it changes, and be validated by non-regressive evaluation.
• Scalability: By internalizing the improvement loop, Self-Harness offers a scalable alternative to human-centric engineering, capable of adapting to the rapid evolution and diversity of modern LLMs.
• Limitations: The authors modestly note that the current work studies bounded edits under fixed benchmarks. The protocol relies on the quality of verifier outcomes and trace records, and higher-stakes changes would require more robust acceptance gates than pass-rate non-regression alone.

The work concludes that while the agent does not change its own weights, it can effectively "create itself" by iteratively refining the harness that mediates its interaction with the environment.