Technical Summary: Harness-R1

Problem Statement

Large Language Model (LLM) agents deployed in dynamic environments accumulate interaction trajectories containing both successful experiences and systematic failures (e.g., tool misuse, state loss, protocol violations). While current approaches to agent improvement focus primarily on updating the model's weights via fine-tuning or reinforcement learning, the "agent harness"—the executable runtime code that constructs context, mediates tools, validates actions, and manages recovery—typically remains static.

Modifying this harness is critical because identical model weights can yield vastly different capabilities under different harness configurations. However, direct harness modification is unreliable. Fixed rules (e.g., Self-Refine) often degrade performance, and prompting large frontier models to edit the harness results in unstable gains because these models optimize for textual plausibility rather than verified task success. The core challenge is to develop a method that learns to edit an existing executable runtime based on observed agent failures, ensuring that edits are validated by actual performance improvements rather than theoretical reasoning.

Methodology: Harness-R1

Harness-R1 introduces a training paradigm that treats harness editing as an online reinforcement learning (RL) problem for a dedicated "harness engineer" model, while keeping the target agent frozen. The system operates on a failure-to-edit-to-rerun loop:

1. Problem Formulation

• Frozen Target: The target agent $A$ (model + base runtime) remains fixed.
• Failure Packets: A batch of tasks $B$ is run, and a deterministic extractor compiles failed episodes into "failure packets" ($s_B$) containing task constraints, action-observation excerpts, and outcomes.
• The Engineer ($H_\theta$): A dedicated 9B model acts as the editor. It reads a failure packet and generates an executable overlay (patch) $P$.
• Executable Hooks: The patch $P$ wraps the agent's lifecycle at four specific points without altering the model weights:
  1. Episode Initialization: Sets up context and state.
  2. Pre-Decision: Augments context with guidance/constraints.
  3. Pre-Action: Validates, rewrites, or vetoes proposed actions.
  4. Post-Feedback: Inspects observations and triggers recovery.
• Reward Signal: The patched target reruns the same batch of tasks. The reward is the difference in performance ($\Delta_B$) between the patched and unpatched runs. This creates a transductive, same-batch objective where the reward is non-differentiable and observable only after execution.

2. Training Pipeline

Harness-R1 employs a two-stage training process:

• Cold-Start Supervised Fine-Tuning (SFT): A strong teacher model proposes editing responses from failure packets. These are validated against the frozen target; only executable, non-regressive patches are retained to form a dataset for initializing the engineer's policy.
• Outcome-Grounded Group Relative Policy Optimization (GRPO):
  • The engineer samples $K$ candidate patches for a failure packet.
  • Each candidate is validated and installed; the frozen target reruns the task batch.
  • Rewards are normalized within the group of $K$ candidates to compute advantages.
  • The engineer's parameters are updated to maximize the token-averaged clipped surrogate objective, optimizing for the realized utility of the patches.

Key Contributions

1. Formulation: The paper formulates failure-conditioned, lifecycle-wide harness editing as an online RL problem for a dedicated engineer, distinct from model weight updates.
2. Harness-R1 Framework: It proposes a system combining cold-start SFT with outcome-grounded GRPO. Unlike prior work that uses editors for program search or candidate selection, Harness-R1 directly updates the editor's policy based on the execution outcomes of its patches on a frozen target.
3. Co-evolution: The method demonstrates that a harness engineer and a target agent can co-evolve. An engineer trained on a vanilla target improves it, and a retrained engineer can further improve a target that has already undergone direct fine-tuning.

Experimental Results

The method was evaluated on three interactive benchmarks: WebShop (web navigation), ALFWorld (embodied text-based tasks), and DBBench (database querying).

• Performance Gains (Vanilla Target): For a frozen Qwen3.5-9B agent, Harness-R1 increased the average success rate from 44.3% to 53.6% (+9.3 percentage points). This outperformed prompt-based strategies (e.g., Self-Refine, which decreased performance) and frontier models prompted as editors (e.g., GLM-5.2 at 48.8%).
• Performance Gains (Fine-Tuned Target): After direct fine-tuning of the target agent (raising baseline to 59.2%), a target-specific Harness-R1 engineer further increased success to 64.2% (+5.0 points), demonstrating that harness optimization remains effective even after model weights are updated.
• Generalization:
  • Cross-Model: The learned editing policy transferred to 20 unseen target models (ranging from 1B to 70B parameters across Llama, Gemma, and Qwen families), yielding positive gains on every target without per-target retuning.
  • Cross-Task: Using sparse failure evidence (10 failures), the engineer generated patches that improved held-out tasks by +8.9 points, significantly outperforming large frontier models which showed negative or unstable results on held-out data.
• Ablation: Removing specific lifecycle hooks (e.g., pre-action mediation or post-feedback recovery) reduced performance, confirming that the gains are distributed across the agent's lifecycle and depend on the specific environment.

Significance and Claims

The paper posits that harness construction is a learnable capability that complements traditional weight updates. Its primary significance lies in shifting the optimization target from the agent's internal decision-making (weights) to the external runtime mechanisms (harness).

• Reliability over Plausibility: The authors argue that prompting large models to edit code is unreliable because it optimizes for "looking correct." Harness-R1 learns from realized outcomes, ensuring edits are genuinely beneficial.
• Adaptivity: Unlike fixed strategies (e.g., Self-Refine) that apply uniform rules, Harness-R1 generates edits conditioned on specific failure modes, adapting to the target agent's unique weaknesses.
• Co-evolution: The results suggest a path toward agents that continually improve after deployment not just by retraining the model, but by evolving the runtime environment that surrounds it. The paper concludes that this approach allows the engineer and the target to co-evolve, offering a robust path for post-deployment agent improvement.