Technical Summary: Scaling Harness Intelligence via Just-in-Time Harness Evolution

1. Problem Statement

The capability of an LLM agent is not solely determined by the foundation model's weights but is jointly defined by the agent harness—the operational scaffold comprising memory management, planning strategies, action protocols, and tool/skill orchestration. While a strong model can fail under an inappropriate harness, a robust harness can only unlock potential if the model can interpret and follow it.

Current approaches to harness optimization predominantly rely on Ahead-of-Time (AOT) methods. These systems treat the harness as a durable artifact optimized over an experience stream to generalize across future tasks. However, AOT approaches struggle with instance-dependency: different tasks (e.g., wide-search vs. terminal tasks vs. deep research) require fundamentally different harness priors. Optimizing a single AOT harness across heterogeneous demands is cumbersome, requiring large design spaces and trajectory accumulation, often failing to match the specific structure of a new problem. The paper posits that harness design should shift from a manual, static engineering task to a Just-in-Time (JIT) capability, where a trained meta-agent synthesizes a task-specific harness on the fly.

2. Methodology: JIT-Agent

The authors propose JIT-Agent, a compact meta-agent designed to synthesize, repair, and evolve task-adaptive harnesses for arbitrary off-the-shelf agentic LLMs.

2.1 Formalization: The Four-Module Protocol

To make harness generation feasible and structured, the paper formalizes the agent harness as a composable, machine-generatable artifact governed by a fixed four-module protocol $\Pi = (M, P, A, F)$:

1. Memory ($M$): Compresses history into a view ($v_t$) of realized history.
2. Planning ($P$): Converts the view into a local directive ($d_t$).
3. Capability Orchestration ($F$): Selects and sequences relevant tools/skills ($C_t$) based on the directive.
4. Action ($A$): Consumes the assembled context to update the controller state and emit the next action ($e_t$).

This factorization turns heterogeneous programs into comparable coordinates within a protocol-compliant space ($H_\Pi$). The authors introduce HarnessFactory, a unified codebase implementing 13 representative contemporary scaffolds (e.g., ReAct, Plan-and-Execute, Recursive agents) under this common interface to serve as a seed bank ($B_0$).

2.2 Training Pipeline

JIT-Agent is trained via a three-stage pipeline to achieve Harness Intelligence (adaptivity, reliability, and evolvability):

• Stage I: Task-Conditioned Customization

  • Goal: Teach the model to map a task specification and a small reference context to a protocol-compliant harness.
  • Method: A frozen, stronger teacher model synthesizes task-adapted harnesses. The training data includes both supervised fine-tuning (SFT) on valid generations and preference learning. The preference objective ($L_{pref}$) optimizes for harnesses that improve task reward without degrading efficiency (latency or cost), using a Pareto-style preference rule.

• Stage II: Repairing Harnesses

  • Goal: Ensure reliability by learning to recover from synthesis failures.
  • Method: Failed generations (compiler errors, interface mismatches, runtime exceptions) are converted into bounded repair trajectories. A teacher proposes structured revisions ($\Delta$) to fix the harness. The model is trained to imitate these repair transitions, learning to use execution feedback to restore protocol-valid execution within a short horizon (max 2 rounds).

• Stage III: Evolutionary Group-Decoupled Policy Optimization (Evo-GDPO)

  • Goal: Enable online evolution where the model learns to propose harnesses that surpass the current archive frontier.
  • Method: During training, the model proposes a group of candidate harnesses. These are evaluated against an incumbent (the best harness in the current bank for that task). The reward signal is decoupled into three channels:
    1. Reward: Primary signal; candidates must match or exceed the incumbent's reward.
    2. Latency & Cost: Efficiency signals activated only if reward is preserved.
  • The policy is updated using a PPO-style objective driven by a normalized, aggregated advantage that rewards overtaking the frontier while maintaining efficiency. The harness bank ($B_n$) is updated conservatively, retaining only designs that advance the frontier.

2.3 Inference Architecture

JIT-Agent supports two inference modes:

• Static Inference: Generates $N$ harnesses in parallel, selects one, and executes it. Useful for maximizing candidate diversity without increasing environment rollouts.
• Streaming Inference: Designed for task streams. The model retrieves references from an evolving bank, generates a harness, executes it, and updates the bank based on feedback. This allows experience to transfer across tasks without updating model parameters during deployment.

3. Key Results

The paper evaluates JIT-Agent across nine benchmarks covering deep research, daily work, planning, and workspace tasks, using backbones including GLM-5.2, DeepSeek-V4-Flash, Qwen3.6, and Mimo-V2.5.

• Performance Gains: Replacing default scaffolds with JIT-generated harnesses yields consistent improvements.

  • GLM-5.2: Average improvement of +7.7 points across nine benchmarks. Notable gains include +20.2 on DeepPlanning-Travel and +4.3 on OdysseyBench.
  • DeepSeek-V4-Flash: Average improvement of +8.8 points. It surpasses GPT-5.6 on DeepSearchQA (+9.1).
  • Generalization: JIT-Agent improves performance across all tested model families (DeepSeek, Qwen, Mimo) and variants (Flash/Pro), indicating the capability transfers across model weights.

• Competitiveness with Fixed Harnesses:

  • JIT-generated harnesses are performance-competitive with mature runtimes like OpenCode and Claude Code.
  • On DeepSeek-V4-Flash, JIT-Agent outperforms the strongest fixed harness (NanoBot) by +4.7 points on DeepSearchQA and +4.0 points on xBench-DS.

• Cost-Efficiency:

  • JIT-Agent achieves the lowest token consumption and API cost in all controlled settings.
  • Compared to the cheapest fixed harness, it reduces per-case costs by 14.9–54.1% (average 36.0%) while improving performance. For example, on DeepSeek-V4-Flash xBench-DS, token usage dropped from 527K to 212K while performance rose from 78.0 to 82.0.
  • Pareto frontier analysis shows JIT-Agent shifts operating points toward the upper-left (higher performance, lower cost), proving gains are not due to longer trajectories but better orchestration.

• Test-Time Evolution:

  • Streaming JIT (which updates the harness bank over a task stream) consistently outperforms Static JIT (independent generations) in cumulative accuracy, demonstrating the value of learning from execution feedback.

4. Significance and Claims

The paper claims to establish Harness Intelligence as a new, trainable, and transferable dimension of agent capability that is orthogonal to model scaling.

• Paradigm Shift: The work moves harness engineering from "Ahead-of-Time" (optimizing a static artifact) to "Just-in-Time" (synthesizing a task-specific scaffold on the fly).
• Model-as-a-Harness: It demonstrates that a trained meta-agent can generate operational scaffolds that allow weaker or more efficient backbones to rival or surpass stronger models with fixed harnesses.
• Scalability: By formalizing the harness as a composable protocol and training a generator to evolve it, the paper suggests a path to scaling agent capability through the optimization of the execution environment itself, rather than solely through model parameter scaling.
• Future Direction: The authors view this as a step toward Model-Harness Co-Design, where foundation models may eventually internalize the ability to construct, revise, and evolve their own execution systems.

The paper concludes that harness intelligence is not merely an implementation detail but a first-order determinant of agent performance, capable of recovering substantial capability that would otherwise require expensive backbone scaling.