Technical Summary: Stop Comparing LLM Agents Without Disclosing the Harness

1. Problem Statement

Current evaluation practices for Large Language Model (LLM) agents on long-horizon tasks systematically conflate model capability with the performance of the execution harness. The execution harness is the infrastructure layer responsible for context construction, tool interaction mediation, output verification, orchestration (retry/escalation logic), and stopping criteria.

Standard leaderboards (e.g., SWE-bench, Terminal-Bench, AgentBench) report a single score per {model, benchmark} pair, implicitly attributing the result solely to the model. This paper argues that for long-horizon tasks involving comparable frontier models, the harness is often the binding constraint on performance. Consequently, current evaluation protocols misattribute harness-level gains to model improvements, rendering cross-model comparisons unreliable and potentially misleading. The conflation arises because:

1. Observational Evidence: Holding a model fixed while changing only the harness infrastructure can produce performance shifts (e.g., 15 percentage points) that dwarf reported model improvements (typically 2–4 points).
2. Structural Reality: Agents operate as closed-loop dynamical systems where the harness acts as the controller and the LLM as a stochastic policy. Long-horizon reliability depends on controller properties (stability, context drift management, control lag) rather than just the policy's raw capability.

2. Methodology and Theoretical Framework

The Binding Constraint Thesis

The paper formalizes the Binding Constraint Thesis: In the regime of long-horizon tasks with comparable frontier models, the variance in benchmark scores induced by the harness ($HV$) is often comparable to or larger than the variance induced by the model choice ($MV$).

The total performance variance is decomposed as:
$$\text{Var}(B(M, H)) = \underbrace{\text{Var}_M[E_H[B(M, H)]]}_{MV} + \underbrace{\text{Var}_H[E_M[B(M, H)]]}_{HV} + \underbrace{\text{Var}(\text{interaction})}_{\text{Interaction}}$$
The thesis asserts that $HV$ and the interaction term dominate $MV$ in current evaluations.

Control-Theoretic Formalization

The authors reframe agent execution from a simple inference loop to a closed-loop dynamical system:

• The LLM ($\pi_\theta$): An open-loop stochastic policy that receives a context projection $c(s_t)$ but has no direct access to the full state $s_t$.
• The Harness ($C_H$): The controller that updates the state, filters observations, and manages the feedback loop.
• Reliability Quantities: The harness determines three critical structural quantities:
  1. Stability: Whether the system makes non-decreasing expected progress toward the goal (Lyapunov-like measure).
  2. Context Drift ($\delta_t$): The rate at which task-relevant information is lost from the context window.
  3. Control Lag ($\tau$): The delay between anomaly detection and the arrival of a corrective signal at the policy.

Empirical Approach

The paper supports the thesis through two lines of evidence:

1. Observational Data: Analysis of public leaderboards (SWE-bench Pro, Terminal-Bench, HAL) showing that "scaffold-only" variations (changing the harness while fixing the model) result in double-digit performance swings and ranking reversals.
2. Controlled Factorial Experiment: A 3x3 grid experiment varying three frontier models (GPT-5.4, Kimi K2.6, GLM-5.1) against three distinct harness configurations ($H_1$: Minimal/Open-loop, $H_2$: Improved/Basic closed-loop, $H_3$: Full/Advanced closed-loop) on a stratified subset of SWE-bench Verified.

3. Key Results

Observational Evidence

• SWE-bench Pro: Under a standardized scaffold, top models span 4.9 points. However, fixing the model (Claude Opus 4.5) and switching from the SEAL scaffold to Claude Code increases the score by 9.5 points.
• Ranking Reversals: Adding a single search subagent (WarpGrep) to identical infrastructure flipped the ranking between MiniMax 2.5 and Claude Opus 4.6.
• Cross-Scaffold Gaps: On SWE-bench Verified Mini, the same model (e.g., Claude Sonnet 4.5) showed a 34-point gap between different scaffolds (SWE-Agent vs. HAL Generalist).

Controlled Factorial Experiment Results

• Variance Dominance: The harness-induced variance ($HV$) was 7.80 times larger than the model-induced variance ($MV$).
  • Average $HV$: 18.48 pp²
  • Average $MV$: 2.37 pp²
• Performance Shifts: Changing the harness moved scores by 8.5 to 13.0 percentage points for fixed models, whereas changing the model within a fixed harness moved scores by only 2.5 to 5.0 points.
• Ranking Instability: 6 out of 9 possible model-pair/harness-pair comparisons resulted in ranking reversals.
• Mechanism Analysis:
  • $H_1 \to H_2$ gains were driven by reduced control noise (better formatting, retrieval, error handling).
  • $H_2 \to H_3$ gains were driven by closing the verification and recovery loop (self-checking, anomaly detection, rollback).

4. Proposed Framework: Harness-Aware Evaluation

The paper proposes a new evaluation regime requiring three components:

1. Harness Card (Structured Disclosure): Every benchmark submission must include a disclosure organized by the ETCSOVG taxonomy:

   • Execution (runtime, sandboxing, timeouts)
   • Tool (list, schema, error contracts)
   • Context (window cap, compression, retrieval, memory)
   • Scheduling (loop, retry, escalation, rollback)
   • Observability (logging, traces)
   • Verification (parsing, self-checking, anomaly detection)
   • Governance (permissions, side-effects)

2. Variance Decomposition Protocol: Evaluations should employ either:

   • Locked-Harness Protocol: A single, specified harness applied to all models for clean ranking.
   • Factorial Protocol: Varying both models and harnesses to report $HV$, $MV$, interaction terms, and ranking reversal counts.

3. Trajectory-Level Metrics: To attribute performance shifts to specific harness layers, the paper proposes:

   • Recovery Rate (RR): Probability of returning to a task-advancing state after an anomaly within $k$ steps (measures Stability).
   • Context Retention: Fraction of task-relevant files/constraints present in the context (measures Drift).
   • Control Lag ($\tau$): Steps between anomaly detection and corrective signal arrival.

5. Significance and Claims

The paper claims that benchmark scores for long-horizon agents are not valid for cross-model comparison unless the execution harness is disclosed.

• Scientific Accuracy: Current leaderboards measure a mixture of model and harness, leading to systematic misattribution of gains. This distorts research incentives, directing investment toward model scaling while under-investing in harness engineering, which is the larger source of variance.
• Practical Impact: Deployment decisions based on current rankings are flawed because they do not isolate the model's contribution. The "binding constraint" is often the infrastructure, not the model weights.
• Call to Action: The field must shift from treating the harness as a hidden variable to treating it as a primary experimental condition. Until harness specifications are disclosed and variance decomposition becomes routine, leaderboard comparisons should be treated as incomplete.

The paper explicitly rejects the notion that "better models will solve this," noting that as models improve, harnesses become more complex (e.g., Claude Code redesigning components rather than removing them) to manage the increased capability. The solution is structural disclosure, not just standardization or scaling.