Technical Summary: From Question Answering to Task Completion: A Survey on Agent System and Harness Design

1. Problem Statement

The field of Large Language Model (LLM)-based agents has shifted from passive question answering (QA) to active, multi-step task completion. While frontier models have achieved saturation on static benchmarks (e.g., MMLU, GPQA), they exhibit significant reliability gaps in interactive, long-horizon agentic tasks (e.g., SWE-bench, WebArena). The central problem addressed by this survey is identifying the source of performance bottlenecks in agent systems. The authors argue that the bottleneck is no longer solely the foundation model's reasoning capability but increasingly resides in the execution harness—the runtime infrastructure that mediates the interaction between the model and the environment. The paper posits that agent quality (success, efficiency, safety, generalization) is an emergent property of the coupling between the model and its harness, rather than a function of the model alone.

2. Methodology and Framework

The survey employs a model–harness lens to analyze the evolution of agent engineering. It does not propose a new algorithm but synthesizes existing literature, benchmarks, and industrial reports through a specific conceptual framework:

• Functional vs. Implementation View: The paper distinguishes between the functional definition of an agent (a goal-directed closed-loop system performing perception, reasoning, action, and adaptation) and the implementation view (an LLM coupled with an execution harness).
• Formal Decomposition: The execution harness ($H$) is formalized as a tuple of six coupled runtime responsibilities:
  1. Observation Interface ($I_{obs}$): Transforming raw environment signals (logs, DOMs, diffs) into model-usable observations.
  2. Context Manager ($C$): Curating, compressing, and refreshing information entering the model's context window.
  3. Control Loop ($L$): Orchestrating the observe-reason-act-feedback cycle, including step scheduling, delegation, and stopping criteria.
  4. Action Interface ($I_{act}$): Mapping model outputs to executable operations (APIs, shell commands, tool calls).
  5. State and Artifact Store ($S$): Persisting execution state, plans, checkpoints, and generated artifacts beyond the active context window.
  6. Verification and Governance ($V$): Checking outcomes, enforcing safety constraints, managing permissions, and handling recovery/rollback.
• Evolutionary Analysis: The literature is organized into four sequential paradigms of agent engineering, analyzing how the dominant performance lever shifted across stages:
  1. Prompt Engineering: Optimizing single-turn instructions (addressing the "expression" problem).
  2. Workflows and Context Engineering: Curating the information lifecycle and tool definitions (addressing the "information" problem).
  3. Harness Engineering: Designing the runtime infrastructure for feedback-driven execution, state persistence, and error recovery (addressing the "execution" problem).
  4. Agent-Native Training and Co-Evolution: Internalizing agentic behaviors (planning, verification) into model parameters and evolving the model-harness pair jointly.

3. Key Contributions

The survey makes three primary distinctions from prior work:

1. Evolution-First Synthesis: Instead of a static component taxonomy, it traces the migration of engineering bottlenecks from prompt optimization to runtime system design and future co-evolution.
2. Harness-Centric Perspective: It treats the execution harness as a first-class technical object, decomposing it into the six runtime components listed above. It argues that harness design determines reliability and efficiency even when the underlying model is fixed.
3. Empirical Evidence Integration: It connects academic benchmarks with industrial practices, using empirical data to demonstrate how runtime design choices (e.g., observation shaping, context management) directly impact agent performance.

Specific Contributions:

• Limits of Model-Centric Scaling: The paper analyzes how model scaling yields diminishing returns on agentic tasks due to resource-performance boundaries and measurement saturation on static benchmarks.
• Task-to-Harness Mapping: It maps task properties (horizon, environment type, autonomy level) to specific harness configurations. For example, software engineering tasks stress Verification, while web/GUI tasks stress Observation and Action interfaces.
• Benchmark Analysis: The survey provides a detailed empirical analysis of SWE-bench, Terminal-Bench, and WebArena, demonstrating that:
  • The same backbone model can achieve double-digit accuracy differences depending on the harness.
  • Minimal scaffolds (e.g., mini-SWE-agent) can sometimes match or approach the performance of complex, full-featured runtimes if the interface design is optimal.
  • Vendor-reported scores often exceed open-source results due to proprietary harness optimizations, highlighting the importance of the "scaffold."

4. Results and Findings

• Harness Impact: Empirical evidence from SWE-bench Verified shows that for a fixed model (e.g., Claude 3.5 Sonnet), resolution rates vary from ~33% to ~53% depending on the harness. Similarly, on Terminal-Bench 2.0, the same model (e.g., GPT-5.3-Codex) varies from 64.7% to 78.4% across different harnesses.
• Bottleneck Migration: As tasks become more complex (moving from L2 short-horizon to L4 open-ended), the bottleneck shifts from the model's reasoning to the harness's ability to manage state, verify outcomes, and recover from errors.
• Cost and Efficiency: Higher accuracy often correlates with higher resource costs (token usage, latency, timeouts). The paper notes that "success" is conditional on runtime configuration, including retry budgets and context policies.
• Multi-Model and Learnable Harnesses: Recent trends show a shift toward compositional runtimes (routing different models for planning, coding, and verification) and learnable harnesses where runtime policies are optimized as artifacts.

5. Significance and Claims

The paper claims that the field has moved beyond "prompt craft" into the study of full systems. Its significance lies in reframing agent development:

• Agent Quality is Emergent: Success, efficiency, and safety are not inherent to the model but emerge from the interaction between model capability, runtime infrastructure, task structure, and evaluation design.
• Necessity of Value-Aware Evaluation: The authors argue for a shift from score-centric leaderboards to value-aware evaluation, which jointly optimizes for task success, cost, latency, safety, and process quality. They propose a formal objective function that balances utility against constraints on cost, latency, and risk.
• Future Direction: The survey identifies the path toward Agent-Native Training, where verification and recovery behaviors are internalized into model parameters, and Co-Evolution, where the model and harness are updated jointly from execution traces. However, it modestly notes that the boundary between model design and system design is still being renegotiated, and reliability in realistic settings remains below deployment needs.

In conclusion, the survey argues that understanding the execution harness is now the central challenge in agent engineering, as it governs how model capabilities are translated into reliable, long-horizon task completion.