🤖 AI

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

Harness-Bench is a diagnostic benchmark comprising 106 realistic, sandboxed tasks that evaluates how different system-layer configurations (harnesses) impact LLM agent performance, revealing that execution outcomes vary significantly across model-harness pairings and highlighting the need to report agent capabilities at the configuration level rather than attributing them solely to the base model.

Original authors: Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang

Published 2026-05-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class chef (the AI Model). This chef can cook anything if given the right instructions. But in the real world, a chef doesn't just stand in a vacuum; they work in a kitchen with specific tools, a set of rules, a manager who checks the orders, and a system for handling mistakes.

This paper, Harness-Bench, is about testing that kitchen system (called the "harness"), not just the chef.

The Problem: We've Been Ignoring the Kitchen

Until now, when people tested AI agents, they mostly asked: "How good is the chef?" They looked at the final dish (did the AI finish the task?) but ignored how the kitchen was set up.

Did the chef have the right knives (tools)?
Did the chef know how to handle a burnt pan (recovery)?
Did the kitchen manager (the system) stop the chef from using the wrong ingredients (permissions)?

The authors argue that you can't judge an AI's ability just by looking at the model. You have to look at the Model + The Kitchen System together. A great chef in a messy, broken kitchen will fail, while a good chef in a perfect, well-organized kitchen might succeed.

The Solution: A "Kitchen Simulator" for AI

The researchers built Harness-Bench, a giant testing ground with 106 different cooking challenges (tasks). These aren't just simple questions; they are complex jobs like fixing code, analyzing financial data, or managing a project.

Here is how they tested it:

The Same Ingredients, Different Kitchens: They took the same 106 tasks and gave them to different AI models.
The Variable: They kept the task exactly the same but swapped out the "kitchen system" (the harness) for each run. Some kitchens were high-tech and strict; others were loose and simple.
The Result: They ran over 5,000 experiments.

What They Found

The results were surprising and clear: The kitchen matters just as much as the chef.

Huge Differences: When they used the same AI model but put it in different "kitchens," the success rate varied wildly. One kitchen system got a 76% success rate, while another got only 52% with the exact same model.
The "Smart" Kitchen Wins: The best-performing systems weren't just the ones with the smartest AI models. They were the ones where the system helped the AI stay on track, recover from mistakes, and use tools correctly.
The "Drift" Problem: The researchers found that AI often starts with a good plan but then "drifts off." It might think logically but forget to actually save the file, or it might ignore a tool error and keep trying the same thing. The best "kitchen systems" caught these drifts and fixed them.

The Big Takeaway

The paper concludes that we need to stop saying "This AI model is 90% good." Instead, we should say, "This AI model is 90% good when paired with this specific system."

If you want to build a reliable AI agent, you can't just pick the smartest brain; you have to build the best body and nervous system (the harness) to go with it. Harness-Bench is the tool they created to help engineers measure and improve that body.

In Short

Think of it like testing a race car. You can't just look at the engine (the AI model) and say it's fast. You have to test the engine on different tracks with different tires and drivers (the harness). Harness-Bench is the new track that proves the car's speed depends heavily on the track and tires, not just the engine.

Technical Summary: Harness-Bench

Problem Statement

Large Language Model (LLM) agents are increasingly deployed as executable systems that interact with external environments, utilizing tools, modifying workspaces, and producing concrete artifacts. In these workflows, practical performance depends not solely on the base model's reasoning capabilities but also critically on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery.

Existing benchmarks face a significant limitation: they typically abstract away execution details, conflate the harness with the full agent system, or hold the harness fixed while comparing models. Consequently, there is a lack of diagnostic protocols to study how specific model–harness configurations affect success rates, token costs, robustness, and traceability in realistic workflows. This obscures the execution-layer variation that often determines whether an agent succeeds or fails.

Methodology

The authors introduce Harness-Bench, a diagnostic benchmark designed to evaluate configuration-level harness effects under shared external conditions.

Benchmark Design

Task Suite: The benchmark consists of 106 sandboxed offline tasks constructed from practical agent-use patterns. These tasks span eight categories, including software engineering, data analysis, workspace operations, knowledge work, office communication, vertical professional workflows, long-running autonomy, and DevOps.
Validation: Each task is manually reviewed for four criteria: Realism (plausible user workflow), Solvability (completable with provided resources), Oracle-checkability (verifiable via deterministic checks or rubrics), and Integrity (preventing agents from bypassing constraints).
Evaluation Protocol: The benchmark fixes external task conditions (task prompt, initial sandbox state, budget, timeout, and evaluator) while varying the harness configuration and model backend.
- Unlike previous works that force systems into a common internal policy, Harness-Bench preserves each harness's native execution behavior (prompting, tool interfaces, state management, and recovery mechanisms).
- This approach treats the harness as the primary variable of interest, measuring the performance of the Model + Harness pairing rather than isolating individual mechanisms.

Execution and Scoring

Data Collection: Each run records final artifacts, execution traces, usage statistics, and validator outputs. The study analyzed 5,194 execution trajectories across 6 configurable harnesses and 8 model backends (plus a reference evaluation of Codex).
Scoring Metrics: The overall score is a multiplicative combination of three components:
1. Security ( $S$ ): A binary gate (0 or 1) for permission or security violations.
2. Completion ( $C$ ): Measured via task-specific deterministic validators or rubric-based judgment.
3. Process ( $P$ ): An aggregate of Robustness, Tool Use appropriateness, and Consistency, evaluated via LLM-based process rubrics.
- Formula: $TaskScore = S \cdot C \cdot P$ .
- This conservative scoring ensures high aggregate credit requires task completion, no security violations, and reliable execution behavior.

Key Contributions

Benchmark Asset: The release of Harness-Bench, a suite of 106 sandboxed offline tasks with task manifests, fixtures, evaluators, and execution traces designed for realistic end-to-end agent workflows.
Evaluation Protocol: A novel protocol that fixes external task conditions while preserving native harness behaviors, enabling configuration-level comparisons across representative harnesses and model backends.
Diagnostic Analysis: A large-scale analysis of 5,194 trajectories revealing substantial variation in completion, process quality, efficiency, and failure modes across different model–harness pairings.

Key Results

Configuration-Level Variation: Under the same task set and model-backend pool, aggregate scores varied significantly across harnesses. NanoBot achieved the highest score (76.2%), while OpenClaw achieved the lowest (52.4%), a gap of 23.8 points. This demonstrates that the harness choice is a critical determinant of performance.
Model-Harness Dependence: Stronger model backends tended to exhibit lower cross-harness variance, suggesting they are more tolerant of differences in prompting and state management. Conversely, weaker backends showed higher sensitivity to the execution substrate.
Efficiency: Higher-scoring harnesses often demonstrated better process profiles (tool use, consistency, robustness) and, in some cases, higher efficiency (fewer tokens/turns) compared to lower-scoring counterparts.
Failure Modes: Analysis of failed trajectories identified recurring "execution-alignment failures" where plausible reasoning became decoupled from tool feedback or verifiable outputs. The most common failure was Contract/Format violations (36.4%), followed by Tool/Recovery issues (24.6%) and Evidence/Grounding gaps (14.6%).

Significance and Claims

The paper argues that agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone. The results suggest that the execution layer is not merely an implementation detail but a fundamental component of agent performance that shapes how capability is exposed, constrained, and realized.

Harness-Bench provides a reproducible foundation for:

Diagnosing recurring execution-alignment failures where reasoning decouples from evidence or state.
Improving reliable, efficient, permission-aware, and auditable agent execution stacks.
Shifting the evaluation paradigm from "model-only" metrics to "system-level" diagnostics that account for the interplay between model reasoning and the execution environment.

The authors maintain a modest stance, noting that the benchmark evaluates complete harness configurations under controlled, sandboxed conditions. Therefore, the results are presented as diagnostic measurements under a fixed protocol rather than statistical guarantees for all possible real-world deployments.