🤖 AI

The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

This paper demonstrates that the evaluation harness significantly influences coding agent performance and efficiency—often more than the underlying model choice—by revealing up to 40x token usage variations and model-independent failure patterns, thereby arguing for the reporting of full harness specifications and token/latency metrics alongside model comparisons.

Original authors: Naman Vats, Oleg Golev

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Naman Vats, Oleg Golev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Invisible Hand Behind the Robot Coder

Imagine you are watching a cooking competition. You see two brilliant chefs, Chef A and Chef B, both trying to bake the perfect chocolate cake. The judges usually just look at the final cake and say, "Chef A made a better cake!" But what if the judges forgot to mention that Chef A was using a high-tech, automated kitchen that pre-chopped every ingredient and pre-heated the oven to the exact second, while Chef B was working in a dusty shed with a rusty knife and a broken stove? The result isn't just about the chefs' talent; it's about the kitchen they were given.

In the world of artificial intelligence, specifically "coding agents" (AI programs that write software), we have been doing exactly that. We have been ranking AI models based on how many coding problems they solve, while ignoring the "kitchen" they are working in. This kitchen is called a harness. Think of a harness as the invisible assistant that hands the AI its tools, manages its memory, and decides when to stop working. Some harnesses are like a super-efficient robot butler that organizes everything perfectly; others are like a chaotic intern who keeps asking the same question over and over, wasting time and energy. The big question researchers are asking is: Does the AI's brain matter more, or does the quality of its assistant matter more?

The Great "Kitchen" Experiment

A team of researchers decided to stop guessing and start measuring. They set up a massive experiment to see what happens when you swap the "kitchen" (the harness) while keeping the "chef" (the AI model) exactly the same. They picked two very smart AI coding models, Qwen 3.6 Plus and MiniMax M2.5, and asked them to solve 50 different coding puzzles. But here is the twist: they ran these same models through three different open-source harnesses, named Goose, OpenCode, and OpenHands-SDK.

The results were shocking, to say the least.

When the researchers looked at how many puzzles the AI solved (the "pass rate"), the difference between the harnesses was tiny. Whether the AI was using Goose, OpenCode, or OpenHands-SDK, it solved roughly the same number of problems—usually between 38% and 50%. Changing the harness didn't make the AI suddenly a genius or a failure; the success rate stayed mostly flat.

However, when they looked at the cost of solving those problems, the story changed completely. The researchers measured this in "tokens," which are basically the units of data the AI processes (like words or pieces of code). They found that the choice of harness changed the cost by a staggering 40 times.

To put that in perspective: If the Goose harness helped the AI solve a puzzle using about 28,000 tokens, the OpenCode harness made the exact same AI use over 1.1 million tokens to solve the same puzzle. That is a 40-fold difference in cost for the exact same result! It's like one chef using a single egg to bake a cake, while the other chef uses 40 eggs for the exact same cake, simply because their kitchen tools were inefficient.

The "Idle Turn" Tax

Why did the cost go up so much? The researchers discovered a hidden culprit: idle turns.

Imagine you are talking to a friend who is trying to fix your computer. Sometimes, they just sit there thinking, "Hmm, let me check this..." without actually typing anything or changing anything on the screen. In the AI world, these are called "no-action turns." The OpenCode harness made the AI sit in these thinking loops about 2 times per task, while the Goose harness only did this about 0.2 times.

Every time the AI sits in one of these loops, it has to send its entire conversation history back to the server to keep track of where it is. This is like sending a 100-page diary every time you pause to think. Because OpenCode made the AI pause and think so much more often, it burned through tokens at a massive rate. The researchers call this the "per-task wait tax." It's not just a cost in money; it's a cost in time. A human developer watching the AI has to wait through these idle loops, staring at a screen that isn't doing anything, just burning money and patience.

The "Fingerprint" of Failure

The study also found that each harness had its own unique way of failing, like a fingerprint. These patterns didn't change based on which AI model was used; they were caused by the harness itself.

Goose was the cautious one. When it got stuck, it would stop and say, "I can't do this," rather than guessing. It rarely wasted time trying to verify a bad idea.
OpenHands-SDK was the persistent one. It would keep trying to verify its answers or run until it hit a hard limit on how many times it could try (the "max turns" limit).
OpenCode was the one that got stuck in loops. It would run out of time (the "wall-time" limit) or get stuck in a "hang" state, spinning its wheels without ever finishing.

This means that if you are a developer, you aren't just choosing an AI; you are choosing a style of failure. Do you want an AI that gives up quickly and honestly, or one that keeps spinning its wheels until you run out of patience?

The Big Takeaway

The main lesson from this paper is that we have been looking at AI coding benchmarks all wrong. We have been ranking models like Qwen or MiniMax as if they are the only thing that matters. But this study shows that the harness (the kitchen) is just as important as the model (the chef).

If you want to know how much an AI will actually cost you or how long it will take to finish a job, you cannot just look at the model's name. You have to look at the pair: the model and the harness it is running in. A smart model in a bad harness can be 40 times more expensive than a smart model in a good harness.

The researchers suggest that in the future, we shouldn't just report "Pass Rate." We need to report the cost per solved task, the number of idle turns, and the failure patterns. Because for a real human developer, the difference between a 50% success rate that costs $1 and a 50% success rate that costs $40 is the difference between a useful tool and a money pit. The "Scaffold Effect" is real, and it's time we stopped ignoring the invisible hand that guides the robot.

Technical Summary: The Scaffold Effect in Coding Agents

Problem Statement
Current public leaderboards for coding agents primarily rank systems by model name and pass rate, treating the surrounding "harness"—the scaffold that issues tools, manages context, and determines termination—as a fixed or under-specified constant. This approach conflates model performance with scaffold efficiency. When harnesses vary, performance metrics (pass rate) and efficiency metrics (token usage, latency) become entangled, obscuring the true cost and oversight burden of deploying an agent. The paper argues that for human-centered evaluation, the unit of comparison should be the harness–model pair, not the model alone, as the harness dictates real-world costs and the nature of human oversight required.

Methodology
The authors conducted a controlled study to isolate the effects of the harness from the model.

Models: Two recent agentic coding models were evaluated: Qwen 3.6 Plus and MiniMax M2.5.
Harnesses: Three distinct open-source harnesses were tested, each with different scaffolding philosophies:
- Goose: A heavyweight IDE agent with eager file-tree pre-injection.
- OpenCode: A persistent tool-loop agent without automatic context pre-loading.
- OpenHands-SDK: A micro-agent architecture featuring sub-agent delegation, internal retries, and explicit verification steps.
Tasks: A stratified subset of 50 tasks was drawn from the Terminal-Bench Pro dataset, covering eight archetypes (Bug fix, Build, Data, Implementation, ML, Puzzle, Security, System). All tasks utilized deterministic pytest evaluation suites.
Protocol: All trials ran through the OpenRouter API with identical task instructions and system prompts (varying only in tool-specific vocabulary). The study held constant the task manifest, sandbox environment, and wall-time cap (900 seconds), while allowing each harness to operate with its native tool APIs, context management, and internal retry logic.
Metrics: The study measured pass rates, total tokens per solved task, average turns, "no-action turns" (turns with no file edits or new commands), and a six-class failure taxonomy (REASON, VERIFY, TIME, MAX TURNS, HANG, ERROR). Statistical significance was assessed using 95% paired-task bootstrap confidence intervals ( $B=10,000$ ).

Key Results

Token Cost Disparity (The 40× Gap): The choice of harness induced a massive difference in efficiency. OpenCode consumed approximately 40× more tokens per solved task than Goose (e.g., ~1.5M tokens vs. ~37K tokens for MiniMax). In contrast, upgrading the model while holding the harness fixed resulted in only a 1.0–1.3× difference in token usage.
Pass Rate Stability: Within a fixed model, pass-rate differences across harnesses were small, ranging from 0 to 8 percentage points. Most pairwise differences were statistically indistinguishable from zero (95% CIs included zero), except for the largest gap.
Failure Fingerprints: Specific failure modes replicated across both models, indicating they are properties of the harness, not the model:
- Goose: Dominated by REASON failures (stopping when stuck) with zero VERIFY failures.
- OpenHands-SDK: Characterized by VERIFY and MAX TURNS failures, indicating a tendency to persist toward closure or exhaust iteration limits.
- OpenCode: Characterized by TIME and HANG failures (exhausting wall-time caps) with zero MAX TURNS (due to lack of a turn-budget flag) and zero VERIFY.
No-Action Turns as an Oversight Tax: OpenCode exhibited 10× more no-action turns (idle reasoning loops) than Goose. These turns accumulate context tokens without making progress, creating a "per-task wait tax" for human users who must monitor these idle loops.
Pareto Efficiency: Goose dominated the Pareto frontier, offering comparable pass rates to other harnesses at a fraction of the token cost. OpenCode was Pareto-dominated by both other harnesses.

Key Contributions

Quantification of the Scaffold Effect: The study provides controlled evidence that harness choice drives a 40× gap in token costs while shifting pass rates by at most 8 percentage points, demonstrating that efficiency is largely a scaffold property.
Model-Independent Failure Fingerprinting: The authors identify that specific failure modes (e.g., REASON vs. VERIFY vs. TIME) are consistent across different models when the harness is fixed, proving these are structural biases of the scaffold.
Mechanistic Proxy for Oversight: The paper introduces "no-action turns" as a proxy for human oversight burden, framing them as a cost (both financial and temporal) that a human-in-the-loop user must absorb, distinct from simple token costs.
Proposed Evaluation Unit: The authors argue that the harness–model pair is the correct unit of comparison for human-centered evaluation, rather than the model name alone.

Significance and Claims
The paper claims that current benchmark reporting is incomplete because it isolates the model from the scaffold. By reporting only pass rates, leaderboards obscure the real-world cost, latency, and supervisory effort required to deploy an agent. The authors assert that for practical deployment, developers must evaluate harness–model pairs under specific token/latency budgets. They recommend that future evaluations report tokens per solved task, average no-action turns, and failure-category vectors alongside pass rates to provide a holistic view of an agent's deployability. The study does not claim to solve the problem of agent evaluation but rather to systematize the observation that "scaffold choice" is a primary driver of efficiency and failure profiles, often outweighing model upgrades.