StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
The paper introduces StateM, an agent-native runtime that leverages harness scaling through durable states, checked transitions, and versioned runbooks to significantly boost long-horizon agent accuracy on Terminal-Bench 2.1 to 95.3% while drastically reducing inference costs compared to baseline models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving field of artificial intelligence, a persistent puzzle has emerged: why do sophisticated computer programs, capable of solving complex steps individually, often fail when asked to complete a long, multi-step job? Imagine a brilliant architect who can design a perfect room but forgets to lay the foundation, or a skilled surgeon who performs a flawless incision but loses track of the patient's vital signs halfway through. These systems, known as long-horizon agents, frequently stumble not because they lack intelligence, but because they lose their place in the process. They may forget what they have already done, skip a necessary check, or stop working before the final goal is truly reached. For years, the standard solution has been to build bigger, smarter brains, hoping that a more powerful model will simply not make these mistakes. However, a new line of inquiry suggests that the problem might not be the brain at all, but rather the environment in which it works.
Researchers have begun to ask a different question: how much of this failure is actually a failure of the system that holds the agent's attention, tracks its progress, and forces it to finish the job? This approach, which the authors call "harness scaling," proposes that we can make an existing, capable model far more reliable by improving the tools and rules that surround it, without changing the model itself. Instead of trying to force the model to remember everything on its own, this method gives the agent a shared, external notebook that it and a human supervisor can both read and update. This notebook acts as a durable record of where the agent is, what it has promised to do next, and what proof is required before it can move forward. By treating the execution process as a series of distinct phases with clear checkpoints, the system ensures that the agent does not drift off course or leave a task half-finished.
The researchers behind this study, led by Ziheng Qin and colleagues, built a lightweight system called StateM to test this idea. They designed it to work with standard computer agents that operate through command lines, much like a human typing instructions into a terminal. The core of their invention is a simple, readable document called a runbook. This document acts as a map and a contract for the agent. It breaks a large task into specific stages, or states, such as planning, building, checking, and handing off the work. When the agent enters a new stage, the system refreshes its instructions and shows it exactly what it needs to accomplish before it can leave. Crucially, the agent cannot simply declare a stage complete; it must pass a verification check. If the agent tries to skip a step or fails a check, the system stops it and forces it to repair the error before proceeding. This creates a shared space where the agent and a human can inspect the work, audit the decisions, and update the rules together.
To see if this approach worked, the team tested it on a rigorous set of 89 complex computer tasks known as Terminal-Bench. They started with a powerful model called GPT-5.5 and applied their new system. The results were striking. Without changing the model's internal code or training it on new data, the system improved the model's success rate from 83.1 percent to 92.1 percent. This gain was so significant that it effectively closed the gap between this older model and a newer, more advanced version of the software. The researchers then took the exact same set of rules, or runbook, and applied it to an even newer model, GPT-5.6, without making any adjustments. The system performed even better, achieving a raw accuracy of 95.3 percent across hundreds of trials. It successfully solved every single one of the 89 tasks at least once, demonstrating that the rules developed for one model could transfer seamlessly to a newer one.
The study also explored whether these benefits could be achieved with cheaper, different types of artificial intelligence models. When they applied the same principles to a model called DeepSeek-V4 Flash, the initial results were mixed because the exact rules did not fit perfectly. However, by spending a small amount of money—less than thirty-eight dollars—to adapt the specific practices to this new model, they were able to raise its performance from 82.7 percent to 88.1 percent. The total cost to prove this result, including the adaptation and final testing, was approximately fifty-two dollars. In contrast, the most expensive public submission for a similar high-level result cost nearly six hundred dollars. This suggests that investing in a better execution system can be far more cost-effective than simply buying the most powerful model available.
The researchers also tested their system on a different set of tasks involving business workflows, such as approving budgets or operating machinery. Here, the results showed that the rules worked best when they matched the specific nature of the work. For tasks that required strict adherence to rules and clear handoffs, the system provided massive improvements, sometimes doubling the success rate. However, for tasks that were very different in structure, the system sometimes made things worse if the rules were too rigid or applied to the wrong part of the process. This taught the researchers that the key is not to have a massive list of rules for every possible situation, but to identify the specific boundaries where errors are most likely to happen and enforce checks only there.
Ultimately, this work suggests that the reliability of artificial intelligence depends as much on how we manage its work as on how smart the model is. The researchers identified three main reasons why capable agents fail: they lack the right knowledge at the moment of decision, they forget lessons learned from previous attempts, or they fail to follow a procedure they already know. Their system addresses these issues by keeping a fresh, clear record of the current phase, storing lessons in a versioned document that can be reused, and forcing the agent to prove it has finished a step before moving on. The findings indicate that we do not necessarily need to wait for the next generation of super-intelligent models to solve complex problems. Instead, by building better systems to guide the models we already have, we can turn capable but unreliable agents into robust, finished-job machines. The path forward is not just about making the brain bigger, but about building a better body to carry it through the work.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.