StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments

In the world of modern computing, there is a growing class of intelligent programs known as agents. These are not just simple tools that follow a single command; they are systems designed to reason, look at information, and take actions to solve complex problems, much like a human employee would. To function, these agents rely on two distinct parts. The first is the brain, a large language model that provides the reasoning and knowledge. The second is the harness, which acts as the agent's hands, eyes, and ears. The harness defines how the agent sees the world, what tools it is allowed to use, and how it checks its own work. For years, scientists have focused almost entirely on making the brain bigger and smarter. However, in the messy, rule-bound reality of business environments, a brilliant brain can still fail if its hands are clumsy or if it does not understand the specific rules of the office it is working in. This raises a practical question: if we cannot change the brain, can we improve the hands and the rules to make the agent work better?

A team of researchers at ServiceNow, along with collaborators from academic institutions, set out to answer this question with a new framework called StarHarness. They focused on a specific challenge found in enterprise settings, where software must manage things like IT systems, financial workflows, and customer service records. In these environments, the software must interact with databases and tools that have strict, often hidden rules. The researchers wanted to see if they could automatically improve the agent's harness without ever touching the underlying brain. They treated the harness as a piece of code that could be evolved, or improved, through a process of trial and error. The goal was to find a better way for the agent to frame its tasks, use its tools, and verify its results, all while keeping the model's internal weights completely frozen.

To test their idea, the researchers created a system that could search for improvements in a controlled way. They started by running the agent on a large set of tasks to see where it typically failed. They then divided these tasks into three groups. One group was used to propose changes, another was kept hidden to test if the changes actually worked, and a third group was saved for a final, unbiased check. The system would suggest a small change to the harness, such as fixing how a tool was called or adding a rule to prevent a common mistake. It would then test this change on the hidden group. If the change made the agent perform better, the system kept it; if not, it discarded it. This process repeated itself, allowing the harness to slowly evolve into a more capable version of itself. The researchers used this method across three different types of business challenges: analyzing computer network failures, managing IT service requests, and handling financial workflows.

The results were striking. After the system made only a handful of accepted changes—typically between four and twelve adjustments per environment—the agent's performance jumped significantly. Across the benchmarks, the full-benchmark performance improved by 20–35 percentage points over the default harness. Specifically, the success rate improved by 35 percentage points on the network analysis tasks, rose by 20 percentage points in the IT service management tests, and climbed by 26 percentage points in the financial workflows. Crucially, these improvements were not just a result of memorizing the specific problems the system had seen during the training. When the researchers tested the evolved harness on new, unseen tasks, the agent still performed much better than before. Even more surprisingly, the improvements transferred to different types of AI models. A harness evolved using one specific model family worked just as well when given to a completely different model family, without needing any further training or adjustment.

The researchers analyzed the changes to understand what the system had actually learned. They found that the improvements fell into three clear categories. First, the system repaired the interfaces between the agent and its tools, fixing errors that caused the agent to call tools incorrectly. Second, it learned the hidden conventions of the environment, such as the specific order in which certain steps must be taken or how to handle dates and priorities correctly. Third, it compressed the search process by adding operational knowledge, allowing the agent to skip unnecessary steps and focus on the most likely causes of a problem. For example, in the financial tasks, the evolved harness learned to check for safety violations before making any changes, which drastically reduced the number of errors. In the network analysis tasks, the agent learned to stop searching once it found a likely cause, rather than wasting time looking for more evidence that was not needed.

These findings suggest that for many real-world business applications, the biggest barrier to success is not the intelligence of the model, but the design of the system that guides it. By evolving the harness, the researchers were able to reduce the number of false diagnoses, shorten the time it took to complete tasks, and lower the cost of running the agent. The study demonstrates that a fixed, unchanging brain can become remarkably more effective when its surrounding tools and rules are carefully tuned to the specific environment it operates in. This offers a practical path forward for organizations that need reliable AI agents today, without waiting for the next generation of larger models to be developed. The work shows that sometimes, the most powerful upgrade is not a new mind, but a better way of working.