From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis
This paper introduces OpsHarness, a self-evolving framework that significantly enhances the root cause analysis capabilities of general-purpose LLM agents by leveraging an adaptive external harness that accumulates and verifies operational expertise from past diagnoses, thereby outperforming both bare general agents and specialized RCA agents.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a complex computer system begins to fail, the problem rarely stays where it starts. In the modern digital world, software is built from thousands of small, interconnected pieces. If one piece stumbles, the error can ripple outward, causing a delay in a user's app or a sudden spike in error messages that looks like a completely different problem. Figuring out where the trouble actually began is a task called root cause analysis. For decades, engineers have relied on maps of how these pieces connect or on statistical patterns to find the culprit. Today, they are increasingly turning to artificial intelligence, specifically large language models, to act as digital detectives. These models are incredibly good at reading logs, understanding code, and reasoning through complex scenarios, much like a brilliant engineer who has read every manual in the library. However, there is a catch: while these models are smart, they do not know the specific quirks of the system they are trying to fix. They lack the deep, accumulated experience that a human expert gains by watching the same system fail and recover over and over again.
Researchers at The Chinese University of Hong Kong and ByteDance have developed a new way to bridge this gap. Instead of trying to build a new, specialized AI from scratch for every system, they created a flexible outer layer that wraps around a powerful, general-purpose AI. They call this layer a harness. Think of this harness as a specialized toolkit and a set of field notes that a general engineer might receive when joining a new team. The general AI provides the reasoning power, but the harness provides the specific context, the right tools to use, and the lessons learned from past mistakes. The most significant innovation in their work, called OpsHarness, is that this layer is self-evolving. It does not just sit there; it watches every diagnosis it helps perform, learns from both the successes and the failures, and updates its own instructions to get better over time.
The team began by testing a common assumption: that building a custom AI agent specifically for troubleshooting is the best approach. They compared these custom-built agents against standard, general-purpose AI models that were simply given the task of diagnosing a problem. The results were surprising. The general models, which had not been trained specifically on troubleshooting, often performed better than the custom agents. This happened because the custom agents were often missing the sophisticated reasoning and planning abilities that the general models had already mastered. However, even the best general models were not perfect. They would often identify the right symptoms but point to the wrong cause because they lacked knowledge of how that specific system behaved under stress. For instance, a model might see a sudden drop in database connections and assume a database failure, when in reality, the drop was just a side effect of a different component, like a server running out of processing power.
To solve this, the researchers designed OpsHarness to act as the bridge between the general intelligence of the model and the specific reality of the system. The system is divided into two main parts. The first part is a knowledge base that stores information in layers. It starts with general rules about how to investigate problems, then adds a profile of the specific system being monitored, and finally, it accumulates a network of specific workflows and rules that have been proven to work. The second part is a library of "idea cards." Instead of hard-coding scripts that might break if the system changes, the harness gives the AI a description of a tool or a method, and the AI writes the code to use it in the moment. This allows the system to adapt to different data layouts without needing to be reprogrammed.
The true power of OpsHarness lies in its ability to learn from experience. After the system helps diagnose a problem, it reviews the entire process. If the diagnosis was correct, the system extracts the successful steps and turns them into a reusable workflow or a rule. If the diagnosis was wrong, it identifies exactly where the reasoning went off track and creates a rule to prevent that specific mistake in the future. This process is not automatic in a reckless way; the system has a strict safety check. Before any new rule is added to its memory, it is tested in a safe, isolated environment to ensure it improves the diagnosis without breaking anything else. This dual-check process prevents the system from learning bad habits or overfitting to a single strange event.
The researchers tested this approach on two public benchmarks containing hundreds of real-world failure cases and also deployed it in a large commercial cloud environment. The results showed a dramatic improvement. When using a general AI model alone, the system correctly identified the root cause about 36 percent of the time. When the same model was equipped with the OpsHarness, that accuracy jumped to nearly 59 percent. This improvement was consistent across different types of AI models and different kinds of systems. The study also found that the system got better the more it was used. In a continuous test where the system diagnosed a series of incidents over time, its accuracy climbed steadily as it accumulated more experience. In contrast, systems that did not evolve remained static, and systems that evolved without the safety checks often made things worse by learning from noise.
The researchers also measured the cost of running this system. While the harness adds some overhead to the process, it does not require significantly more computing power or time than using a general AI model alone. In fact, by guiding the AI to the right path more quickly, it often reduces the number of steps the AI needs to take to solve a problem. The entire system, including all the learned rules and tools, takes up very little storage space, making it practical for real-world use.
This work suggests a shift in how we build intelligent systems for complex tasks. Rather than trying to train a new AI from the ground up for every specific job, the most effective path may be to take a powerful, general AI and wrap it in a smart, evolving layer that teaches it the specific details of the environment. By focusing on this external layer, the researchers have created a system that can turn the raw intelligence of a general model into a specialized expert, capable of learning from its own history and improving with every incident it helps to resolve. The findings indicate that the future of automated troubleshooting may not lie in smarter models, but in smarter ways of teaching those models how to work.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.