When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
This paper introduces AGENTCHAOSBENCH, a benchmark dataset of 275 agentic execution traces with injected runtime faults, to demonstrate that current LLM-based methods struggle to accurately detect and localize diverse operational failures from telemetry alone.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Modern software is no longer just a single program running on a computer; it is often a team of artificial intelligence agents working together. Imagine a digital workforce where one agent plans a trip, another books the flights, and a third checks the weather, all communicating with each other and using external tools like maps or calendars. These systems rely on a complex web of connections: the agents talk to each other, they ask a large language model for advice, they call upon external tools to do work, and they follow strict safety rules to ensure they do not say or do something harmful. When everything works, the team produces a correct answer. But when something goes wrong, the final answer might be wrong, or the system might simply stop working, leaving the user with no clue why. The problem is that looking only at the final result tells us nothing about the breakdown. Did the system fail because the AI made a bad decision, or because a tool it needed was slow, or because a safety check was accidentally skipped? To fix these systems, engineers need to see the entire journey, not just the destination.
A team of researchers at the University of Toronto has built a new testing ground to help solve this mystery. They created a benchmark called AgentChaosBench, which is essentially a controlled environment for breaking these AI teams on purpose to see if diagnostic tools can find the problem. The researchers took five different real-world applications—systems that write SQL code, draft books, manage social media, create landing pages, and assist with hiring—and simulated ten different ways they could fail. These failures included tools that refused to answer, tools that took too long to respond, messages that got lost between agents, and safety rules that were bypassed. For every broken scenario, they also ran a perfect, fault-free version of the same task using the exact same starting instructions. This pairing allowed them to know exactly what went wrong and where, creating a set of 275 detailed records of these digital journeys.
The core of their work was to see if an automated system could look at the record of a failed execution and correctly identify the cause. They stripped away any labels that would give the answer away, leaving only the raw data of what happened: the timing of calls, the content of messages, and the status of each step. They then asked various artificial intelligence models, ranging from smaller local models to the most powerful frontier models available, to act as detectives. These models had to read the record, figure out which of the ten fault types had occurred, and pinpoint exactly which part of the system was responsible. The researchers also tested whether giving the detective a copy of the perfect, fault-free run to compare against would help.
The results showed that this task is far more difficult than anyone might have hoped. Even the most advanced models, which can write poetry and solve complex logic puzzles, struggled significantly to diagnose these runtime faults. When asked to identify the type of failure from a single record, the best model got the correct answer less than 25 percent of the time. For smaller models, the success rate was even lower, hovering around 13 to 19 percent, which is barely better than random guessing. The problem was even harder when the models had to point to the specific component that failed. They managed to locate the correct part of the system only about 31 percent of the time. When asked to do both at once—name the fault and find the location—the success rate dropped to just 22 percent for the best model.
The study revealed that some failures are easier to spot than others. Errors that produce obvious signals, like a tool returning a clear error message or a connection timing out, were identified more frequently. However, the most dangerous and subtle failures remained nearly invisible. When a safety rule was bypassed, allowing a request that should have been blocked to proceed, the models almost always failed to notice. Similarly, when a tool response was corrupted or when the system ran out of memory space, the models could not reliably distinguish these issues from normal behavior. The researchers found that providing a perfect reference run to compare against did help in some cases, such as identifying when a tool was unusually slow or when the system tried to use too much memory. However, this comparison did not help with the safety bypasses or corrupted data, because the broken output still looked plausible and the safety check still appeared to have passed.
This work demonstrates that while we have built sophisticated systems to coordinate multiple AI agents, we have not yet built the tools to reliably understand why they fail. The current generation of diagnostic models, even the largest and most capable ones, cannot yet consistently tell the difference between a broken tool, a slow network, a skipped safety check, and a normal operation. The researchers conclude that fixing these systems will require new methods that go beyond simply asking a large language model to read a log. They suggest that future solutions may need to rely on comparing current executions against known good ones or developing specialized tools designed specifically to look for these operational faults, rather than relying on general-purpose intelligence alone. The path forward involves building better ways to see the invisible cracks in the digital machinery before they cause the whole system to collapse.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.