The Reliability Gap: Repeated-Trial Consistency and the Deployable Autonomy Horizon of Frontier AI Agents

In the rapidly evolving world of artificial intelligence, a new kind of software has emerged: agents that do not just answer questions but take actions. These systems can plan a sequence of steps, use digital tools, and navigate complex environments like software code or web browsers, often with little to no human supervision. To track how capable these agents are becoming, researchers have developed a way to measure their progress using a "time horizon." This concept describes the length of a task, measured in the time it would take a human expert to complete it, that an AI agent can finish successfully half of the time. If an agent has a time horizon of one hour, it means it can reliably solve problems that take a human about an hour to finish, but it will likely fail on tasks that take two hours. This metric has become a standard language for describing AI progress, with reports showing that this horizon is doubling roughly every seven months, suggesting a future where machines can handle increasingly complex work.

However, a critical distinction exists between what an agent can do on average and what it can be trusted to do every single time. For a human manager assigning a recurring job or a regulator overseeing an autonomous system, a 50 percent success rate is often insufficient. If a task happens repeatedly, the goal is not for the agent to succeed half the time, but to succeed every time the task appears. A system that fails once in every two attempts is not reliable enough for many real-world applications, yet the standard "time horizon" metric does not capture this risk. It measures the frontier of capability, not the floor of reliability. This gap between average performance and consistent performance is the central mystery that a new study seeks to solve, asking how far behind the true reliability of these agents lies compared to their headline-grabbing capability scores.

To answer this, researchers turned to a massive collection of public records from the METR organization, which has been running evaluations of AI agents on hundreds of different tasks since 2023. The study focused on 30 different models released between 2023 and early 2026, analyzing over 60,000 individual attempts at various tasks. Instead of looking at whether an agent succeeded or failed just once, the researchers examined what happened when the same agent tried the same task multiple times. They specifically looked at the point where an agent could complete a task four times in a row with a 50 percent probability of total success. This "reliability horizon" represents the length of a task that an agent can be expected to handle consistently, rather than just occasionally.

The results revealed a significant and persistent gap between what agents can do and what they can be trusted to do. For every single model tested, the time horizon for reliable, repeated success was much shorter than the standard capability horizon. On average, the reliable horizon was about 2.4 times shorter than the average-success horizon. To put this in perspective, if a top-tier model is capable of handling a task that takes a human 715 minutes to complete with a 50 percent chance of success, that same model can only be trusted to complete a task of about 355 minutes four times in a row with the same level of confidence. This gap has not closed significantly over the last three years; despite models becoming more powerful, the distance between their average performance and their consistent performance has remained roughly the same.

One might assume that if an agent fails a task, it is simply a matter of bad luck or a random glitch, and that trying again would likely yield a different result. The study found that this assumption is incorrect. The researchers discovered that the outcomes of repeated attempts are strongly linked to the specific task itself, rather than being random. When an agent fails a task, it tends to fail every time it tries that same task. Conversely, when it succeeds, it tends to succeed every time. This means that the reliability of an agent is not determined by a roll of the dice for each attempt, but by the inherent difficulty of the task relative to the agent's skills. Because of this strong connection, the observed reliability is much higher than it would be if every attempt were independent. If the attempts were truly random and unrelated, the reliable horizon would be drastically shorter, but the reality is that the agent's performance is consistent within the context of a specific problem.

The study also pinpointed exactly where these inconsistencies happen. The most unpredictable behavior occurs not on the hardest tasks, where the agents consistently fail, nor on the easiest tasks, where they consistently succeed. Instead, the "mixed outcomes"—where an agent succeeds some times and fails others—cluster tightly around the edge of the agent's capability. This is the zone just below the standard time horizon, where a task is difficult enough to challenge the agent but not so hard that it is impossible. It is precisely in this zone that new, delegated tasks are most likely to fall, making it the most critical area for understanding risk.

Despite the clear evidence of a gap, the researchers found that the rate at which this gap is closing is not statistically significant. While the data shows a slight tendency for the reliability horizon to catch up to the capability horizon, the trend is too small to be distinguished from random variation given the current number of models tested. This suggests that while AI agents are getting better at solving harder problems, they are not necessarily becoming more consistent in their execution at the same pace. The study concludes that for anyone deploying these systems, relying solely on average success rates provides an overly optimistic view of the future. To truly understand what an agent can be trusted to do, reports must include a reliability-adjusted horizon that accounts for the need for repeated, consistent success. Without this adjustment, the promise of autonomous agents remains a measure of potential rather than a guarantee of performance.