🤖 AI

Measuring AI Ability to Complete Long Software Tasks

This paper introduces the "50%-task-completion time horizon" metric to quantify AI capabilities by comparing them to human effort, finding that frontier models can now complete tasks in 50 minutes that humans typically take 50 minutes to finish, with a doubling trend suggesting AI could automate month-long software tasks within five years.

Original authors: Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkov

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence Chan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're watching a race between a human and a robot, but instead of running on a track, they are both trying to solve a giant, ever-growing pile of software puzzles. For years, we've been trying to measure how good these AI robots are using tricky tests, but those tests often feel like artificial video game levels that don't really tell us how fast the robot can actually work in the real world.

To fix this, a team of researchers from METR decided to measure something much more intuitive: How long can an AI work before it starts to fail?

They called this the "Time Horizon." Think of it like a battery life for intelligence. If an AI can solve a puzzle that takes a human 10 minutes to finish, its time horizon is 10 minutes. If it can handle a task that takes a human 4 hours, its horizon is 4 hours. The researchers wanted to find the specific point where an AI has a 50% chance of getting the job done right.

The Big Discovery: The Exponential Explosion

The team gathered a massive collection of 170 different tasks, ranging from tiny, 1-second checks (like spotting a specific file name) to massive, 8-hour research projects. They hired skilled human experts to time how long it took them to finish these tasks. Then, they let 12 different AI models, from the old GPT-2 (released in 2019) to the brand-new o3 (released in 2025), try to solve them.

Here is the wild part: The AI's "battery life" is doubling every seven months.

In 2019, the best AI could only reliably handle tasks that took a human about 2 seconds to finish. By 2025, the top models (like o3) could handle tasks that take a human about 110 minutes (nearly two hours) with a 50% success rate.

The researchers found that this growth isn't just a little bit faster; it's an exponential curve. It's like watching a snowball roll down a hill, getting twice as big every time it passes a certain tree. They measured this trend from 2019 to 2025 and found the "doubling time" to be approximately 207 days (roughly seven months).

Why Are They Getting Better?

The paper suggests that the AI isn't just getting "smarter" in a vague way; it's getting better at specific, practical skills. When the researchers looked at why the older models failed and the newer ones succeeded, they found the newer models were:

Better at using tools: They know how to use a calculator or a code editor without breaking them.
Better at fixing mistakes: If an old AI made a mistake, it would often just repeat the same error over and over. The new models can spot the error, say "Oops," and try a different path.
Better at logical reasoning: They can follow a chain of thought without getting lost.

The "Messy" Caveat: Real Life is Harder

However, the paper is very careful not to say the robots have won the war yet. The researchers explicitly argue that these tests might be too "clean."

Imagine the AI is playing a video game where the rules are written in a clear manual, the enemies are predictable, and there are no sudden power outages. The researchers call real-world work "messy." In the real world, tasks often have unclear instructions, require talking to other people, or involve resources that run out.

When the team tested the AI on tasks that were "messier" (less structured, more chaotic), the AI's performance dropped significantly. The paper suggests that while the AI is getting great at the "clean" puzzles, it might still struggle with the chaotic, unpredictable nature of a real software engineer's day. They found no evidence that the AI's improvement slows down on messy tasks yet, but they warn that the gap between "clean test" and "messy reality" is a big unknown.

What Does This Mean for the Future?

The researchers did some math to see where this curve leads. If the trend of doubling every seven months continues, they predict that within 5 years (sometime between mid-2028 and mid-2031), AI systems might be able to autonomously complete software tasks that currently take a human one month (about 167 work hours) to finish.

They emphasize that this is a projection, not a guarantee. It depends on whether the AI keeps improving at this exact speed and whether those "clean" test results apply to the "messy" real world. If the trend holds, we could see AI agents that can essentially work as full-time employees on complex projects in the near future. But if the trend slows down or the "messiness" of real life proves too difficult, that timeline could change.

The Bottom Line

The paper doesn't claim AI has solved everything. Instead, it gives us a new ruler to measure progress. It shows that for the last six years, AI's ability to handle long, complex tasks has been growing at a terrifyingly fast pace, doubling its "work day" roughly every seven months. While there are still big gaps in how well they handle messy, real-world chaos, the trajectory suggests that the day an AI can work a full month-long shift on its own might be closer than we think.

Technical Summary: Measuring AI Ability to Complete Long Software Tasks

Problem Statement

Despite rapid progress in AI benchmarks, the real-world meaning of benchmark performance remains ambiguous. Existing benchmarks suffer from three primary limitations: they often consist of artificial tasks lacking economic value; they are frequently adversarially selected to highlight current model weaknesses, biasing comparisons against human performance; and they saturate quickly, making it difficult to compare models of vastly different capabilities (e.g., GPT-2 vs. Claude 3.7) on a unified scale. Consequently, there is a lack of a general, intuitive, and quantitative metric to track AI capability progress over time in terms of human labor.

Methodology

The authors propose a new metric: the task completion time horizon. This is defined as the duration of tasks that an AI model can complete with a specific success probability (e.g., 50%). This metric relates AI performance directly to the typical time required by human experts to complete similar tasks.

Data and Task Suite

To operationalize this metric, the authors constructed a dataset of 170 tasks spanning a wide range of difficulties, categorized into three suites:

HCAST (Human-Calibrated Autonomy Software Tasks): 97 diverse software tasks (cybersecurity, ML, software engineering) ranging from 1 minute to 30 hours.
RE-Bench: 7 difficult machine learning research engineering tasks, each designed to take a human expert approximately 8 hours.
Software Atomic Actions (SWAA): A novel suite of 66 short, single-step tasks (1 second to 30 seconds) designed to measure pre-2023 models and fill the gap for very short durations.

Human Baselining

To ground the AI performance, the authors measured the time taken by skilled human professionals (software engineers, ML researchers, cybersecurity experts) to complete these tasks.

Total Effort: Over 800 human baselines totaling 2,529 hours.
Metric: Task difficulty is defined by the geometric mean time of successful human baselines.
Context: Baseliners were given minimal task-specific context (similar to an external contractor), distinct from the deep institutional knowledge of internal maintainers.

AI Evaluation and Modeling

The authors evaluated 12 frontier and near-frontier models released between 2019 and 2025. Each model was run multiple times per task using standardized agent scaffolds (e.g., modular-public, triframe) that provided tools like Python and Bash execution.

Binarization: Continuous scores were converted to binary success/failure based on task-specific thresholds representing human performance levels.
Logistic Regression: The authors fitted a logistic regression model inspired by Item Response Theory (IRT) to estimate the time horizon ( $h_{agent}$ ) where the probability of success is 50%:
$P_{success}(agent, task) = \sigma((\log h_{agent} - \log t_{task}) \cdot \beta_{agent})$
Where $t_{task}$ is the human baseline time and $h_{agent}$ is the learned parameter representing the 50% time horizon.

Key Results

Exponential Growth in Capability

The study finds that the 50% task completion time horizon has grown exponentially from 2019 to 2025.

Doubling Time: The time horizon doubles approximately every 207 days (roughly 7 months).
Range: The horizon expanded from ~2 seconds for GPT-2 to approximately 110 minutes for the o3 model.
Recent Acceleration: The growth rate between 2023–2025 appears to be about 20% faster than the 2019–2025 average, though this is within the margin of error.

Drivers of Improvement

Qualitative analysis of failure modes (comparing GPT-4 1106 and o1) indicates that progress is driven by:

Reliability and Adaptation: A marked reduction in "repeating failed actions" and an increased ability to recover from mistakes.
Tool Use: Improved capability in using tools (e.g., editing files, running commands) without syntax errors.
Logical Reasoning: Better performance on tasks requiring multi-step logical reasoning and code generation.
Limitations: Models still struggle significantly in "messier" environments lacking clear feedback loops or requiring proactive information seeking.

External Validity and Robustness

SWE-bench Verified: Applying the methodology to SWE-bench Verified yields a similar exponential trend with an even steeper slope (doubling time ~70 days), though the authors note this may be due to underestimation of human time by annotators for easier tasks.
Messiness Factors: Tasks were rated on 16 "messiness" factors (e.g., dynamic environments, resource constraints). While absolute performance is lower on messier tasks, the rate of improvement over time is similar for both high and low messiness subsets.
Internal PRs: Testing on internal pull requests revealed that AI performance correlates better with the time taken by external contractors (low context) than by repository maintainers (high context), suggesting current metrics reflect low-context labor capabilities.

Extrapolation and Forecasts

Using the observed exponential trend, the authors extrapolate when AI systems might achieve a one-month time horizon (approx. 167 working hours), a threshold often associated with transformative economic value or catastrophic risk.

Central Estimate: If the trend continues, AI systems capable of autonomously completing one-month software tasks are predicted to emerge between mid-2028 and mid-2031.
Sensitivity: If the faster 2024–2025 trend persists, this could occur as early as early 2027.
Caveats: The authors emphasize that this extrapolation assumes the trend generalizes to real-world tasks and does not account for potential future shifts in the trend (e.g., saturation, algorithmic breakthroughs, or AGI).

Significance and Claims

The paper claims to provide the first robust, quantitative metric for tracking AI agent progress in terms of human labor hours, moving beyond static benchmark scores.

Governance: The authors argue that such metrics are foundational for responsible AI governance, allowing developers to determine necessary risk mitigations as systems become more powerful.
Safety Implications: The rapid doubling of time horizons suggests that AI systems capable of autonomously performing complex, multi-week tasks (including potentially dangerous capabilities like CBRN development or self-replication) may arrive sooner than previously anticipated if current trends hold.
Modesty: The authors explicitly state that their results are limited to software and research tasks and may not perfectly generalize to all forms of intellectual labor. They caution that external validity concerns and future changes in the trend rate introduce significant uncertainty into long-term forecasts.