Technical Summary: BRIDGE – Predicting Human Task Completion Time From Model Performance

Problem Statement

Evaluating the real-world capabilities of AI systems typically relies on benchmark scores and aggregate metrics. However, these scores are difficult to interpret as measures of actual task difficulty or real-world effort. Improvements in scores may reflect gains on short, routine instances while leaving longer, multi-step tasks unchanged, and comparable score changes can correspond to vastly different shifts in practical capability.

A prominent alternative, proposed by METR (Kwa et al., 2026), measures capability in terms of the "human task completion time" required for an AI agent to succeed. While this provides a human-interpretable scale, it relies on expensive, noisy, and difficult-to-scale human annotations. As new benchmarks emerge, the reliance on bespoke human studies to anchor each benchmark creates a growing gap between benchmark-centric evaluation and human-centric notions of difficulty. The core challenge is: Can we predict human task completion time for new benchmarks without conducting new human studies?

Methodology: The BRIDGE Framework

The authors propose BRIDGE (Benchmark Response Inferred Difficulty Grounded in Elapsed human time), a unified psychometric framework that aligns latent task difficulty inferred from model performance with human task completion time.

1. Item Response Theory (IRT) Modeling

The framework utilizes a Two-Parameter Logistic (2PL) IRT model to jointly estimate:

• Latent Task Difficulty ($b_i$): A parameter characterizing the difficulty of individual tasks.
• Model Capability ($\theta_j$): A parameter characterizing the ability of individual models.
• Discrimination ($a_i$): A parameter measuring how well a task differentiates between models of varying capabilities.

The probability of a model $j$ successfully completing task $i$ is modeled as:
$$P(y_{i,j} = 1 | \theta_j, a_i, b_i) = \sigma(a_i(\theta_j - b_i))$$
where $\sigma$ is the logistic function. The model is fitted using Markov chain Monte Carlo (MCMC) with hierarchical priors on binary success/failure outcomes across multiple benchmarks.

2. Anchoring to Human Time

While IRT provides a unified latent scale, its parameters are inherently scale-invariant and lack intrinsic semantic meaning. BRIDGE resolves this by anchoring the latent difficulty scale to human completion time.

• Calibration: Using a dataset of tasks with known human completion times (specifically from METR), the authors fit a linear relationship between the IRT-inferred latent difficulty ($b_i$) and the logarithm of human completion time ($h_i$):
  $$\log(h_i) = \text{slope} \times b_i + \text{intercept}$$
• Prediction: Once calibrated, this log-linear mapping allows the prediction of human completion time for new benchmarks (without human annotations) solely based on their IRT-inferred latent difficulty.

3. Forecasting Horizon

The framework tracks the evolution of frontier model capabilities by identifying the best-performing model in specific release windows. By mapping the model's estimated ability ($\theta_{max}$) to the difficulty scale ($b = \theta_{max}$ at 50% success), the system forecasts the "task-length horizon"—the duration of tasks a model can complete with a given probability.

Key Contributions

1. Unified Psychometric Framework: BRIDGE bridges the gap between model performance metrics and human-interpretable time units by calibrating IRT latent difficulty against human completion time.
2. Log-Linear Alignment: The paper demonstrates a strong empirical finding that latent task difficulty varies linearly with the logarithm of human completion time ($R^2 = 0.81$ on METR data). This allows for the inference of human task duration from model performance alone.
3. Scalable Forecasting: The framework enables the forecasting of frontier model capabilities in terms of human task length without requiring new human studies for each benchmark.
4. Robustness: The method is shown to be robust to sparsity in model-task coverage, maintaining high correlation in difficulty estimates even when up to 70% of observed entries are removed.

Experimental Results

The authors evaluated BRIDGE on several out-of-distribution benchmarks: SWE-bench Verified, MLE-bench, GDPval, and Cybench.

• Accuracy of Time Prediction:
  • On SWE-bench Verified, BRIDGE's predicted time buckets aligned significantly better with human annotations than both a logit success-rate heuristic and LLM-based estimators (Gemini 3 Pro, GPT-5.2). BRIDGE achieved a weighted macro F1 score of 0.284 compared to 0.198 for the logit baseline and 0.048 for GPT-5.2.
  • On Cybench, BRIDGE achieved an $R^2$ of 0.45 against human first-solve times, with 92.3% of tasks falling within a 0.5×–2× tolerance band. In contrast, logit baselines systematically underestimated duration, while LLM estimators consistently overestimated it.
• Forecasting Trends:
  • Using only model performance logs, BRIDGE forecasts that the "50%-task-completion time horizon" (the length of tasks solvable with 50% probability) is growing exponentially.
  • The estimated doubling time is approximately 6 months.
  • As of the latest frontier models (e.g., Claude 4.5 Opus, GPT-5), the 50% success horizon is estimated at roughly 1.4 to 2.5 hours of human effort, depending on the benchmark.
  • At a stricter 80% success threshold, the horizon contracts to approximately 40 minutes, but the doubling time remains consistent at ~6 months.

Significance and Claims

The paper claims that BRIDGE provides a scalable, reliable, and human-grounded method for evaluating and forecasting AI progress.

• Independence from Human Studies: It demonstrates that human-centric task difficulty can be inferred from model-centric data, reducing the cost and friction of tracking real-world AI progress.
• Validation of Scaling Laws: The results independently reproduce and corroborate METR's findings regarding exponential growth in task-length horizons, validating the use of model performance as a proxy for human time.
• Interpretability: By translating abstract benchmark scores into "hours of human effort," BRIDGE offers a more actionable metric for understanding what AI systems can reliably do and how quickly their practical capabilities are improving.

The authors conclude that while IRT is inherently scale-invariant, anchoring it to human time creates a "Rosetta stone" for AI benchmarks, enabling the transfer of difficulty estimates between humans and models across diverse and evolving task distributions.