Technical Summary: Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models

Problem Statement

Current safety monitoring strategies for frontier AI models often rely on inspecting Chain-of-Thought (CoT) reasoning traces. The assumption is that if a model is reasoning about harmful objectives or deceptive strategies, it will externalize this process in natural language tokens. However, if models develop the capacity for complex latent reasoning—performing sophisticated internal deliberation without outputting explicit reasoning tokens—this monitoring mechanism would be undermined.

The paper addresses two core questions:

1. Q1: How much reasoning can frontier models perform latently, without externalizing it in CoT?
2. Q2: How is this latent reasoning capacity changing over time?

The authors argue that without quantifying this "no-CoT" capability, safety evaluations may significantly underestimate the risk of models engaging in deceptive or misaligned behavior internally.

Methodology

Benchmark Suite and Elicitation

The authors constructed a suite of 43 benchmarks spanning over 30,000 questions across domains including mathematics, coding, puzzles, causality, theory-of-mind, and strategic reasoning. These benchmarks were converted to the UK AISI's Inspect Framework.

• Task Classes: The suite includes short-answer tasks, generative tasks (code/text without natural language reasoning), and multi-turn agentic tasks (tool calls only).
• No-CoT Enforcement: To measure latent reasoning, the authors strictly prevented models from outputting reasoning tokens before their final answer. They used benchmark-specific prompts, few-shot examples, and structured output constraints. If a model attempted to output reasoning in more than 5% of outputs, they enforced stricter constraints.

Metrics: Time and Token Horizons

To quantify latent reasoning capacity, the authors introduced two primary metrics:

1. 50%-Task-Completion Time Horizon (TH): The estimated human time required to complete a task that a model solves with a 50% success rate. This anchors model capability to human effort.
2. 50% Reasoning Token Horizon: The minimum number of reasoning tokens an anchor model (specifically o3-mini) needs to solve a task that the evaluated model solves without CoT at a 50% success rate. This provides a model-grounded, independent baseline.

Human Solve Time Estimation

A significant portion of the methodology involved estimating human solve times for benchmarks lacking existing data.

• Data Collection: For 12 benchmarks without existing data, the authors collected 733 problem attempts from 17 recruited expert participants.
• Calibration: They used Claude Opus 4.7 to estimate solve times for the remaining benchmarks, calibrating these estimates against real human data from 20 benchmarks. They modeled the uncertainty of these estimates using a conditional Gaussian distribution and propagated this noise through the analysis via hierarchical bootstrap resampling.

Statistical Analysis

• Logistic Fitting: For each model, they fitted a logistic curve to predict success rates as a function of human solve time (or token count) to determine the 50% threshold ($h_{model}$).
• Trend Modeling: They fitted an exponential trend to the 50% THs of models on the Pareto frontier (SOTA at release) to determine the doubling time of latent reasoning capabilities.
• Uncertainty Propagation: They used hierarchical bootstrap resampling (resampling benchmarks, problems, model runs, and human time estimates) to generate 95% confidence intervals for both per-model THs and the overall trend doubling times.

Key Contributions

1. Comprehensive Benchmark Suite: A curated suite of 43 benchmarks (over 30k questions) designed specifically to evaluate latent reasoning across diverse domains, implemented within the Inspect Framework.
2. Uncertainty Modeling: A novel method for modeling and propagating uncertainty in human task-duration estimates, combining real solve times, collected baselines, and calibrated model-based estimates.
3. Reasoning Token Anchor: A method for quantifying latent reasoning capacity relative to the minimum number of reasoning tokens an anchor model (o3-mini) requires, providing a metric independent of human time estimates.
4. Empirical Trends: A longitudinal analysis of 14 frontier models (from GPT-2 to GPT-5.5) revealing the growth rate of no-CoT capabilities.
5. Scaling Analysis: An ablation study on 35 open-weight models correlating THs with parameter counts, layer counts, and architecture types (dense vs. MoE).

Results

Growth of Latent Reasoning

• Doubling Time: The no-CoT 50% Time Horizon has been doubling approximately every 373 days (95% CI: 167–691 days) over the past six years.
• Current State: As of the latest frontier models (e.g., GPT-5.5), the estimated no-CoT TH exceeds 3 minutes, equivalent to roughly 1,500 reasoning tokens (based on the o3-mini anchor).
• Token Horizon Doubling: The reasoning token horizon doubles every 437 days (95% CI: 341–571 days). The difference in doubling times between the time and token anchors is attributed to a sub-linear power law relationship where token counts scale faster than human solve times as problem difficulty increases.

Projections

Extrapolating the median trend suggests:

• By 2028, no-CoT THs could exceed 7 minutes.
• By 2030, no-CoT THs could exceed 25 minutes.
• The authors emphasize that these projections carry substantial uncertainty, with wide confidence intervals (e.g., the 2030 projection ranges from 9 to 615 minutes).

Comparison with CoT Capabilities

The no-CoT TH doubling time (373 days) is roughly twice as slow as the doubling time for full-affordance agentic tasks with CoT reported by Kwa et al. (182 days). This divergence became notable after the release of GPT-4, suggesting that while CoT capabilities are advancing rapidly, the gap between explicit and latent reasoning is widening.

Scaling Laws

• Doubling the 50% TH requires a 4.2× increase in total parameters or a 1.3× increase in layer count.
• Architecture Differences: Dense models show a faster scaling relationship (2.2× parameter increase for doubling TH) compared to Mixture-of-Experts (MoE) models (8.1× parameter increase), potentially explaining the slower no-CoT scaling observed in some proprietary models.

Robustness

The results are robust to:

• Excluding any single model from the trend fit.
• Using only benchmarks with real human solve times (though this increases noise).
• Including longer-form generation and multi-turn agentic tasks (which slightly reduces the doubling time to ~319 days).
• Different noise models for human time estimation.

Significance and Claims

The paper claims that frontier models have developed the capacity to perform tasks requiring minutes of human effort without externalizing their reasoning. This capacity has grown predictably, doubling roughly annually since 2019.

The authors argue that this trend has critical implications for AI safety:

1. Monitoring Limitations: CoT monitoring is only effective if reasoning surfaces in the output. The existence of a rapidly growing "headroom" for latent reasoning suggests that models could increasingly perform complex, potentially misaligned reasoning internally without detection.
2. Call to Action: Frontier developers should explicitly track latent reasoning abilities (no-CoT THs) as models advance. The authors note that measuring these capabilities is relatively cheap and does not require large amounts of inference compute.
3. Uncertainty Reduction: Reducing uncertainty regarding future models' THs is a safety priority in itself.

The paper concludes that while current models are not necessarily "routing" reasoning away from CoT, the capacity to do so is growing predictably, necessitating new evaluation and monitoring strategies that account for latent deliberation.