Technical Summary: How Inference Compute Shapes Frontier LLM Evaluation

1. Problem Statement

As frontier AI benchmarks saturate, evaluation tasks are shifting toward harder, longer-horizon problems requiring multi-step planning, tool use, and iterative problem solving. Performance on these tasks is increasingly sensitive to the amount and allocation of compute available at test time ("inference compute"). However, many existing evaluations report performance at a single, restrictive budget (e.g., fixed token limits or turn caps). This approach risks understating model capability, as a failure may indicate the model exhausted its inference budget rather than an inability to solve the task. Furthermore, fixed-budget scores may fail to track progress in a model's ability to convert additional inference-time compute into performance, particularly as newer generations emerge.

2. Methodology

The authors conducted a systematic study of inference scaling across seven challenging benchmarks spanning software engineering, mathematics, medicine, and cybersecurity. The study evaluated up to 12 frontier language models (released between May 2025 and March 2026) using a unified experimental framework.

Experimental Setup

• Models: Six frontier models from two families (Anthropic Opus and OpenAI GPT) were evaluated on five main benchmarks. Additionally, data from two cybersecurity benchmarks (involving 10 models) were analyzed for inference scaling.
• Benchmarks:
  • Stateful: TerminalBench 2.0, SWE-Bench Pro, FrontierMath, Cyber CTFs, The Last Ones.
  • Stateless: HealthBench, Humanity's Last Exam (HLE).
• Inference-Scaling Interventions: The authors applied three simple, reproducible interventions uniformly across all tasks:
  1. Expanded Token Budgets: Total trajectory budgets of 5M–30M tokens (1–3 orders of magnitude above typical defaults).
  2. Context Compaction: Replacing earlier turns with model-generated summaries to enable serial scaling beyond nominal context windows.
  3. Iterative Resubmission: Unlimited submission attempts (capped at 999) with a repetition guard to prevent unproductive loops.
• Feedback Conditions: Two conditions were tested for each model/benchmark pair:
  • No Feedback: Ambiguous acknowledgement only; the model must self-judge task completion.
  • Oracle Score Feedback: Explicit correctness signals (or partial credit for HealthBench); trajectories terminate upon the first correct submission.
• Analysis: The study measured cumulative performance as a function of total tokens consumed (inference scaling curves) and decomposed generational gains into Reach (tasks unlocked), Efficiency (tokens per solve), and Reliability (success rate on unlocked tasks). Parallel scaling (pass@k) was also analyzed by splitting fixed budgets across multiple independent trajectories.

3. Key Contributions and Results

3.1. Inference Scaling is Substantial but Benchmark-Dependent

The study finds that larger token budgets substantially improve performance on several benchmarks, but the magnitude of gains varies significantly:

• High Headroom: Benchmarks like FrontierMath, Humanity's Last Exam (HLE), and Cyber CTFs show substantial performance gains well beyond typical published budgets. For example, extending FrontierMath from 1M to 10M tokens yielded an average gain of +11.7%.
• Low Headroom: Software engineering benchmarks (TerminalBench, SWE-Bench Pro) and HealthBench showed limited improvement when budgets were extended. TerminalBench and SWE-Bench Pro saw gains of only ~1.3% and ~0.3% respectively when budgets were doubled or tripled. HealthBench showed minimal sensitivity to serial scaling interventions under the tested protocol.
• Plateauing: Diminishing returns vary by model generation and feedback condition. Newer models often continue to improve at larger budgets where older models plateau, suggesting fixed-budget evaluations may miss the capabilities of frontier models.

3.2. Generational Gains Driven by Reach and Reliability

Task-level decomposition reveals that cross-generational improvements at large budgets are driven primarily by:

• Increased Reach: Newer models unlock a larger proportion of tasks, particularly harder ones.
• Increased Reliability: Newer models solve unlocked tasks more consistently across repeated trajectories.
• Conditional Efficiency: Token efficiency gains (solving tasks with fewer tokens) are uneven across benchmarks and are often secondary to the expansion of the set of solvable tasks. Newer models do not consistently solve already reachable tasks more efficiently; rather, they solve new tasks that were previously inaccessible.

3.3. Protocol Choices Shape Elicitation

The value of inference scaling depends heavily on how compute is allocated and what feedback is provided:

• Repeated Submission: Improves performance on all benchmarks. The benefit is most pronounced when Oracle Feedback is available to guide continued search (e.g., HLE and SWE-Bench Pro).
• Serial vs. Parallel Scaling:
  • Parallel Scaling (splitting a fixed budget across multiple independent trajectories) yields significant gains on stateless benchmarks like HealthBench (+0.283 pass@10 gain) and HLE (+0.183), but minimal gains on stateful benchmarks like FrontierMath and SWE-Bench Pro.
  • Newer Models: Generally benefit less from parallel sampling than older models, suggesting they extract more value from a single, deep trajectory.

4. Significance and Recommendations

The paper argues that benchmark scores are protocol-dependent and that a single fixed-budget score provides an incomplete picture of frontier capability. The authors propose three key shifts in evaluation practice:

1. Report Capability as a Function of Compute: Evaluations should report performance curves across a range of inference-time compute rather than a single fixed-budget number. This distinguishes between near-saturated evaluations and those where additional compute yields measurable gains.
2. Explicit Protocol Specification: Choices regarding iteration, feedback, and compute allocation (serial vs. parallel) should be treated as integral parts of the evaluation design and reported explicitly, as they significantly alter measured performance.
3. Matched Budget Comparisons: When comparing model generations, especially for safety or policy-relevant settings, models should be evaluated over a large, shared compute range at matched budgets. Fixed-budget comparisons may increasingly misrepresent progress as models become better at utilizing extended inference time.

The study concludes that while simple inference-scaling interventions (larger budgets, context compaction, resubmission) can elicit significant capabilities, the extent of these gains is highly dependent on the task structure and the specific protocol used. Therefore, evaluations must move beyond single-point metrics to characterize the full potential of frontier models under varying inference-time constraints.