Human vs Machine Mathematical Difficulty on Project Euler: An Experimental Analysis

Technical Summary: Human vs Machine Mathematical Difficulty on Project Euler

Problem Statement
This paper investigates how the effort and success probability of frontier AI systems scale relative to human difficulty on computational mathematics problems. Motivated by a proposal from Timothy Gowers regarding the scaling of proof-search algorithms, the authors seek to determine whether AI systems scale worse than humans as task difficulty increases. Specifically, they test the hypothesis that while machines may be faster on simple tasks, their expected running time grows much more rapidly than that of human mathematicians as the complexity of the problem (measured by proof length or solution trajectory) increases. The study utilizes Project Euler, a platform of computational math problems where human solve times are publicly recorded, and the MathArena benchmark, which provides extensive AI attempt data across 50 recent problems (943–992) and 26 model configurations.

Methodology
The authors operationalize "human difficulty" ($t_{human}$) as the geometric mean of the five fastest recorded human solve times for each problem, serving as a proxy for strong-human continuous work time. "Machine effort" ($t_{machine}$) is defined not by wall-clock time but by the total number of generated tokens required to achieve a successful answer, calculated as the total tokens across all attempts divided by the number of successful attempts.

The study employs three distinct analytical frameworks:

1. Hypothesis T (Time-Scaling Model): Tests a power-law relation $t_{machine} = a \cdot t_{human}^b$. The exponent $b$ indicates scaling behavior: $b > 1$ implies machines scale worse than humans (effort grows super-linearly with difficulty), while $b < 1$ implies machines scale better. Fits are performed on model-problem pairs with at least 50% empirical success rates to avoid dominance by failure cases.
2. Hypothesis P (Exponential Success Model): Tests whether the probability of success ($p_{success}$) decays exponentially with human task time: $p_{success} = p_{model}^{t_{task}}$. This is modeled as a linear relation between $\log p_{success}$ and $t_{human}$, constrained to pass through $(0, 1)$. Data is aggregated into quantile bins to fit the decay rate.
3. METR-Style Horizons: Fits a logistic curve to success probability as a function of human time to extract specific capability horizons: $h_{50}$ (human time at 50% success) and $h_{80}$ (human time at 80% success). This approach is adapted from the METR task-length benchmark [KWB+25].

The dataset comprises 3,840 attempts across 26 model configurations (25 included in final fits after excluding one non-representative configuration). Statistical uncertainty is quantified using nonparametric bootstrap resampling (1,000 samples) to generate 90% confidence intervals for fitted parameters.

Key Results

• Reversal of Scaling Prediction (Hypothesis T): Contrary to the initial prediction that machines would scale worse ($b > 1$), the study finds that for 20 of the 25 models with usable fits, the exponent $b < 1$. This includes the strongest high-coverage base models: Claude Opus 4.6 ($b=0.722$), GPT-5.4 (xhigh) ($b=0.644$), and Gemini 3.1 Pro Preview ($b=0.467$). The 90% bootstrap confidence intervals for these top models lie entirely below 1. This suggests that, within this operationalization, the token cost per successful answer grows sub-linearly with human difficulty, meaning the relative efficiency of machines improves as problems get harder, at least for problems where the model is moderately reliable.
• Exponential Decay of Success (Hypothesis P): The data shows moderate empirical support for the exponential decay model. The median bin-level $R^2$ across 22 well-covered configurations is 0.92, with many top models exceeding 0.95. The fitted half-lives (the human time at which success probability drops to 50%) range from 0.12 hours (Gemini 2.5 Pro) to 4.07 hours (GPT-5.4 xhigh).
• Capability Horizons: The strongest base model configurations reach 50% success horizons ($h_{50}$) between roughly 2.5 and 4.3 hours of human work time. Agentic wrappers (e.g., GPT-5 Euler Agent, Grok 4 Fast Euler Agent) demonstrate substantially larger horizons than their base models, though these comparisons are observational.
• Progress Trajectory: A log-linear fit through the state-of-the-art (SOTA) frontier of base models over the dataset's ~345-day span implies a doubling time of approximately 75 days for the $h_{50}$ horizon, increasing from 0.09 hours to 4.28 hours.

Significance and Claims
The paper claims that its findings challenge the specific "search-efficiency" narrative derived from Gowers's framing, which predicted that AI systems would fall behind humans on difficult problems due to rapidly scaling search costs. Instead, the results indicate that on Project Euler, frontier models scale better than strong human solvers in terms of token cost per success as difficulty increases.

However, the authors are cautious about the implications. They note that while scaling is favorable, the absolute reliability of these systems remains limited; models hit the 50% success threshold at human work times of only a few hours, far short of the sustained effort humans can apply to a single problem. The authors propose five speculative explanations for the discrepancy between favorable scaling on Project Euler and the lack of visible acceleration in research-level mathematics:

1. Implementation vs. Thinking: Project Euler problems heavily weight implementation (coding), which LLMs handle faster than humans, potentially masking slower mathematical reasoning.
2. Long-Trajectory Degradation: Success probability decays over time due to accumulated errors or "context rot," limiting the horizon of coherent reasoning.
3. Tool-Building Limits: LLMs excel at using standard tools but may lack the ability to invent new mathematical frameworks required for research.
4. Scaffolding Gaps: Current architectures lack the persistent memory and long-horizon workflows necessary for research.
5. Overthinking: Models may incur constant token overheads on easy problems (self-checking), artificially flattening the scaling slope.

The paper concludes that the three analyses (scaling, decay, and horizons) provide a coherent picture: the limiting factor for these systems on this benchmark is reliability (exponential decay of success) rather than search efficiency (sub-linear scaling of effort). The findings are presented as descriptive summaries of current capabilities on a specific benchmark, not as universal laws of AI progress.