Technical Summary: CurveShift: Is Agent Progress Scalar?

Problem Statement

Progress in large language models (LLMs) and agents is frequently summarized using a single scalar metric, such as an aggregate benchmark score, a latent ability estimate, or a time-horizon doubling rate. While these summaries capture overall performance trends, they conflate two distinct phenomena:

1. Level Shift: A uniform upward shift in overall capability where performance improves proportionally across all task difficulties.
2. Shape Change: A disproportionate gain concentrated on difficult tasks, implying a qualitative change in the difficulty-response curve (e.g., "emergent" abilities on hard tasks).

Current scalar summaries cannot distinguish between these two. If gains migrate toward harder tasks, it is unclear whether this reflects a true change in the model's capability profile (shape change) or merely an artifact of a rising scalar ability hitting ceiling effects on easier tasks (level shift). Furthermore, in agentic benchmarks, a confounding variable exists: newer models are typically evaluated with newer "harnesses" or scaffolds. This collinearity between model era and scaffold era makes it impossible to attribute gains to the model itself versus the evaluation infrastructure.

Methodology

The authors introduce CurveShift, an analysis framework that separates overall ability growth from difficulty-specific gains using Item Response Theory (IRT).

1. Data Sources and Identification Strategy

The study utilizes three datasets but identifies the core effect using LiveCodeBench, a competitive programming benchmark.

• METR Time-Horizon & SWE-bench: Agentic benchmarks where model era and scaffold era are collinear. These serve to demonstrate the "apparent" trend but cannot isolate the model effect.
• LiveCodeBench: A no-scaffold benchmark where models generate code directly from problem statements without an agent loop. Crucially, it possesses an exogenous difficulty ordering (based on human contest difficulty) independent of model performance. This allows the authors to separate model capability from scaffold improvements.
• Dataset: A panel of 66 dated models (spanning 2019–2026) evaluated on 1,055 problems (322 Easy, 383 Medium, 350 Hard).

2. Statistical Framework

The analysis employs a longitudinal IRT approach to test for Differential Item Functioning (DIF) across eras (pre- vs. post-September 2024, the "reasoning era" breakpoint).

• Null Hypothesis (Rasch Model): A single rising ability parameter ($\theta_m$) explains all performance changes. The probability of success is $logit(p) = \theta_m - b_i$.
• Alternative Hypothesis (Era-DIF Model): A post-era indicator interacts specifically with hard items. The model is:
  $$logit(p_{mi}) = \theta_m - b_i + \delta \cdot post_m \cdot hard_i$$
  Here, $\delta$ represents the additional log-odds of success on hard items for post-era models, beyond what their overall ability predicts.

3. The CurveShift Decomposition (Anchored Sensitivity Analysis)

A standard two-parameter logistic (2PL) model fails on this data because item discrimination ($\alpha$) and the era effect ($\delta$) are nearly aliased, leading to unstable estimates. To resolve this, the authors use an anchored sensitivity design:

1. Freeze Ability: Estimate model ability ($\hat{\theta}_m$) using only Easy and Medium items. This prevents hard-item residuals from being absorbed into the ability estimate.
2. Pin Discrimination: Fit the model on Hard items with $\hat{\theta}_m$ fixed as an offset. Instead of estimating discrimination freely, they pin the hard-item discrimination parameter ($\alpha_{hard}$) to a grid of values: $\{0.55, 0.83, 1.00\}$.
   • $\alpha_{hard} = 1.00$: Assumes equal discrimination between hard and easy items (conservative headline).
   • $\alpha_{hard} = 0.83$: A debiased estimate.
   • $\alpha_{hard} = 0.55$: Represents the deflated discrimination artifact from binarization.
3. Estimate $\delta$: Report the era effect $\delta$ across this grid. Uncertainty is quantified via cluster bootstrapping (clustering by item, contest, and model family).

Key Results

1. The Scalar Null Explains Most "Migration"

On the METR Time-Horizon data, the observed migration of gains toward harder tasks (where the fastest-improving difficulty band shifts from easy to medium/hard over time) is largely reproduced by a simple Rasch model with a single rising ability.

• Mechanism: As overall ability rises, easy items hit the ceiling (flattening the slope), while intermediate items sit on the steepest part of the logistic curve. This creates an apparent shift toward harder tasks without any actual change in the difficulty-response curve shape.
• Conclusion: The "migration" is primarily a measurement artifact of ceiling effects and scalar growth, not a qualitative change in capability.

2. A Residual Hard-Item Effect Survives

After controlling for the scalar level shift using the CurveShift decomposition on LiveCodeBench, a small but significant residual effect remains:

• Magnitude: Under the conservative assumption of equal discrimination ($\alpha_{hard} = 1.00$), post-era models (released after September 2024) show a gain of +0.40 logits on hard items beyond what their easy/medium performance predicts.
• Impact: This raises the solve rate on the hardest problems from approximately 18% to 25%.
• Robustness: The effect remains positive across the entire grid of discrimination values up to $\alpha_{hard} \approx 1.22$. It holds across different model families (though led by the strongest reasoning series) and different contest sources (Codeforces and AtCoder).
• Specificity: The effect is specific to hard tasks requiring short reasoning paths (competitive programming) and is not observed as a general property of long-horizon autonomy in agentic benchmarks where scaffolds confound the signal.

Significance and Claims

The paper claims to offer a correction to the narrative of agent progress by disentangling "level" from "shape":

1. Deflation of Emergent Claims: The apparent shift of gains toward harder tasks is largely an artifact of scalar ability growth and ceiling effects, similar to how "emergent abilities" were previously shown to be artifacts of non-linear metrics.
2. Isolation of a Real Effect: Despite the deflation, a genuine, smaller shape change exists. Models released after September 2024 solve the hardest problems more often than their performance on easier tasks would predict, even when controlling for overall ability.
3. Benchmark Validity: The study demonstrates that standard agentic benchmarks (METR, SWE-bench) cannot resolve this question due to the collinearity of model and scaffold eras. Only a no-scaffold benchmark with exogenous difficulty ordering (LiveCodeBench) can isolate the model's native capability.
4. Scope of Claim: The authors explicitly limit their claim to short-horizon reasoning tasks (competitive programming). They do not claim this effect extends to long-horizon autonomous agents where scaffolding is integral to the task definition.

The paper concludes that while the aggregate trend is largely scalar, a "shape change" concentrated on the hardest items is a real, measurable phenomenon specific to the post-September 2024 era, driven primarily by the strongest reasoning models.