Technical Summary: Benchmarking the Residual

Problem Statement

Long-horizon benchmarks for language-model agents consistently demonstrate that success rates decline as tasks extend in length. However, the paper argues that a raw decline in end-to-end performance is insufficient to diagnose the underlying failure mechanisms. A lower success rate on a long task may simply result from ordinary error compounding (the statistical probability of failure increasing with the number of independent stages) rather than a specific "long-horizon" limitation such as context degradation, state contamination, or planning collapse.

Current benchmarks often conflate three distinct sources of performance degradation:

1. Error Compounding: More stages create more opportunities for failure, even if individual stages are unchanged in difficulty.
2. Harder Local Work: Longer tasks may inherently contain more ambiguous or specialized stages.
3. History/State Dependence: Accumulated context, tool outputs, or prior errors make subsequent steps harder (termed trajectory-induced degradation or context rot).

Without distinguishing these factors, a declining performance curve cannot identify whether the agent fails because the task is long, because the stages are hard, or because the agent's history actively degrades its future performance.

Methodology

The paper proposes a diagnostic framework centered on the Horizon Residual ($\Gamma_H$), a metric that compares actual end-to-end performance against a baseline prediction derived from matched short-stage evaluations.

1. Decomposition and Checkpoints

The methodology requires a pre-specified decomposition of a long task into verifiable semantic stages (e.g., "find bug," "update API") rather than arbitrary turn counts.

• Checkpoints: The deployed agent (model, prompt, tools, memory) is run from declared checkpoints. A checkpoint includes the specific environment state, visible history, revealed information, and local budget.
• Conditional Probabilities ($q_i$): For each stage $i$, the probability of success is measured given that all prior required stages were completed and the agent started from the declared checkpoint: $q_i = \Pr(S_i = 1 \mid S_{<i} = 1, I)$.

2. The Product Baseline ($P_{expected}$)

Using the chain rule, the paper constructs an auditable baseline prediction for the full task success rate by multiplying the conditional stage probabilities:
$$P_{expected} = \prod_i q_i$$
This baseline assumes that if the agent can succeed at each stage independently under matched conditions, the full task success should be the product of those probabilities. The paper acknowledges that this is a "null model" that may not account for recovery, branching paths, or state changes, and suggests extensions (e.g., checkpoint simulators) for heterogeneous tasks.

3. The Horizon Residual ($\Gamma_H$)

The core metric is the log-ratio between the expected success (from the baseline) and the observed success ($P_{observed}$) of a natural end-to-end rollout:
$$\Gamma_H = \log \left( \frac{P_{expected}}{P_{observed}} \right)$$

• $\Gamma_H > 0$: The natural rollout performs worse than the matched short-stage prediction, indicating a mismatch (potential trajectory-induced degradation).
• $\Gamma_H \approx 0$: The natural rollout aligns with the prediction, suggesting performance decline is due to ordinary error compounding.
• $\Gamma_H < 0$: The natural rollout outperforms the prediction (e.g., due to feedback or recovery mechanisms omitted in the baseline).

4. Structural Annotations

To separate task structure from local difficulty, the paper advocates reporting two complementary annotations:

• Work Exposure ($N_{work}$): The number of required, separately verifiable stages.
• Dependency Depth ($H_{dep}$): The depth of the longest required dependency chain.
  This allows benchmarks to distinguish between tasks that are "long" due to volume (many independent steps) versus those that are "long" due to deep inter-stage dependence.

Key Contributions

1. Diagnostic Distinction: The paper argues for treating raw performance decline as a descriptive deployment result and the Horizon Residual as a separate, protocol-dependent diagnostic. It asserts that a claim of "long-horizon failure" requires evidence that the full rollout differs from a baseline built from matched short stages.
2. Auditable Baseline Proposal: It proposes a simple, auditable product baseline ($P_{expected}$) and outlines extensions (conditional models, checkpoint simulators) for tasks with branching or recovery. It emphasizes that the baseline must be declared before evaluating the natural rollout.
3. Reporting and Intervention Agenda: The paper articulates specific reporting requirements (decomposition, checkpoint states, budgets) and an intervention agenda. It posits that the residual is not a causal explanation but a signal to direct targeted experiments (e.g., history reset, state repair, context compression) to identify the specific mechanism of failure.

Results and Evidence

The paper does not present new experimental results but synthesizes existing literature to demonstrate the necessity of this framework:

• SWE-Milestone vs. Continuous Execution: Independent milestones often show >80% success, while continuous evolution on the same codebase drops to ~38%. The paper notes that without composing the independent scores into a chain-level counterfactual, this gap is ambiguous.
• ChainSWE: Shows a drop from 58.9% accuracy (oracle-correct prior state) to 36.5% (agent-generated prior state), illustrating the value of state repair but noting the lack of a composed chain-level prediction.
• SWE-Marathon: Shows that context compaction correlates with failure, but without matched interventions, it is unclear if compaction causes failure or merely marks failing trajectories.
• Illusion: Controlled history manipulations show that error rates in prior history reduce subsequent accuracy, isolating content over length.

The paper reviews representative benchmarks (SWE-EVO, NL2Repo-Bench, SWE-Chain, etc.) and concludes in Table 4 that no existing study implements the full sequence: matched local measurement, declared composition, natural rollout, state-compatibility handling, targeted intervention, and protocol-sensitivity analysis.

Significance and Claims

The paper's significance lies in reframing "longer is harder" from a descriptive observation into an auditable research program.

• Deployment vs. Diagnosis: It clarifies that deployment benchmarks ask if a system succeeds, while diagnostic benchmarks ask why it fails. The Horizon Residual bridges these by quantifying the gap between local competence and global performance.
• Causal Discipline: The authors claim that the residual is not a causal answer. A positive residual only indicates a mismatch with the chosen baseline. It directs researchers to specific hypotheses (e.g., "Is it context rot?") which must then be tested via targeted interventions.
• Falsifiability: The position is falsifiable if:
  1. The sign of $\Gamma_H$ is unstable across admissible protocols.
  2. Matched checkpoints cannot be constructed at manageable cost.
  3. The residual provides no incremental diagnostic information beyond existing persistent-vs-reset comparisons.

Ultimately, the paper argues that until a declining curve is accompanied by a pre-specified compositional prediction and the resulting residual, claims about long-horizon mechanisms remain unlicensed. The framework turns the observation of failure into a structured investigation of history, state, information, and recovery.