Technical Summary: The Horizon Gap in Long-Horizon LLM Agents

Problem Definition

The paper identifies a critical disconnect in current AI capabilities, termed the horizon gap: the distance between a frontier language model's ability to solve complex reasoning problems in a single forward pass and its ability to reliably complete tasks spanning hours or requiring hundreds of sequential steps when embedded in an agent loop. While single-step benchmarks show rapid progress, deployed agents frequently fail in deployment-specific ways: losing track of earlier decisions, declaring half-finished jobs complete, or drifting from their original goals.

The authors argue that the field's vocabulary often conflates three logically independent properties, obscuring the specific nature of these failures:

1. Long-horizon: A property of the task (the number of sequential decisions required).
2. Long-context: A property of the model (the number of tokens it can attend to in one pass).
3. Long-term memory: A property of the system (whether information persists across steps or sessions).

The survey posits that the horizon gap is the primary bottleneck preventing durable real-world deployment of agent systems.

Methodology

The paper presents a systematic survey of 1,547 arXiv papers published between 2024 and 2026. The corpus construction involved a rigorous, two-stage filtering process to address the overloading of the term "agent" in literature:

1. Seed Harvest: A systematic query across eight thematic threads (planning, memory, execution, training, evaluation, orchestration, error accumulation, self-correction) yielded 1,939 unique papers.
2. Bleed Filter: A two-stage filter excluded 26.8% of raw hits (520 papers) deemed off-topic, primarily removing classical multi-agent reinforcement learning, robotic manipulation, and time-series forecasting work that does not focus on LLM-driven agents.
3. Targeted Supplement: A supplemental harvest added 128 papers specifically to address under-covered areas, particularly in theory, safety, and oversight.

The corpus was classified into a six-category taxonomy (Table 1) tracking the lifecycle of a long-horizon task, crossed with a second axis defining where the horizon is carried:

• Within-context: The full history fits in one forward pass.
• Within-task-beyond-context: The task exceeds the context window but remains one continuous episode (via summarization or external memory).
• Cross-task-persistent: Information or skills accumulate across distinct episodes or sessions.

The authors explicitly treat critical and diagnostic literature (papers demonstrating that benchmarks or methods fail to do what they claim) as a first-class thread throughout the survey rather than segregating it into a separate critique chapter.

Key Contributions and Technical Findings

1. Planning & Decomposition (§3)

The field has shifted from "plan-then-execute" (decomposing a task fully before acting) toward interleaved planning and search-based approaches as environmental uncertainty increases.

• Findings: Early commitment to a full plan is fragile in unpredictable environments. Methods like PIVOT (refining plans via feedback) and Tree of Thoughts (searching over candidate continuations) show that robustness often comes at the cost of compute.
• Limitation: The "cross-task-persistent" thread in planning is underdeveloped; storing planning experience often blurs into memory problems.

2. Memory & Context Management (§4)

Memory is organized along a persistence-fidelity trade-off:

• In-context: High fidelity but bounded by window size; suffers from "context rot" where information degrades or is lost in the middle of long prompts.
• External Store: Unbounded but introduces retrieval costs and interference (e.g., outdated or irrelevant information). The corpus shows a decisive shift toward external storage (294 papers) over context management (103 papers).
• Weights: Information baked into model weights is durable but requires training to update and is hard to audit.
• Security: Memory is identified as a new attack surface, susceptible to "memory poisoning" where adversarial inputs persist across sessions.

3. Execution Control & Recovery (§5)

This is the largest category (584 papers). The central finding is that harness engineering (orchestration, recovery logic) is often the binding constraint on task length, not the underlying model's per-step quality.

• Orchestration: The dominant scaling strategy is horizontal (splitting tasks across multiple agents) rather than vertical (deepening a single agent's loop).
• Recovery: Intrinsic self-correction (without external verification) is shown to be unreliable and can degrade performance. Effective recovery often requires external grounding or structured exception handling.
• Attribution: The ratio of orchestration papers to recovery papers (roughly 1.4:1) suggests the field invests more in scaling out than hardening error correction, despite evidence that the latter is the harder unsolved problem.

4. Training for Long Horizons (§6)

As horizons lengthen, outcome-only supervision (a single reward at the end) becomes sparse and noisy.

• Credit Assignment: The field is moving toward process reward models (PRMs) and segment-level credit assignment to provide denser signals.
• Implicit Signals: Algorithms like GRPO, typically described as outcome-based, are found to implicitly perform process-level credit attribution.
• Distinction: The paper distinguishes its scope from "recursive self-improvement" literature; the focus here is on training a policy to act over more steps, not on a system that rewrites its own training loop.

5. Evaluation & Measurement (§7)

Agentic benchmarks are evolving faster than the methodology to validate them.

• Validity Crisis: A significant portion of the literature interrogates whether benchmarks (e.g., SWE-bench) measure genuine capability or artifacts like test-suite weakness, data leakage, or memorization.
• New Metrics: The field is moving toward trajectory-level diagnostics (analyzing the path, not just the pass/fail outcome) and human-time-to-complete metrics to measure the actual horizon an agent can carry.
• Cross-Task Evaluation: Benchmarks that measure learning across sessions (e.g., SWE-Bench-CL) are rare but emerging.

6. Foundations, Limits & Safety (§8)

This section addresses the theoretical lack of a unified model for long-horizon degradation.

• Compounding Error: Real degradation does not follow a simple exponential decay model. Failures are often bimodal (agents either succeed or catastrophically derail) and exhibit "strained coherence" (pre-failure signals).
• Goal Drift: Safety constraints can be silently erased via context compaction, leading to goal drift.
• Oversight: Human-in-the-loop oversight is infeasible for long horizons; the literature proposes tiered, ensemble, or dynamic autonomy frameworks as alternatives.

Results and Significance

The paper does not propose a single new algorithm but offers a structural map of the field's response to the horizon gap. Its primary significance lies in three areas:

1. Disambiguation: It clarifies that "long-horizon," "long-context," and "long-term memory" are distinct axes, arguing that conflating them leads to misdiagnosed failures (e.g., citing a context-window paper to solve a memory-persistence problem).
2. The Harness vs. Model Attribution Problem: The survey highlights a critical open measurement problem: it is currently unclear how much long-horizon capability resides in the underlying model versus the "harness" (scaffolding, orchestration, recovery logic) wrapped around it. The corpus suggests the field is currently prioritizing harness engineering, but the boundary is not yet empirically resolved.
3. Correlated Measurement Bias: The authors identify a structural risk where the process-level signals used to train long-horizon agents and the signals used to evaluate them rest on shared, unverified assumptions about what constitutes "progress." If these assumptions are flawed, the entire field may be optimizing for a false signal.

Conclusion:
The survey concludes that the field has moved from outcome-based evaluation to trajectory-level analysis as the shared unit of analysis. While engineering solutions (better memory, orchestration, and process rewards) are advancing, a general predictive theory of long-horizon degradation does not yet exist. The most consequential open problems are disentangling model capability from harness engineering and establishing independent, non-correlated signals for training and evaluation to avoid systemic bias.