Technical Summary: Long-Horizon-Terminal-Bench

Problem Statement

Current AI agents, built on Large Language Models (LLMs), have demonstrated proficiency in short, well-scoped terminal tasks (e.g., fixing a specific code issue or issuing a few shell commands). However, existing terminal benchmarks (such as Terminal-Bench and SWE-Bench) largely focus on these short-horizon problems, typically evaluated within minutes and graded solely on a binary "pass/fail" outcome. This evaluation paradigm creates two critical gaps:

1. Underestimation of Difficulty: Short horizons fail to capture the complexity of real-world workflows that require hundreds of steps, sustained planning over hours, and iterative debugging.
2. Sparse Reward Signals: Binary grading ignores intermediate progress. An agent that completes 90% of a complex workflow but fails the final step receives the same score as an agent that fails immediately, obscuring the agent's actual capability and making it difficult to diagnose specific failure modes (e.g., premature stopping vs. timeout).

Methodology

The authors introduce Long-Horizon-Terminal-Bench (LHTB), a benchmark designed to stress-test agents on long-horizon, domain-specific terminal tasks using dense, process-based reward grading.

Task Construction

• Scope: The benchmark consists of 46 tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing.
• Format: Tasks follow the Terminal-Bench formulation: a containerized Docker environment with a natural language instruction, relevant files/tools, and an oracle solution. Agents interact entirely via the terminal.
• Complexity: Tasks are explicitly designed to require hundreds of steps and execution times ranging from tens of minutes to hours. They involve reading/modifying code, running long scripts, and inspecting partial outputs.
• Difficulty Calibration: Tasks were built by creating "deliberately broken" terminal-only projects. To prevent overfitting, public checks are minimal (validating only CLI behavior and simple examples), while the majority of the reward is derived from hidden stress cases (e.g., nested manifests, schema variations, injected noise) that require robust, generalizable solutions.

Grading Mechanism: Dense Rewards

Unlike binary evaluation, LHTB employs a subtask-based grading scheme:

• Each task is decomposed into semantically meaningful subtasks ($s_1, \dots, s_K$).
• A deterministic grader assigns a normalized score $r_k \in [0, 1]$ to each subtask based on objective evidence (files, outputs, test results).
• The final reward $R$ is a weighted average: $R = \frac{\sum w_k r_k}{\sum w_k}$.
• This allows for partial credit, capturing how far an agent progresses even if it fails to reach the final goal.

Evaluation Setup

• Models: 15 frontier models were evaluated (including GPT-5.5, DeepSeek V4 Pro, Kimi K2.7 Code, etc.).
• Harness: The Terminus-2 agent harness was used for most models, interacting via a single long-horizon terminal session.
• Constraints: Tasks have a 90-minute wall-clock timeout.
• Metrics: Pass@1 (at thresholds $R \ge 0.9, 0.95, 1.0$), mean normalized reward, episodes per task, execution time, and estimated cost.

Key Results

The evaluation reveals that long-horizon execution remains a significant bottleneck for even the most advanced models.

• Low Success Rates: Even the strongest model, GPT-5.5, achieved a pass rate of only 15.2% at the $R \ge 0.95$ threshold and 10.9% at perfect completion ($R=1.0$). The mean pass rate across all 15 models was just 4.3% at $R \ge 0.95$.
• Resource Consumption: Tasks are extremely demanding, averaging 231 episodes, 9.9 million tokens, and 85.3 minutes of execution time per run.
• Cost Efficiency: There is a wide variance in cost-efficiency. GPT-5.5 is the most expensive ($21/task) but most effective. Hy3 anchors the low-cost frontier ($2.5/task) with moderate success. Many models incur high costs with negligible success, indicating that increased inference spending alone does not solve long-horizon failures.
• Failure Modes:
  • Timeouts: 79% of unresolved runs ended due to the 90-minute budget expiring while the agent was still working, often with low mean rewards (0.10–0.35).
  • False Finishes: A significant portion of "early exits" (agents stopping voluntarily) occurred at high reward levels ($R \ge 0.75$) where the agent incorrectly judged the task as complete. This highlights a weakness in self-verification and stopping criteria.
  • Binary vs. Dense: Under strict binary grading ($R \ge 1.0$), 10 of the 15 models solved zero tasks. Dense grading revealed that 62.8% of runs made meaningful partial progress, distinguishing models that were "close" from those that failed immediately.

Significance and Claims

The paper claims that Long-Horizon-Terminal-Bench is necessary to move beyond the limitations of outcome-only evaluation. Its primary contributions are:

1. Revealing Hidden Progress: Dense rewards expose "near-misses" and partial progress that binary grading obscures, providing a more nuanced view of agent capabilities.
2. Identifying the Bottleneck: The results suggest that the primary limitation for current agents is not local reasoning correctness (which short benchmarks measure) but reliable long-horizon completion. Agents struggle with budgeting time, maintaining state over long trajectories, and calibrating stopping decisions.
3. Benchmarking Reality: By requiring hundreds of steps and hours of execution, LHTB provides a more realistic stress test for autonomous agents than prior benchmarks, showing substantial headroom for improvement.
4. Resource for Future Work: The authors release the benchmark and evaluation harness to support the development of agents capable of planning, verifying, and executing reliably over extended horizons.

The authors conclude that while current agents can perform local steps correctly, they lack the robustness to sustain progress and verify completion over the long horizons required for practical, real-world workflows.