Technical Summary: TERMINALWORLD: Benchmarking Agents on Real-World Terminal Tasks

Problem Statement

Autonomous agents powered by Large Language Models (LLMs) are increasingly capable of automating complex terminal workflows by issuing commands, composing tools, and interpreting feedback within interactive Command Line Interface (CLI) sessions. However, reliably evaluating these agents on authentic, real-world terminal tasks remains an open challenge.

Current evaluation standards rely heavily on manually curated benchmarks (e.g., Terminal-Bench, LongCLI-Bench). These benchmarks suffer from three primary limitations:

1. Authenticity Gap: Experts often prioritize adversarial puzzles to maximize difficulty, causing tasks to diverge from the authentic, routine workflows developers actually perform.
2. Scalability Bottleneck: Manual curation is labor-intensive and struggles to keep pace with evolving terminal practices and diverse emerging tools, leading to benchmarks that are narrowly scoped and quickly outdated.
3. Validation Deficit: Recent automated synthesis methods attempt to bypass manual curation but are primarily designed for training data generation. They rarely undergo the rigorous validation required to guarantee true authenticity and test soundness.

Consequently, there is a lack of an evaluation system that is both authentic to real-world developer practices and scalable enough to evolve alongside them.

Methodology: The TERMINALWORLD Data Engine

To address these limitations, the authors introduce TERMINALWORLD, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. The engine operates on a corpus of 80,870 terminal session recordings harvested from the asciinema platform, a public repository where developers voluntarily share session transcripts.

The engine processes raw recordings through four systematic stages to transform noisy logs into rigorous benchmarks:

1. Collecting and Filtering Human Recordings

The engine retrieves transcripts and metadata from asciinema. A multi-stage filtering process ensures data quality and safety:

• Privacy & Safety: Exclusion of recordings containing Personally Identifiable Information (PII), sensitive credentials, or malicious commands (e.g., rm -rf *).
• Scope Definition: Discarding recordings involving Text User Interfaces (TUIs like vim or nano) or GUIs, focusing strictly on pure CLI workflows.
• Reproducibility: Removing sessions dependent on inaccessible URLs, Windows environments, or proprietary software that cannot be containerized.
• Quality Scoring: Using an LLM to filter out opaque or purely exploratory sessions, retaining only high-quality, deterministic workflows.
• Result: 9,492 high-quality recordings were retained for synthesis.

2. Synthesizing Terminal Tasks

Raw transcripts are noisy, containing typos, retries, and verbose system output. The engine uses an LLM (e.g., Claude Sonnet 4.6) to distill these into two artifacts:

• Task Instruction Formalization: The LLM infers the developer's underlying intent to generate an outcome-oriented instruction. Crucially, the instruction specifies the final state (e.g., "save blocked IPs to /app/result.txt") rather than the procedural path, preventing solution-specific hints from leaking into the task.
• Reference Solution Extraction: The LLM extracts a clean, executable Bash script from the transcript, removing duplicates and failed attempts to create a ground-truth reference solution.

3. Reproducing Executable Environments

Since raw recordings depend on the original developer's specific system state, the engine must reverse-engineer the execution environment.

• Environment Synthesis: An LLM agent (e.g., Claude Code) infers dependencies (base images, system packages, runtimes) from the reference solution to synthesize a Dockerfile and docker-compose.yaml.
• Execution-Based Refinement Loop: To ensure the environment is faithful and not hallucinated, the agent builds the Docker image, attempts to replay the reference solution, and uses runtime failures (e.g., missing libraries, unconfigured variables) as feedback to iteratively repair the Dockerfile.
• Result: This loop successfully reproduces executable environments for 5,035 tasks.

4. Generating Test Suites

To automatically judge task success, the engine generates a test suite within the reproduced Docker container using a trial-based refinement loop:

• Test Synthesis: An agent generates assertions based on the formalized instruction and the actual filesystem changes (state deltas) caused by the reference solution.
• Three-Trial Validation:
  1. AllPassing Trial: Runs the reference solution; all tests must pass (ensures solvability).
  2. Nop Trial: Runs nothing; all tests must fail (ensures non-triviality).
  3. Partial Trials: Runs truncated/ablated solutions; at least one test must fail (ensures discriminability against incomplete solutions).
• Result: Only tasks where the test suite passes all three trials are admitted. This process yields 1,530 validated terminal tasks for the full benchmark.

From this full set, the authors curated a VERIFIED subset of 200 tasks, manually reviewed by authors to ensure semantic alignment and correctness, serving as a rigorous testbed for frontier models.

Key Contributions

1. TERMINALWORLD Benchmark: A dataset of 1,530 validated tasks spanning 18 real-world categories (e.g., container orchestration, CI/CD, security) and covering 1,280 unique commands. Notably, 91% of these commands are absent from existing benchmarks like Terminal-Bench.
2. Automated Data Engine: A fully automated pipeline that reverse-engineers tasks from real-world recordings, ensuring the benchmark is "authentic and scalable by construction."
3. VERIFIED Subset: A manually vetted core of 200 tasks designed for high-fidelity evaluation of complex workflows.
4. Comprehensive Benchmarking: Extensive evaluation of eight frontier LLMs and six leading terminal agents on the VERIFIED subset.

Experimental Results

The authors evaluated frontier models (e.g., Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro) and agents (e.g., Terminus-2, Claude Code, OpenHands) on the TERMINALWORLD-VERIFIED subset.

• Performance Ceiling: Current systems struggle significantly with authentic terminal workflows. The best-performing model (Claude Opus 4.7) achieved a pass rate of only 62.5%.
• Efficiency Paradox: There is a weak negative correlation between resource consumption (turns, tokens) and success. Models often spend excessive compute exploring environments without making progress. Failed attempts consume significantly more resources (3.3× more tokens) than successful ones.
• Domain Specificity: Performance varies sharply by domain. Models perform well on environment setup (87.5%) but struggle with performance optimization (28.1%) and scripting/automation (39.1%).
• Agent Framework Impact: Agent frameworks primarily influence cost-effectiveness rather than the underlying capability ceiling. For instance, Terminus-2 and mini-SWE-agent achieved similar pass rates to Claude Code but at a fraction of the cost.
• Weak Correlation with Existing Benchmarks: Scores on Terminal-Bench 2.0 show only a weak correlation (Pearson $r = 0.20$) with TERMINALWORLD-VERIFIED scores. Models that dominate expert-curated challenges (e.g., GPT-5.5) often underperform on real-world workflows, suggesting existing benchmarks do not fully capture necessary capabilities.
• Divergence from Human Workflows: Agents rarely mimic the specific command paths of the original human recordings. The median command-set overlap between agent trajectories and human reference solutions is only 21.4%, indicating agents often solve tasks via valid but distinct alternative strategies.

Significance and Claims

The paper claims that TERMINALWORLD establishes a new standard for evaluating terminal agents by shifting from adversarial, expert-curated puzzles to authentic, scalable, real-world workflows.

• Authenticity: By reverse-engineering tasks from actual developer recordings, the benchmark captures the nuances of real-world dependencies, tool diversity, and workflow complexity that synthetic or manually crafted benchmarks miss.
• Scalability: The automated nature of the engine allows the benchmark to evolve continuously as new recordings are uploaded to asciinema, preventing the dataset from becoming stale.
• Diagnostic Value: The benchmark reveals that current state-of-the-art models and agents lack robust tool-use abilities in open-ended, real-world action spaces. It highlights that high scores on existing benchmarks may overestimate an agent's competence in practical, evolving terminal environments.

The authors conclude that TERMINALWORLD provides a necessary foundation for developing agents that are reliable in real-world terminal environments, moving the community beyond static, expert-curated evaluations.