Endless Terminals: Scaling RL Environments for Terminal Agents

1. Problem Statement

Reinforcement Learning (RL) for Large Language Model (LLM) agents has shown success in domains like mathematics and coding, but it faces a critical bottleneck in terminal agents (agents that interact with a command-line interface).

• The Bottleneck: Current terminal benchmarks (e.g., TerminalBench) were designed for evaluation, not training. They contain only hundreds of manually curated tasks, which is insufficient for the data-hungry nature of RL.
• Limitations of Prior Work: Existing approaches rely on:
  • Repurposing fixed benchmarks for training (risking overfitting).
  • Supervised Fine-Tuning (SFT) via distillation from stronger proprietary models (inheriting the teacher's ceiling and requiring expensive API access).
  • Human-curated datasets (limited by annotation cost and scale).
• The Gap: There is a lack of a fully autonomous pipeline capable of generating an endless stream of diverse, verifiable terminal tasks without human annotation.

2. Methodology: Endless Terminals Pipeline

The authors introduce Endless Terminals, a procedural generation pipeline that synthesizes terminal-use tasks autonomously. The pipeline operates in four distinct stages (illustrated in Figure 2 of the paper):

Phase I: Task Description Generation

• Process: A language model is prompted to generate task descriptions paired with "privileged ground truth" data.
• Diversity: Prompts sample randomly from three dimensions:
  • Categories: File operations, log management, data processing, scripting, database operations, etc.
  • Complexity: Ranging from simple 2-3 command tasks to multi-step sequences (5-10 commands).
  • Context: Scenarios like a DevOps engineer debugging logs or a database administrator managing backups.
• Output: The model produces a user-facing task instruction and a hidden "truth" section containing exact file contents, paths, and expected states for verification.

Phase II: Container Setup & Validation

• Environment Creation: The system generates a container definition (Docker or Apptainer) to establish the required environment.
• Iterative Validation: An iterative refinement loop is used. The model generates a setup script; the system builds the container and runs prerequisite tests (checking for required files, directories, processes).
• Correction: If tests fail, the error output is fed back to the model for correction (up to 3 rounds). Tasks failing to produce a valid container are discarded.

Phase III: Completion Test Generation

• Verification Logic: A second test file is generated to verify the system state after task completion.
• Logic: These tests check for created files, modified configurations, and computed results based on the privileged ground truth.
• Sanity Check: The system verifies that these tests do not pass in the initial state, ensuring they measure actual task completion rather than trivial success.

Phase IV: Solvability Filtering

• Validation: To ensure tasks are solvable, the system samples 16 solution attempts from a capable frontier model (o3) using the agent framework.
• Filtering: Tasks are retained only if at least one solution succeeds (pass@16 > 0). This removes underspecified or impossible tasks.
• Result: This filtering step discards roughly 50% of generated candidates, leaving a high-quality, solvable dataset.

3. Agent Architecture and Training

• Minimal Scaffolding: The agent uses a simple interaction loop with no retrieval, no multi-agent coordination, and no specialized tools (e.g., no vim or htop).
• Interaction Loop:
  • The model receives conversation history (reasoning + shell outputs).
  • It outputs a command wrapped in <command>...</command> or a completion signal <command>done</command>.
  • The environment executes the command, returns stdout/stderr/exit code, and the loop repeats.
• Training Algorithm: Vanilla Proximal Policy Optimization (PPO).
  • Reward Signal: Binary episode-level reward (1 if final tests pass, 0 otherwise). No intermediate shaping or KL penalties.
  • Hyperparameters: 16 turns per rollout during training, max 2048 tokens per turn, temperature 0.6.
• Base Models: Llama-3.2-3B, Qwen2.5-7B, and Qwen3-8B-openthinker-sft (the latter pre-trained on 15k distilled traces).

4. Key Results

The pipeline generated 3,255 verified tasks spanning file operations, log management, data processing, and more.

Performance on Held-Out Development Sets

Training on Endless Terminals yielded substantial gains across all model sizes:

• Llama-3.2-3B: Improved from 4.0% to 18.2%.
• Qwen2.5-7B: Improved from 10.7% to 53.3%.
• Qwen3-8B-openthinker-sft: Improved from 42.6% to 59.0%.

Transfer to Human-Curated Benchmarks (TerminalBench 2.0)

Crucially, the improvements transferred to a completely unseen, human-curated benchmark:

• Llama-3.2-3B: 0.0% → 2.2% (outperforming base and other RL variants).
• Qwen2.5-7B: 2.2% → 3.4%.
• Qwen3-8B-openthinker-sft: 1.1% → 6.7%.
• Note: These results outperformed alternative approaches, including models with complex agentic scaffolds (e.g., Terminus-2).

Failure Mode Analysis

On TerminalBench 2.0, the primary failure modes were:

1. Loop Failures (39%): The model repeats the same command sequence after an error. Successful runs showed significantly higher command diversity (0.49) compared to failed runs (0.18).
2. Turn Exhaustion (26%): Hitting the turn limit.
3. Domain Gaps: Zero success on mathematics, machine learning, and model-training tasks, indicating the procedural generation pipeline currently lacks coverage in these specialized domains.

5. Significance and Contributions

• Scalability over Sophistication: The paper demonstrates that simple RL setups succeed when environments scale. The gains came not from complex algorithms or scaffolds, but from the sheer volume and diversity of procedurally generated, verifiable tasks.
• Autonomous Pipeline: It provides the first fully autonomous pipeline for terminal task generation that does not rely on human annotation or distillation from stronger models for the training data itself.
• Complementarity of SFT and RL: The results suggest that SFT (distillation) provides a "warm start" that RL can effectively build upon, with the best results seen in the model that combined both.
• Community Impact: The work argues that the field must invest in automated task generation pipelines alongside algorithmic advances to unlock the full potential of self-improving agents.

6. Limitations and Future Work

• Task Realism: Generated tasks resemble competitive programming problems rather than the "fuzzy," underspecified requests of real users.
• Capability Ceiling: The solvability filter relies on a frontier model (o3), meaning the pipeline cannot generate tasks beyond the current state-of-the-art capabilities.
• Reward Sparsity: The binary reward signal is sparse; future work could explore partial rewards or learned world models to improve sample efficiency.