Technical Summary: DeployBench

Problem Statement

While Large Language Model (LLM) agents have demonstrated progress in software engineering and research tasks, current evaluations often assume the existence of a pre-configured, runnable environment. In practice, reproducing research artifacts released with academic papers remains a significant bottleneck. Setting up these environments from scratch involves resolving version-drifted dependencies, aligning system-level components (e.g., GPU/CUDA stacks, custom Linux kernels), and patching legacy code against modern toolchains. Existing benchmarks for environment setup fail to capture the full scope of this challenge, as they often rely on containerized builds, static checks, or narrow Python-based AI repositories, neglecting system-level dependencies and legacy artifact compatibility.

Methodology

Benchmark Construction (DeployBench)

The authors introduce DeployBench, a multi-domain benchmark comprising 51 research-artifact deployment tasks spanning AI/ML, computer systems, and scientific computing. The tasks are drawn from top-tier venues between 2008 and 2025. Key characteristics of the benchmark include:

• Diversity: Tasks cover 11 programming language ecosystems (including Python, C/C++, Rust, Fortran, etc.) and include 22 GPU-dependent workloads, 5 tasks requiring custom Linux kernel compilation and booting in QEMU, and 10 legacy artifacts (2011–2018) requiring compatibility repair.
• Environment: Tasks are executed on fresh, bare-metal Linux cloud VMs with no pre-installed drivers or dependencies. Docker is explicitly disallowed to ensure agents perform native dependency resolution and system setup.
• Inputs: Agents receive the research paper, the code repository, and a clean VM.
• Verification: Success is determined by a hidden, two-layer verifier:
  1. Layer 1 (Global): A rule-based log parser detecting common failures (dependency errors, compilation crashes, out-of-memory errors).
  2. Layer 2 (Task-Specific): Executes a lightweight version of the paper's designated experiment (e.g., training a model for one epoch) and validates expected outputs and runtime evidence.
• Reference: All tasks were validated by the authors manually constructing reference deployments to ensure reproducibility.

Evaluation Setup

The authors evaluated four state-of-the-art LLMs using the OpenHands agent scaffold:

• GPT-5.3-Codex
• GPT-5.4-Mini
• Gemini-3.1-Pro
• Grok-4.20

Agents were tasked with setting up the environment, running a simple test, and generating a RUNBOOK.MD. A separate diagnostic agent was used post-failure to attribute root causes.

Key Results

Overall Performance

Even the strongest model, GPT-5.3-Codex, achieved a pass rate of only 51.0%. The other models scored significantly lower: Gemini-3.1-Pro (27.5%), Grok-4.20 (11.8%), and GPT-5.4-Mini (7.8%). This indicates that end-to-end autonomous deployment of research artifacts remains a highly challenging task.

Failure Analysis

Analysis of 154 failed runs revealed five recurring root-cause patterns:

1. Dependency and package resolution issues.
2. GPU/CUDA setup failures (missing drivers, mismatched libraries).
3. Native build and toolchain compilation errors.
4. Artifact and output mismatches.
5. System/VM-level execution failures (QEMU, kernel boot).

Completion Judgment as a Dominant Failure Mode:
A critical finding is that 97 of 154 failures (63%) were "agent-terminated self-stops." In these cases, the agent decided the task was complete based on internal checks that were either:

• Self-validation drift: The agent validated a different or weaker target than the paper-specific task (e.g., running a toy model instead of the paper's actual experiment).
• Shallow checks: The agent relied on imports, --help flags, or partial runbooks without executing the real artifact.
  Only 12 of the 97 self-stops involved a meaningful check of the actual target, and even then, the check often failed to capture specific requirements.

Trajectory and Token Usage

• Token Usage: There was no direct correlation between token consumption and success. Gemini-3.1-Pro consumed the most tokens (2.55M avg) but had a lower success rate than GPT-5.3-Codex (2.15M avg).
• Recovery Behavior: Successful runs were more likely to inspect logs or adapt commands after failures (48.5%), whereas failed runs tended to enter repetitive loops or wait states (60.3%).
• Legacy vs. Recent: Legacy artifacts (pre-2020) showed lower success rates, primarily due to the need for compatibility repair against modern compilers and APIs, rather than the age of the paper itself.

Contributions and Significance

The paper claims the following contributions:

1. A Realistic Benchmark: DeployBench is the first benchmark to combine multi-language ecosystems, per-task specific verification (beyond generic build signals), system-level setup (GPU, kernels, VMs), and legacy artifact compatibility.
2. Identification of Gaps: The results highlight a significant gap between current LLM agents and the capability required for autonomous research deployment. The primary bottleneck is not just code generation, but completion judgment—the ability to determine if a setup truly satisfies the specific scientific requirements of a paper.
3. Diagnostic Insights: The benchmark provides a testbed for analyzing agent behaviors, revealing that agents often fail to distinguish between a "runnable" state and a "correctly configured" state.
4. Future Directions: The authors suggest that future agents require improved capabilities in dependency repair, system-level configuration, and, crucially, the derivation of verifier-like completion checks from research papers rather than relying on self-assessment or shallow smoke tests.

The authors conclude that DeployBench offers a realistic testbed for scientific research agents, demonstrating that while progress exists, fully autonomous deployment of diverse research artifacts from raw infrastructure is not yet a solved problem.