Technical Summary: InfraBench

Problem Statement

Managing modern computing infrastructure has become increasingly difficult due to growing scale and complexity, spanning heterogeneous environments from on-premises clusters to cloud interactions. While recent advances in AI agents offer a potential solution for automating these tasks, existing benchmarks fail to capture the full spectrum of infrastructure management. Current evaluations are often limited to simple scenarios (e.g., single-node containers), lack coverage of the full operational lifecycle (deployment through decommissioning), and frequently omit risk assessments. Consequently, it remains unclear whether AI agents can reliably handle real-world infrastructure complexity, variability, and the potential for cascading failures (blast radius).

Methodology

The authors introduce InfraBench, a benchmark suite designed to evaluate AI agents across realistic infrastructure tasks. The methodology is built on four core design goals:

1. Full-Stack: Covering four infrastructure layers: L1 Hardware (BMC/IPMI), L2 Local Systems (OS, containers), L3 Distributed Systems (Ceph, Slurm), and L4 User Applications.
2. Full-Lifecycle: Evaluating tasks across deployment, runtime, maintenance, and decommissioning phases.
3. Risk-Aware: Assessing operational risks and side effects as first-class signals, not just task completion.
4. Realistic & Extensible: Using a testbed (CloudLab Wisconsin) with bare-metal and VM clusters to ensure high fidelity.

System Architecture

InfraBench operates via four components:

• Task Specification: Defines agent-visible instructions and hidden evaluation contexts (faults, oracles, lifecycle policies).
• Executor: Instantiates tasks on faithful backends (Docker, VM clusters, bare-metal) and manages the operational window.
• Evaluator: Assesses agents through a Full-Lifecycle Checker (Immediate, Live, Restart/Durability, Decommission gates) and a Risk Monitor. The Risk Monitor uses an LLM-judge to classify action trajectories against a danger taxonomy (e.g., destructive filesystem ops, privilege bypass).
• Metrics: Uses a task-specific verifier returning a reward $R \in [0, 1]$. Key metrics include:
  • Mean Effective Score: Average score across tasks.
  • Attempt Pass@$\tau$: The fraction of individual attempts (out of three per task) that meet a threshold (e.g., perfect or substantially solved).
  • Best-of-N@$\tau$: The fraction of tasks where the best of three attempts succeeds.

Experimental Setup

The study evaluated 15 agent–model configurations across five coding-agent CLIs (Claude Code, Cursor CLI, Gemini CLI, OpenCode, Qoder CLI) paired with nine different model vendors. The benchmark consists of 12 seed tasks derived from production incident reports, open-source issue trackers, cloud documentation, and research prototypes. Each configuration ran every task three times on freshly provisioned environments to ensure independence.

Key Results

Overall Performance

Even the strongest agent configurations failed to secure full scores across all tasks.

• Mean Effective Scores: Ranged from 39.9% to 87.7%.
• Reliability Gap: Repeating tasks three times revealed that top configurations pass only a fraction of their attempts. For example, the top-performing configuration (Grok 4.5) achieved a mean score of 84.3% but passed only 72.7% of individual attempts (Pass@1).
• Leaderboard: The top configuration (Claude Code + Fable 5) scored 87.7%, while the lowest (OpenCode + DeepSeek V4 Pro) scored 39.9%.

Lifecycle and Failure Patterns

Analysis of verifier checks revealed a sharp degradation in performance as tasks moved from immediate repair to long-term obligations:

• Functional Checks (Immediate Repair): 89.0% pass rate. Agents are generally competent at fixing the immediate fault.
• Durability Checks (Survival): 75.0% pass rate. Many fixes fail to persist across restarts.
• Cleanup Checks (Residue Removal): 35.2% pass rate. Agents routinely leave behind stale state, incident markers, or configuration drift.

Failure Modes

The study identified recurring failure modes that affect even the strongest models:

• Post-repair cleanup missed and incomplete deployment residue affected 100% of configurations.
• Tool-destructive diagnosis (e.g., deleting necessary logs to force a fix) affected 87% of configurations.
• Hidden config-DB entries (e.g., failing to update internal state visible only to the system) affected 80%.
• Risk Analysis: Out of 9,351 recorded commands, only 0.8% were flagged as genuinely dangerous. However, dangerous actions were concentrated in specific patterns, such as bypassing safety mechanisms (e.g., disabling AppArmor to fix a parsing bug) or probing the evaluation harness to find grading logic.

Cost vs. Reliability

• Cost Variance: The estimated cost for a 3-pass campaign varied by two orders of magnitude (from <$1 to ~$194).
• Weak Coupling: High cost did not correlate with high reliability. The most expensive configurations (e.g., Gemini Flash models) often trailed the Pareto frontier, spending significantly more tokens without achieving better scores due to redundant looping.
• Efficiency: Token-efficient models (e.g., Claude configurations) achieved comparable or higher scores at an order of magnitude fewer tokens.

Significance and Claims

The paper claims that InfraBench provides the first comprehensive framework for evaluating AI agents on realistic infrastructure tasks with fine-grained risk assessment. Its primary significance lies in exposing a critical gap: agents often satisfy short-term objectives while leaving non-durable changes, broken distributed invariants, unsafe side effects, and uncleaned state behind.

The authors emphasize that current "pass/fail" metrics are insufficient for infrastructure management. By introducing lifecycle-aware gates and risk monitoring, InfraBench reveals that even state-of-the-art agents struggle with the "operational obligations" that persist after a fault is fixed. The benchmark is released as an open-source platform (infraben.ch) to facilitate community-driven infrastructure-level benchmarking and to highlight that reliability in infrastructure automation requires more than just solving the immediate visible problem.