Technical Summary: AGENTCYBERRANGE

Problem Statement

Frontier AI systems are increasingly demonstrating capabilities in interactive cybersecurity tasks, such as codebase inspection, vulnerability detection, and exploit generation. However, the systematic evaluation of their autonomous offensive capabilities remains constrained by a lack of open, reproducible, multi-host cyber-range infrastructure.

Existing public benchmarks (e.g., CTFs, vulnerability reproduction, or single-exploit generation) often isolate individual skills from the end-to-end structure of realistic intrusions. Realistic cyber attacks involve an operational chain: discovering exposed services, gaining an initial foothold, collecting internal information, and expanding compromise across networked hosts through post-exploitation. Current benchmarks fail to capture this workflow, making it difficult to observe emerging cyber risks early under conditions that preserve the full attack lifecycle.

Methodology

The authors introduce AGENTCYBERRANGE, an open, multi-range evaluation infrastructure designed to measure the autonomous cyber attack capability of frontier AI systems in realistic environments. The system comprises two main components:

1. The Benchmark Suite

The benchmark covers two core stages of realistic attacks:

• Web Exploitation (WebExploitBench): Consists of 110 vulnerabilities across 15 real-world web applications (e.g., CMS, e-commerce, AI workflow engines). It includes real zero-day and one-day vulnerabilities, as well as synthetic vulnerabilities embedded in application workflows. Tasks require agents to explore hidden endpoints, infer parameters, and exploit vulnerabilities from the exposed attack surface.
• Post-Exploitation (PostExploitBench): Comprises 8 enterprise-like cyber ranges containing 156 internal hosts. These environments feature multi-layer network topologies, realistic host roles (DMZ, internal apps), and defensive pressures (honeypots, anti-virus, EDR-like monitoring). Tasks evaluate the ability to establish internal access, escalate privileges, recover credentials, and perform lateral movement.

Difficulty Levels: Both tracks define three difficulty levels to control information availability:

• Level-0: Minimal information (Target URL or Entry-point IP only).
• Level-1: Intermediate information (Vulnerable URLs or Network Topology).
• Level-2: High information (Vulnerability types or Concrete CVEs/Credentials).

2. The CAGE Evaluation Pipeline

To ensure scalable and reproducible execution, the authors built CAGE, a modular toolchain:

• Agent Adapters: Unify heterogeneous CLI agents (e.g., Codex, Claude Code) under a common interface.
• Agent Manager: Controls runtime lifecycles, isolates containers, and records execution traces.
• Benchmark Manager: Deploys web applications and cyber ranges in isolated environments, manages task states, and resets environments between runs.
• Verifier: Automatically checks reported success against runtime evidence (e.g., verifying SQL injection via canary strings, checking privilege escalation via file markers in /root).

Experimental Setup

The authors evaluated six frontier AI systems under matched prompts and step budgets:

1. GPT-5.5 (with Codex)
2. Claude-Opus-4.7 (with Claude Code)
3. GLM-5.1 (with Claude Code)
4. DeepSeek-V4-Pro (with Claude Code)
5. Qwen-3.7-Max (with Qwen Code)
6. Kimi-2.6 (with Kimi Code)

Tasks were limited to 150 steps for web exploitation and 500 steps for post-exploitation, with a two-hour timeout per task.

Key Results

Web Exploitation Performance

• GPT-5.5 with Codex achieved the highest success rates, solving 16.1% of tasks under Pass@3 (Avg.) and 28.18% under Pass@3 (Max) in the Level-0 (black-box) setting.
• Under Level-2 (with concrete hints), the success rate for GPT-5.5 increased to 33.0%.
• Behavioral Analysis: GPT-5.5 was the only agent to visibly utilize security-specific endpoint discovery tools (e.g., ffuf). Most other agents relied heavily on curl and python3.
• Limitations: Agents struggled with "vulnerability depth." Detection rates dropped from 35% at depth 2 to 11% at depth 6, indicating a failure to navigate deep application workflows. High run-to-run variance was observed, with many vulnerabilities found in only a single attempt.

Post-Exploitation Performance

• GPT-5.5 with Codex again led, achieving 31.7% Pass@3 (Avg.) in Level-0 and 46.3% in Level-2.
• Claude-Opus-4.7 faced significant safety refusals (12 trials stopped), highlighting the tension between safety alignment and offensive capability evaluation.
• Capabilities: Agents demonstrated the ability to chain steps (e.g., exploiting a service, recovering credentials, pivoting to internal hosts) and mutate payloads to bypass anti-virus defenses.
• Failures: Agents often wasted steps on non-vulnerable hosts and failed to systematically search for downstream credentials (e.g., failing to find GitLab credentials inside a compromised Confluence wiki). They also frequently triggered honeypots and left warning logs, lacking the stealth of human pentesters.

Out-of-Benchmark Findings

During evaluation, systems identified valid vulnerabilities not included in the benchmark reference set. Notably, GPT-5.5 discovered a zero-day arbitrary file write vulnerability in ComfyUI (a popular AI workflow engine), demonstrating that frontier AI can identify real-world risks beyond the intended test scope.

Significance and Claims

The paper claims that open, end-to-end cyber-range evaluation is necessary to observe emerging offensive capabilities under realistic conditions.

• Current State: Frontier AI systems are not yet reliable end-to-end attackers. They exhibit high variance, struggle with long-horizon planning, and lack stealth.
• Emerging Risk: However, they can already complete a non-trivial fraction of realistic attack tasks, including exploiting real web applications, navigating enterprise networks, and adapting to defenses.
• Implication: The authors argue that model release assessments must evolve to include controlled attack workflows connecting web exploitation with post-exploitation. Relying solely on isolated benchmarks (like CTFs) is insufficient to capture the operational risks posed by autonomous AI agents.

The work provides a foundation for tracking these capabilities over time and strengthening defenses against future autonomous AI-driven cyber threats. The benchmark and pipeline are open-sourced to facilitate reproducible research in this domain.