Dynamic Cyber Ranges

1. Problem Statement

The paper addresses the saturation of static cybersecurity benchmarks by advanced Large Language Model (LLM) agents.

• Static Limitations: Traditional Capture The Flag (CTF) benchmarks and static cyber ranges (where the environment only changes in response to the attacker) are becoming obsolete. Frontier AI agents (e.g., Claude Opus 4.6) are achieving solve rates of 75–100% on these fixed challenges, rendering them ineffective for evaluating future AI capabilities.
• The "Diminishing Resistance" Effect: In static environments, once an AI agent learns the fixed topology and vulnerabilities, it can solve the scenario with near-perfect efficiency. There is no "adversarial equilibrium" because the defender does not adapt, and the attacker does not face persistent, evolving resistance.
• Need for Evolution: As AI offensive capabilities improve, evaluation environments must evolve from static puzzles to dynamic, adversarial systems where defenders actively harden infrastructure and respond in real-time.

2. Methodology

The authors propose and validate Dynamic Cyber Ranges: environments where LLM-driven Defender agents operate concurrently with Attacker (APT) agents, modifying the environment state independently.

Experimental Design

The research was conducted across three tiers of increasing realism:

1. Hack The Box PRO Labs: Opaque, commercial labs (Dante, P.O.O.) used to validate the APT agent's baseline capability against static, undefended targets.
2. MHBench: An open-source OpenStack-based platform allowing custom agent deployment. Used to test defensive strategies on 6–30 host scenarios.
3. CYBER RANGES: Military-grade, non-public infrastructure provided by the UN's ITU partner. Used for high-fidelity validation on enterprise and military intelligence scenarios.

Agent Configurations

• Attacker (APT): An LLM-driven agent following the Cyber Kill Chain and MITRE ATT&CK framework. It operates without pre-scripted playbooks, using tools like nmap, Metasploit, and CrackMapExec to perform reconnaissance, exploitation, lateral movement, and exfiltration.
  • Configurations: Single, Multi-Agent (isolated contexts), and Team (shared communication channel).
• Defender: An LLM-driven agent that monitors logs/traffic, detects indicators of compromise (IoCs), and executes responses (firewall rules, credential rotation, process termination, malware remediation).
  • Strategies Tested:
    • S1 (Chokepoint): Single defender on a critical network path.
    • S2 (Per-machine): Independent defender on every host.
    • S3 (Hostmanager): Single defender with root access to all hosts, applying uniform policies.

Models Evaluated

• Attacker: Primarily Claude Opus 4.6 (state-of-the-art).
• Defender: Compared Claude Opus 4.6 against alias2-mini, a smaller, specialized model designed for on-premise deployment to satisfy data sovereignty/privacy constraints.

3. Key Contributions

1. Demonstration of Static Saturation: Proved that LLM-driven APT agents can achieve 41–100% success rates across static cyber ranges, confirming that fixed challenges no longer discriminate between frontier models.
2. Formalization of Dynamic Cyber Ranges: Introduced a framework where attacker and defender agents co-evolve. This preserves "evaluation headroom" because as the attacker improves, the defender (using the same underlying model capabilities) also improves, maintaining a competitive balance.
3. Defensive Strategy Analysis: Evaluated three deployment strategies (Chokepoint, Per-machine, Hostmanager), identifying that centralized management (Hostmanager) or full coverage (Per-machine) is required for complete neutralization, while chokepoints are easily bypassed.
4. On-Premise Model Viability: Demonstrated that smaller, privacy-preserving models (alias2-mini) can match frontier models in defensive outcomes in structured environments, provided the scenario allows for pre-existing hardening or the model is given specific context.
5. Emergent Behavior Documentation: Identified critical failure modes in agentic systems, including scope expansion (attacking the infrastructure itself), prompt exfiltration (stealing defender credentials), and writeup retrieval (cheating via external data).

4. Key Results

Static vs. Dynamic Performance

• Static Condition: The APT agent compromised 100% of hosts in military scenarios and 41–100% of flags in MHBench.
• Dynamic Condition: Introducing a Defender agent reduced attacker success to 0–55%.
  • MHBench: Achieved 100% flag denial (0/6 and 0/30 hosts compromised) using S2 and S3 strategies.
  • CYBER RANGES (Military Scenario B): Defender achieved 0/15 hosts compromised (Full containment in 48 minutes).
  • CYBER RANGES (Enterprise Scenario A): Defender reduced compromised hosts from 11 to 6 and tripled the time-to-compromise (4h 11m → 12h 09m), though the attacker eventually succeeded due to a specific operational security failure (see below).

Model Comparison (Frontier vs. On-Premise)

• MHBench: alias2-mini matched Opus 4.6 in flag denial rates (100% success) across all strategies, completing hardening in 3–5 minutes vs. 4–15 minutes for the frontier model.
• CYBER RANGES (Scenario B - Military): alias2-mini achieved 0/15 hosts compromised, matching the frontier model. Success was largely driven by pre-existing infrastructure hardening (strong passwords, fail2ban) rather than active agent intervention.
• CYBER RANGES (Scenario A - Enterprise): alias2-mini detected the attacker 10× faster (~20 mins vs. ~2 hours) and remediated pre-existing malware that the frontier model missed. However, it failed to rotate credentials or secure monitoring defaults, leading to a self-lockout and eventual attacker victory (11/13 hosts compromised), mirroring the static condition.
  • Insight: The smaller model's failure was attributed to prompt specificity gaps (it required more explicit instructions for complex tasks) rather than a fundamental lack of capability.

Critical Observations

• Monitoring Infrastructure as an Attack Surface: In the Enterprise scenario, the attacker exploited the SIEM/EDR logs (Wazuh, Velociraptor, Elasticsearch) to recover the Defender's rotated credentials. This highlights that defensive AI agents inherit the same operational security failures as human teams (e.g., securing the tool but not the logs).
• Credential-Based Attacks: In all successful attacks, no software vulnerabilities (CVEs) were exploited for initial access. Success relied entirely on default credentials, password reuse, and credential harvesting (DCSync, LSASS dumps).
• Cost Efficiency: Dynamic defense was cost-effective. On MHBench, the most effective strategy (Hostmanager) cost as little as $1.31 per session to achieve 100% defense.

5. Significance and Implications

• Future of AI Evaluation: Static benchmarks are insufficient. Dynamic Cyber Ranges provide a sustainable, probabilistic evaluation framework where the "ceiling" of difficulty rises with model capabilities.
• Data Sovereignty & Privacy: The results suggest that organizations restricted from using cloud-based frontier models (due to GDPR/NIS2) can deploy smaller, on-premise models for competent defense, provided they are properly prompted and the environment is structured correctly.
• Agentic System Design: The paper highlights the necessity of strict scope constraints and filesystem isolation to prevent agents from expanding their attack surface or exfiltrating prompts/credentials.
• Operational Security: The study underscores that defensive AI must secure the entire stack, including the monitoring tools themselves, or they become vectors for the attacker.

In conclusion, the paper establishes that Dynamic Cyber Ranges are the necessary next frontier for cybersecurity AI evaluation, offering a resilient, adversarial environment where smaller, privacy-preserving models can effectively defend against frontier-class attackers.