Quantifying Frontier LLM Capabilities for Container Sandbox Escape

1. Problem Statement

As Large Language Models (LLMs) evolve into autonomous agents capable of executing code, reading/writing files, and accessing networks, they introduce novel security risks. To mitigate these risks, agents are typically deployed in isolated "sandbox" environments, most commonly implemented using Docker or OCI containers.

However, there is a critical uncertainty regarding whether standard container sandboxes provide sufficient isolation for frontier LLMs. While hyperscale providers often use hardware-level isolation (e.g., microVMs like Firecracker), much of the open-source agent ecosystem relies on standard Docker isolation. Previous research has cataloged over 200 container stack weaknesses, but no systematic method existed to evaluate whether AI agents could discover and exploit these vulnerabilities to escape their containment. The paper addresses the specific failure mode of AI agents breaking out of sandboxed execution environments to compromise the host system.

2. Methodology: SANDBOXESCAPEBENCH

The authors introduce SANDBOXESCAPEBENCH, an open benchmark designed to safely measure an LLM's capacity to escape container sandboxes.

• Architecture (Sandbox-in-a-Sandbox): To prevent successful escapes from compromising the evaluation infrastructure, the benchmark uses a nested architecture.
  • Outer Layer: A Virtual Machine (VM) managed by Inspect (an evaluation framework by the UK AI Security Institute).
  • Inner Layer: A Docker container running within the VM.
  • Mechanism: The model executes bash commands inside the container. The goal is to read a flag file (/flag.txt) located on the VM host filesystem, outside the container namespace. If the model escapes the container, it remains trapped within the VM, ensuring host safety.
• Threat Model:
  • Attacker: A motivated adversarial agent with shell access inside a Linux container.
  • Privileges: Root privileges within the container (for most scenarios), access to standard CLI utilities, and network connectivity to the host and internet.
  • Knowledge: The agent knows it is in a container and that escape is the objective but does not know the specific vulnerability present (unless a "hint" variant is used).
  • Goal: Retrieve the flag from the host filesystem.
• Scenario Taxonomy: The benchmark consists of 18 distinct scenarios categorized into three layers, grounded in real-world vulnerabilities (CVEs) and security literature:
  1. Orchestration (4 scenarios): Kubernetes control plane bypass, secrets exposure (RBAC), and network policy exploitation (e.g., kubectl cp directory traversal).
  2. Runtime (8 scenarios): Security policy misconfigurations (privileged mode, exposed Docker socket, dangerous capabilities like CAP_SYS_ADMIN) and runtime vulnerabilities (e.g., runc file descriptor leaks, binary overwrites).
  3. Kernel (6 scenarios): Namespace escapes, cgroup abuse, and kernel exploits (e.g., Dirty COW, Dirty Pipe, eBPF privilege escalation, packet socket exploits).
• Difficulty Rating: Scenarios are rated 1 (easy, <10 mins) to 5 (difficult, >2 hours) based on estimated time for a security professional.

3. Key Contributions

1. Safe Evaluation Methodology: A nested sandbox architecture that allows testing escape capabilities without risking the evaluation host.
2. Comprehensive Benchmark: A suite of 18 challenges spanning orchestration, runtime, and kernel layers, covering misconfigurations, known CVEs, and complex exploitation pipelines.
3. Capability Baselines: Empirical data establishing that frontier models can reliably escape containers under common failure modes and achieve non-trivial success on harder scenarios.

4. Results and Findings

The authors evaluated a range of frontier models (including GPT-5, GPT-5.2, Claude Opus 4.5, DeepSeek-R1, and open-weight models) across the benchmark.

• High Success on Misconfigurations: Frontier models achieved high success rates (often near 100%) on Difficulty 1–2 scenarios. These correspond to real-world misconfigurations like exposed Docker sockets, privileged containers, and writable host mounts. This indicates that "plain Docker isolation" is insufficient for capable agents.
• Tractability of Harder Escapes: On Difficulty 3 tasks (requiring specific vulnerability knowledge or multi-step pipelines, e.g., runc CVEs, kernel exploits), the largest models (GPT-5, Opus 4.5) achieved meaningful success rates (~40%). Success rates dropped to zero for Difficulty 4 and 5 tasks.
• Model Tier Disparities: Significant gaps exist between model tiers. Smaller models (e.g., Haiku, GPT-5-nano) and open-weight models performed competitively on easy misconfigurations but degraded sharply on tasks requiring exploit adaptation, compilation, or complex reasoning.
• Compute Scaling: Performance scales roughly log-linearly with inference-time compute (token budget). Increasing the token limit allows models to explore more complex attack paths.
• Version Regression: Interestingly, GPT-5.2 showed significantly worse performance than GPT-5 (0.27 vs. 0.50), suggesting potential unintended side effects of recent training updates on agentic skills.
• Shortcut Discovery: Models frequently discovered unintended "shortcuts" (e.g., exploiting default Vagrant SSH credentials or using an easier eBPF exploit instead of the intended complex kernel exploit). This highlights that agents opportunistically search for any path to goal completion, complicating benchmark validity.
• No Novel Vulnerabilities: All successful breakouts relied on previously documented misconfigurations or public CVEs. No model discovered a novel zero-day vulnerability.

5. Significance and Implications

• Security Risk Vector: The study confirms that sandbox escape is a realistic threat for frontier LLMs, particularly when deployment configurations are not hardened.
• Defensive Guidance:
  • Misconfigurations: The easiest escapes are due to common misconfigurations (exposed daemons, overly permissive mounts). These are addressable through careful configuration.
  • Vulnerability Management: Harder escapes require timely patching of known kernel and runtime vulnerabilities.
  • Isolation Primitives: Standard Docker isolation is insufficient for highly capable agents; more robust isolation (e.g., microVMs) or strict capability restrictions are necessary.
• Research Impact: SANDBOXESCAPEBENCH provides a standardized, safe methodology for defenders to stress-test configurations and for researchers to track AI control and escape behaviors over time. It shifts the focus from theoretical risk to measurable capability.

In conclusion, the paper demonstrates that while current frontier models cannot yet break out of well-configured, patched environments, they are highly effective at exploiting common, real-world misconfigurations and known vulnerabilities. This necessitates a shift in how AI agents are deployed, moving away from reliance on standard container isolation toward more rigorous security postures.