🤖 AI

Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture The Flag Challenges

This paper introduces DeepRed, an open-source benchmark that evaluates LLM agents on realistic Capture The Flag challenges using partial-credit scoring and automated log analysis, revealing that current models achieve limited success (averaging 35% checkpoint completion) and struggle with non-standard discovery and long-horizon adaptation tasks.

Original authors: Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek, Roland Vízner, Arie van Deursen, Maliheh Izadi

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek, Roland Vízner, Arie van Deursen, Maliheh Izadi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a brilliant, hyper-intelligent robot detective. You give it a map, a magnifying glass, and a list of tools, and you ask it to solve a series of locked-room mysteries. The question isn't just, "Did it find the treasure?" but rather, "How far did it get before it got confused?"

This paper, titled "Do Agents Dream of Root Shells?", is about testing exactly that: how good are today's AI "detectives" at solving cybersecurity puzzles called Capture The Flag (CTF) challenges?

Here is the breakdown of their research, explained through simple analogies.

1. The Setup: A Safe "Playground" for AI

The researchers built a digital playground called DeepRed.

The Attacker: They put an AI agent inside a virtual computer (a "Kali" machine), which is like giving the detective a fully stocked toolbox and a uniform.
The Target: They connected this to a separate, isolated virtual machine containing the "mystery" (the CTF challenge).
The Rules: The AI can only talk to the target through a text terminal (like an old-school command line) and can occasionally use a search engine. It cannot cheat by looking at the answer key or breaking the walls of the simulation.

Think of it like putting a chess grandmaster in a room with a locked safe. They can try every combination, pick the lock, or hotwire the door, but they can't just walk out the back door.

2. The Problem: The "All-or-Nothing" Trap

In the past, researchers only cared if the AI solved the whole puzzle.

The Old Way: If the AI found the flag, it got 100 points. If it failed at step 1, it got 0.
The Reality: Current AI is like a student who can solve the first half of a math problem but gets stuck on the final equation. Under the old system, that student gets a zero, even though they did 80% of the work. This makes it hard to see if the AI is actually getting smarter.

3. The Solution: The "Partial Credit" System

The authors introduced a new way to grade the AI, similar to a teacher grading a long essay.

Checkpoints: Instead of just looking for the final "Flag," they broke every challenge down into smaller milestones (e.g., "Found the server," "Got a username," "Escalated to admin").
The Grading Machine: Since humans can't watch every single second of the AI's work (it would take forever), they built an automated grading system.
- Step 1 (The Summarizer): An AI reads the messy, long log of what the detective did and writes a neat summary.
- Step 2 (The Judge): Another AI reads that summary and checks off the boxes: "Did it find the server? Yes. Did it get the password? No."
The Result: They can now say, "This AI got 35% of the way there," which is much more useful than just saying "It failed."

4. The Findings: The "Smart but Clumsy" Detective

They tested 10 different commercial AI models on 10 different puzzles. Here is what they found:

The Score: The best AI only managed to complete about 35% of the checkpoints on average. None of them solved the puzzles from start to finish.
The Strength: The AIs were good at the "easy" stuff. If the puzzle was a standard web hack (like finding a hidden door in a website), the AI did well.
The Weakness: The AIs struggled with long-term planning.
- Analogy: Imagine the AI is a dog chasing a ball. It runs fast to get the ball (finds the first clue), but then it forgets why it's running, runs in circles, or gives up when the ball rolls behind a bush.
- They often failed to connect the dots between steps. They would find a clue, forget it, and start over.
The Cost: Some models used a massive amount of "brain power" (tokens) to get very little done, while others were efficient but still got stuck.

5. Why This Matters

This paper is a reality check.

For Security: It shows that while AI is getting better at hacking, it's not yet a "super-hacker" that can autonomously break into complex systems without human help. We don't need to panic about AI taking over the internet today, but we need to keep watching.
For AI Research: It proves that we need to stop grading AI only on "Did it win?" and start grading them on "How did it think?" The new "partial credit" system helps researchers see exactly where the AI is failing so they can fix it.

The Bottom Line

The AI agents are like promising apprentices. They have the tools and the knowledge to start the job, but they lack the patience and the long-term memory to finish the whole house. The researchers built a better ruler to measure their progress, showing us exactly where they are stumbling so we can help them learn to finish the job.

1. Problem Statement

While Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, their capabilities in realistic offensive settings remain poorly understood. Existing evaluations often rely on binary success metrics (solved/unsolved), which are too coarse for current-generation agents that rarely solve complex challenges end-to-end but frequently make measurable partial progress. Furthermore, many benchmarks lack the isolation and realism required to simulate actual penetration testing, often relying on curated settings or containerized environments that do not reflect the persistence and statefulness of real-world CTF (Capture The Flag) scenarios.

The authors address the research question: To what extent can LLM-based agents autonomously solve CTF challenges across different categories?

2. Methodology: The DeepRed Framework

To answer this, the authors developed DeepRed, an open-source benchmark harness and agent framework designed for isolated, reproducible, and fine-grained evaluation.

A. Architecture and Environment

Virtualized Isolation: Unlike container-based approaches, DeepRed uses full Virtual Machines (VMs) to ensure strong isolation. It deploys a Kali Linux attacker VM and a Target Challenge VM connected via a private internal network.
Agent Interface: The agent interacts exclusively through a terminal interface (via a getty service exposed over a Unix domain socket) and an optional web search tool (DuckDuckGo). Direct references to flags or known writeups are filtered from search results to prevent leakage.
Agent Implementation: The framework uses SmolAgents, allowing models to write Python code to coordinate command execution, maintain state, and implement multi-step logic, rather than relying solely on JSON-based tool calling.
Reproducibility: After each run, the Kali VM is restored to a clean snapshot, and the target VM is reset, ensuring independent and reproducible episodes.

B. Dataset Curation

Challenge Selection: The authors curated 10 VM-based CTF challenges from HackMyVM, covering categories such as web exploitation, privilege escalation, fuzzing, SSH key manipulation, steganography, and service exploitation.
Criteria: Challenges were selected based on solvability via terminal, availability of public writeups (for grading rubrics), and the presence of clear intermediate milestones.
Difficulty: The set focuses on low-to-moderate difficulty to ensure agents can make measurable progress, as current models struggle with high-difficulty end-to-end solutions.

C. Evaluation Strategy: Partial-Credit Scoring

Recognizing that binary "solved" metrics are insufficient, DeepRed introduces a partial-credit scoring system:

Checkpoint Decomposition: Each challenge is broken down into a sequence of binary checkpoints derived from public writeups (e.g., identifying a service, recovering credentials, obtaining an initial shell, privilege escalation).
Automated Labelling Pipeline: To scale evaluation, the authors implemented a two-stage LLM process:
- Summarization: Raw execution logs (which can be very long) are condensed into a structured step-by-step summary by a "Summary LM."
- Judging: A "Judge LM" evaluates the summary against the challenge-specific rubric to assign pass/fail labels for each checkpoint.
Validation: The pipeline was validated against human annotators. The best configuration (Gemini 3 Flash for summarization + Claude Sonnet 4.6 for judging) achieved a Cohen's kappa ( $\kappa$ ) of 0.7234 with human raters, demonstrating high reliability.

3. Key Contributions

DeepRed Benchmark: An open-source, extensible architecture for evaluating LLM agents in realistic, isolated CTF environments with full execution trace logging.
Partial-Credit Evaluation: A novel methodology that decomposes challenges into intermediate checkpoints, enabling fine-grained performance assessment rather than binary outcomes.
Automated Labelling Pipeline: A validated "summarize-then-judge" pipeline that reduces human evaluation time from hours to minutes while maintaining high agreement with human experts.
Comprehensive Benchmarking: The first large-scale evaluation of 10 commercially accessible LLMs across 10 diverse CTF challenges, providing a baseline for current agentic capabilities in offensive security.

4. Results

The authors evaluated 10 LLMs (including GPT-5.1 Codex Max, Gemini 2.0 Flash, and open-weight models) on the 10 challenges.

Overall Performance: Current agents remain limited. The best-performing model, GPT-5.1 Codex Max, achieved only 35% average checkpoint completion. The weakest model (Nemotron-3 Nano) achieved only 5%.
Token Efficiency: High token consumption did not correlate with better performance. For instance, GPT-5.1 achieved the best score with fewer tokens than several weaker models.
Challenge Difficulty:
- Easier: Challenges following standard "web foothold $\to$ privilege escalation" patterns (e.g., Whitedoor, Quick, SysAdmin) were solved more frequently.
- Harder: Challenges requiring non-standard discovery, long-horizon planning, or specific tool manipulation (e.g., Fuzzz, Thirteen, Todd) were significantly harder, with completion rates dropping below 15%.
Failure Modes:
- Lack of Perseverance: Agents often gave up prematurely or failed to adapt after initial failures.
- State Management: Agents struggled to maintain context over long horizons, frequently repeating failed commands or failing to escalate privileges after gaining initial user access.
- Planning: Agents excelled at reconnaissance but struggled with synthesizing observations into a coherent, multi-step exploitation strategy.

5. Significance and Implications

Beyond Binary Metrics: The paper demonstrates that binary success rates obscure significant differences in capability. Partial-credit scoring reveals that agents can perform meaningful intermediate tasks (reconnaissance, enumeration) even if they fail to capture the final flag.
Security Risks: The study highlights that while current agents are not yet fully autonomous in solving complex CTFs, they can perform specific offensive sub-tasks. This underscores the need for robust defensive measures and careful deployment of agentic systems in security pipelines.
Future Directions: The results suggest that future improvements in LLM agents for cybersecurity must focus on long-horizon planning, explicit memory systems, and hierarchical architectures rather than just raw reasoning or tool integration.
Reproducibility: By releasing the framework, datasets, and evaluation scripts, the authors enable the community to track the evolution of agentic capabilities and conduct rigorous failure analysis.

In conclusion, DeepRed provides a critical infrastructure for moving beyond hype in AI security, offering a rigorous, scalable, and nuanced way to measure how close LLM agents are to becoming truly autonomous offensive security tools.