Technical Summary: Autonomous LLM Agents & CTFs: A Second Look

Problem Statement

Despite the proliferation of Large Language Model (LLM) agents promising to automate offensive security tasks, recent claims of near-human success rates in Capture-the-Flag (CTF) challenges warrant scrutiny. The cybersecurity industry faces a significant workforce shortage, while attackers exploit new vulnerabilities rapidly. While LLMs possess knowledge of vulnerability classes and tools, offensive security requires multi-step planning, adaptive tool use, and coherent context management over long interaction horizons—capabilities that current agents struggle to demonstrate reliably. Furthermore, many prior studies rely on closed-source systems or proxy tasks (e.g., vulnerability recognition) rather than full exploit execution, leaving a gap in understanding the true capabilities and limitations of autonomous agents in penetration testing.

Methodology

The authors conduct a rigorous evaluation of LLM-based agents on a curated set of 30 web-based CTF challenges spanning 14 vulnerability classes (e.g., SQL injection, XSS, business logic flaws, race conditions). All challenges were manually solved by the authors to provide ground-truth reference solutions for fine-grained analysis.

Agent Architectures

The study compares three domain-specific, modular architectures against a general-purpose baseline:

1. Executor (E): A monolithic agent responsible for discovery, planning, and execution.
2. Executor + Evaluator (E+E): A multi-agent system where an Evaluator scores the Executor's proposed actions before execution, providing feedback for refinement.
3. Planner + Executor + Evaluator (P+E+E): A structured pipeline where a Planner (supported by a Recon Node) formulates a high-level attack plan, which is then executed by the Executor and monitored by the Evaluator. This design decouples planning from execution.
4. Baseline (claude-code): A production-grade, general-purpose agentic assistant (Opus 4.5) that autonomously determines its internal architecture, including the ability to spawn sub-agents.

Experimental Setup

• Models: The modular architectures were tested with GPT-4.1 and GPT-5. The baseline used claude-code (Opus 4.5).
• Execution: Each architecture was run three times per challenge (90 total runs per architecture) to measure consistency, while the baseline was run once per challenge due to token constraints.
• Environment: Agents interacted with isolated Dockerized web applications via SSH terminals, using run_command and run_python tools.
• Metrics: Success rate (max across runs), average interaction steps, cost (USD), and wall-clock duration.

Key Contributions and Results

1. Performance Ceiling and Baseline Comparison

The study reveals that current agents are still below human-level capability.

• Success Rate: Both the best engineered architecture (P+E+E with GPT-5) and the general-purpose claude-code solved exactly 19 out of 30 challenges (63%).
• Model Dependency: The LLM backbone is decisive; GPT-4.1 solved only 9/30 tasks, whereas GPT-5 reached 19/30, indicating that complex multi-step reasoning requires the latest generation models.
• Architecture vs. General Purpose: While claude-code achieved the same peak success rate as the most complex engineered solution without domain-specific engineering, the engineered solutions offered superior efficiency.

2. Efficiency and Consistency

While peak solvability was identical across strong systems, architectural complexity significantly impacted efficiency and reliability:

• Cost and Steps: The P+E+E architecture was the most cost-efficient, reducing average steps from 31.56 (Executor only) to 24.09 and lowering costs by 34% compared to the monolithic approach.
• Consistency: Multi-agent architectures demonstrated higher robustness. The P+E+E setup solved the same challenge in all three independent runs for 16/30 tasks, compared to 12/30 for the monolithic Executor.
• Runtime Overhead: Despite step efficiency, structured architectures incurred higher wall-clock times due to the overhead of instantiating multiple agents, whereas claude-code remained fast by rarely spawning sub-agents.

3. Failure Analysis and Bottlenecks

The study identifies that failures consistently cluster around specific categories, regardless of the agent architecture or model used.

• Primary Bottleneck: Vulnerability recognition is the main failure point. The Planner correctly identified the target vulnerability in 23/30 cases; in the remaining 7, neither the agents nor claude-code could formulate a plan. Once a vulnerability was recognized, exploitation succeeded in the vast majority of cases.
• Systematic Failure Modes:
  • Cross-Site Scripting (XSS): Failures were due to environmental limitations (lack of browser rendering for JavaScript execution) rather than reasoning errors.
  • Blind SQL/Command Injection: Failures stemmed from context window constraints. Long, iterative payload refinement degraded reasoning quality and consistency.
  • Business Logic Flaws: Agents consistently failed to reason about application workflows, defaulting to searching for standard technical exploits (e.g., injections) rather than understanding intended logic.
  • Race Conditions: Both systems failed to execute attacks requiring concurrent or synchronized actions, as the architectures operated sequentially and did not infer the need for parallelism.

Significance and Claims

The paper provides a "reality check" on claims of near-human performance in autonomous offensive security. Its primary significance lies in:

1. Establishing Strong Baselines: Demonstrating that general-purpose agents like claude-code are strong baselines that match the peak performance of hand-crafted, domain-specific multi-agent systems.
2. Identifying Architectural Value: Showing that while architecture does not increase the ceiling of solvable tasks, structured orchestration (specifically the Planner-Executor-Evaluator model) significantly improves consistency, cost-efficiency, and step reduction within the solvable subset.
3. Defining Limitations: Clarifying that current limitations are not merely architectural but stem from fundamental gaps in LLM capabilities:
   • Technical: Lack of environmental capabilities (browser rendering, concurrency).
   • Cognitive: Inability to reason about business logic and maintain coherence over long, stateful interaction chains.

The authors conclude that future progress requires addressing these specific environmental and cognitive barriers, particularly improving vulnerability recognition and long-term context management, rather than simply increasing architectural complexity.