Technical Summary: Decoupling Reconnaissance and Exploitation in LLM-Based Web Penetration Testing

Problem Statement

While Large Language Models (LLMs) show promise for automating cybersecurity tasks, existing evaluations of LLM-based penetration-testing agents rely heavily on end-to-end black-box testing. This approach suffers from error cascading: failures in early reconnaissance (e.g., misinterpreting unstructured telemetry or failing to identify a vulnerability) prevent the agent from reaching the exploitation phase. Consequently, an agent's actual capability to generate and execute exploits is masked by earlier perception errors, leading to statistical false negatives. This makes it difficult for researchers to distinguish whether a failure stems from a lack of factual knowledge, a flaw in the reasoning architecture, or an inability to parse environmental noise.

Methodology

To address these evaluation bottlenecks, the authors propose a two-stage decoupled evaluation framework that separates autonomous reconnaissance from vulnerability exploitation.

1. Decoupled Framework Architecture

The framework aggregates the traditional penetration testing pipeline (gathering, scanning, verification, exploitation) into two orthogonal phases:

• Reconnaissance Phase: The agent attempts to identify the target vulnerability (e.g., producing a correct CVE identifier) using probing tools and interpreting unstructured logs.
• Exploitation Phase: The agent attempts to construct and execute a payload. Crucially, if the agent fails the reconnaissance phase, a Ground-Truth Injection mechanism intervenes. The framework pauses execution, injects the authentic vulnerability context (CVE and details) into the prompt, and allows the agent to proceed to exploitation. This isolates exploitation performance from reconnaissance noise.

2. Experimental Setup

• Dataset: A high-fidelity benchmark of 70 web vulnerability testbeds reconstructed from Vulhub and Vulfocus, covering OWASP Top 10 and CWE standards (e.g., SQL Injection, IDOR, Deserialization).
• Agents: Five open-source penetration-testing frameworks with heterogeneous architectures were evaluated:
  • Hexstrike-AI (Multi-agent, MCP-driven)
  • CyberStrikeAI (Monolithic)
  • RedAmon (Graph-driven, Multi-agent)
  • NeuroSploit (Constrained, Symbolic Tree)
  • Decepticon (Graph-driven, Multi-agent)
• Metrics:
  • Targeted Vulnerability Recall (TVR): Measures the agent's ability to autonomously identify the correct vulnerability during reconnaissance.
  • Functional Success Rate (FSR): Measures the agent's ability to successfully trigger the exploit (e.g., reverse shell) after ground-truth injection.
• Ablation: Experiments were conducted with and without a Verified Empirical Knowledge Base (VEKB) to isolate the impact of external knowledge on code generation and tool invocation.

Key Results

1. The Capability Gap

The study reveals a substantial gap between an agent's ability to find a vulnerability and its ability to exploit it when given the context:

• Exploitation Capability (FSR): When provided with accurate vulnerability context (via ground-truth injection), agents achieve a Functional Success Rate of up to 90.0% (specifically Hexstrike-AI).
• Reconnaissance Capability (TVR): Under autonomous conditions without prior knowledge, the Targeted Vulnerability Recall plateaus at approximately 50.0%.
• Implication: Many "failures" in end-to-end tests are actually reconnaissance failures masking successful exploitation logic. The agents often fail to parse unstructured telemetry rather than failing to construct the exploit.

2. Architectural Trade-offs

Different architectures demonstrated distinct strengths based on the vulnerability class and interaction complexity:

• Multi-Agent Systems (e.g., Hexstrike-AI): Enforcing state isolation via protocols like MCP proved most effective for long-sequence interactions (e.g., insecure deserialization) and handling noisy error logs. They achieved the highest FSR (90.0%) and were robust against attention drift.
• Monolithic Architectures (e.g., CyberStrikeAI): Performed better on short-chain injection flaws where context collapse is less likely, but struggled with long interaction chains.
• Graph-Driven Designs (e.g., RedAmon, Decepticon): Showed strength in cross-session access-control vulnerabilities (e.g., IDOR) due to their ability to maintain global state and compare privilege boundaries across personas.
• Constrained Architectures (e.g., NeuroSploit): Relied on symbolic trees and regex, which reduced hallucinations but limited payload mutation, making them suitable for deterministic N-day verification but less flexible for novel scenarios.

3. Impact of Knowledge and Model Scale

• Knowledge Injection: Integrating the VEKB significantly improved performance across all frameworks, reducing omission rates in reconnaissance and fixing low-level protocol errors in exploitation.
• Model Scale: Simply increasing the base model size (e.g., switching from Gemini 3 Pro to GPT-5) did not guarantee better performance. In monolithic frameworks, larger models sometimes suffered from attention dispersion when processing verbose error telemetry without state isolation.

Significance and Claims

The paper claims to provide the first fine-grained benchmarking protocol that decouples reconnaissance from exploitation to accurately measure the capability boundaries of LLM-based offensive agents.

• Diagnostic Value: The framework allows researchers to attribute failures specifically to "perception errors" (reconnaissance) versus "execution errors" (exploitation), moving beyond the "black box" of end-to-end success rates.
• Design Guidance: The results suggest that future automated offensive security agents must prioritize strict state isolation (e.g., via multi-agent designs) to handle the noise of real-world environments, rather than simply scaling up base models.
• Transition in Security: The work underscores a shift in LLM-based security from static code generation toward autonomous exploit execution, where the ability to maintain context amidst adversarial noise is the primary bottleneck.

The authors note that while their benchmark uses disclosed CVEs, the decoupled paradigm offers a foundation for future work on zero-day scenarios by potentially injecting abstract vulnerability semantics rather than specific CVE identifiers.