Towards Optimal Agentic Architectures for Offensive Security Tasks

1. Problem Statement

The current generation of Large Language Model (LLM) agents for offensive security (e.g., penetration testing) relies heavily on heuristic-driven architecture choices. Prior systems often fix a single coordination topology (e.g., a specific multi-agent workflow) without empirical evidence regarding when additional agents improve performance versus when they merely add cost, latency, and failure modes.

The core problem is the lack of a controlled, empirical framework to determine which agent coordination topology (single-agent vs. various multi-agent structures) is optimal under specific constraints (budget, observability, and task domain). The authors argue that architecture selection should be treated as a systems engineering question rather than a design intuition.

2. Methodology

The authors constructed a rigorous, controlled benchmark to evaluate agent architectures under matched conditions.

Benchmark Construction

• Targets: 20 intentionally vulnerable local targets:
  • 10 Web/API services (Python HTTP).
  • 10 Binary services (C TCP).
  • Each target exposes exactly one ground-truth vulnerability (distinct CWE) and is accessible via a network endpoint.
• Access Modes:
  • Whitebox: The agent has read-only access to the source code tree.
  • Blackbox: The agent interacts only with the live network endpoint (no source code).
• Models: Three distinct model families were tested: GPT-5.2, Claude Opus 4, and Kimi K2.

Architectures Evaluated

The study compares five coordination topologies (families) across 600 total runs (5 architectures × 3 models × 20 targets × 2 modes):

1. SAS (Single-Agent): One agent generates a list of claims and validates them sequentially.
2. MAS-Indep (Multi-Agent Independent): Three independent agents audit the target; the best validated result is selected (Best-of-N).
3. MAS-Decent (Decentralized Peer Voting): Three agents scan independently, vote on the top CWE, and the winner is validated.
4. MAS-Central (Centralized Hub): An orchestrator emits a claim list; a specialist validates only the top claim.
5. MAS-Hybrid (Hierarchical): Two branches (orchestrator and sandbox) generate candidates; the strongest validated result is selected.

Evaluation Metrics

• Detection-Any: The system identifies the correct vulnerability class (even without a working exploit).
• Validated Detection: The system identifies the correct class and provides a reproducible, validator-confirmed exploit (Proof of Concept).
• Efficiency: Cost per validated finding, Time-to-First-Validation (TTFV), and token usage.

3. Key Contributions

1. Controlled Benchmark: A novel dataset of 20 interactive targets (Web + Binary) with ground-truth vulnerabilities, evaluated in both whitebox and blackbox modes, specifically designed to isolate topology effects.
2. Comprehensive Empirical Study: Execution of 600 runs covering 5 topologies, 3 models, and 2 access modes, providing the first direct comparison of agent architectures in offensive security.
3. Cost-Quality Frontier Analysis: The paper establishes that there is no single "best" architecture. Instead, there is a non-monotonic trade-off frontier where broader coordination improves coverage but increases cost and latency.
4. Artifact Availability: Full code, prompts, target specifications, and evaluation artifacts are open-sourced to enable reproducibility.

4. Key Results

Overall Performance

• Validated Detection Rate: The system achieved an overall validated detection rate of 49.8% (58.0% for "Detection-any").
• Top Performer (Accuracy): MAS-Indep achieved the highest validated detection rate at 64.2%.
• Top Performer (Efficiency): SAS (Single-Agent) was the most efficient baseline, costing $0.058 per validated finding with a median TTFV of 53.0s.

Impact of Observability (Whitebox vs. Blackbox)

• Whitebox significantly outperformed Blackbox: 67.0% vs. 32.7% validated detection.
• The gap was most pronounced in binary targets, where blackbox detection dropped to near zero for many complex vulnerabilities.

Impact of Domain (Web vs. Binary)

• Web targets were significantly easier to audit than Binary targets: 74.3% vs. 25.3% validated detection.
• Binary targets often resulted in "Partial" outcomes (correct class identified but exploit failed), highlighting the difficulty of the "exploit loop" in binary contexts.

Architecture Trade-offs

• MAS-Central performed poorly (36.7% validated), suggesting that a single orchestrator bottleneck limits exploration.
• MAS-Indep provided the best coverage but at a higher cost ($0.143/finding) and latency (111.9s).
• SAS remained competitive in accuracy (50.8%) while being substantially cheaper and faster, challenging the assumption that multi-agent systems are always necessary.

Statistical Significance

• Bootstrap confidence intervals and paired target-level deltas confirmed that observability (white/blackbox) and domain (web/binary) are the dominant factors affecting performance, often outweighing the choice of topology.
• In blackbox modes, MAS-Indep significantly outperformed MAS-Central (+33.3 points), whereas in whitebox modes, the gap between topologies narrowed, suggesting that source visibility reduces the need for complex coordination.

5. Significance and Conclusion

The paper shifts the paradigm of agentic security from "building the most complex multi-agent system" to optimizing the coordination topology based on task conditions.

• Non-Monotonic Frontier: The study proves that "more agents" does not linearly equal "better results." Once latency, token costs, and exploit-validation difficulty are factored in, simple single-agent systems (SAS) often dominate centralized multi-agent designs, while independent parallel agents (MAS-Indep) are only superior when maximum recall is required in difficult (blackbox/binary) regimes.
• Adaptive Routing: The authors propose that future systems should not hardcode a topology but should learn to route tasks to the appropriate architecture (e.g., SAS for easy web targets, MAS-Indep for hard binary blackbox targets) based on observable task features.
• Safety & Ethics: The work emphasizes controlled evaluation on intentionally vulnerable targets to prevent misuse, while providing a rigorous framework to improve the reliability of autonomous security auditing.

In summary, the paper provides the empirical evidence needed to move offensive security agents from heuristic experimentation to data-driven architectural selection.