💻 computer science

Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing

This paper argues that future evaluations of autonomous penetration testing agents must establish controlled, model-matched plain-agent baselines to accurately distinguish performance gains derived from architectural innovations versus those attributable to underlying model improvements.

Original authors: Ananda Dhakal, Krish Neupane, Aarjan Chaudhary

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ananda Dhakal, Krish Neupane, Aarjan Chaudhary

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, ever-changing city of digital buildings. Some of these buildings have hidden traps and broken locks—security holes that bad actors could use to sneak in. For decades, the job of finding these holes belonged to human experts, like digital locksmiths who walk through every room, testing doors and windows. But the city is growing too fast; new buildings pop up every second, and the locksmiths can't keep up. So, scientists started building "robot locksmiths" using Artificial Intelligence. These robots are powered by Large Language Models (LLMs), which are like super-smart brains that have read almost everything on the internet and can write code, reason through problems, and follow instructions.

The big question in this field isn't just "Can the robot find the lock?" but "How much of the robot's success comes from its brain, and how much comes from the fancy tools we gave it?" Imagine you have a genius student. If you give them a simple notebook and a pen, they might solve a math problem. If you give them a high-tech calculator, a team of tutors, and a special map, they might solve it faster. But did the calculator make them smarter, or did it just help them? In the world of AI security, researchers have been building complex "harnesses"—teams of AI agents, special memory systems, and search tools—to help the AI hack websites. But because they keep changing the AI brain and the tools at the same time, nobody knew if the fancy tools were actually doing the heavy lifting, or if the AI was just getting better on its own.

This paper, titled "Baselines Before Architecture," steps in to settle the score with a very careful experiment. The researchers set up a controlled test using a standard set of 104 digital challenges called the XBOW benchmark. Instead of building a new, fancy robot, they took three existing, "plain" coding assistants—named Codex, OpenCode, and Pi—and asked them to solve the challenges using a single, powerful AI brain (GPT-5) and no special security tools. They wanted to see how well a "naked" AI could do before adding any extra gadgets.

The results were a mix of surprises and practical truths. First, the plain coding agents were surprisingly strong. The "Codex" agent, running on the GPT-5 model, solved about 67% of the challenges on its first try. When the researchers let it try again (running the same test twice and combining the results), it solved 77.9% of the challenges. This is a big deal because it means a simple, general-purpose coding robot can already handle a huge chunk of the work without needing a complex security team.

The study also tested two specific ideas that many people thought were essential. One was "security prompting," where you give the AI a special instruction sheet telling it to think like a hacker. The other was the idea that complex, multi-agent architectures (where different AI roles talk to each other) are necessary for top performance. The paper found that the special "hacker instructions" actually made the AI perform worse, solving fewer challenges and costing more money. As for the complex architectures, the paper compared the plain Codex results against two famous, high-tech systems called MAPTA and PentestGPT V2.

Here is the nuance: The fancy systems did still have a slight edge. When matched with the same AI brain, the specialized MAPTA system solved about 9.6 percentage points more challenges than the plain Codex. However, the paper showed that this gap wasn't as huge as it seemed. When the researchers upgraded the plain Codex agent to a newer, smarter brain (GPT-5.5), it solved 92.3% of the challenges on its first try and 95.2% when combining two runs. This new plain agent actually beat the headline scores of the older, complex systems, even without any of their fancy tools.

The main takeaway is a warning for the scientific community: Don't assume that a complex new architecture is the hero just because it gets a high score. Often, the score comes from the AI model getting smarter, not the tools getting fancier. The paper suggests that before claiming a new "security harness" is a breakthrough, researchers must first show how much better it is than a simple, plain coding agent using the exact same AI brain. While specialized tools can still help save money and time, the "plain" AI is already doing most of the heavy lifting, and simply waiting for the next generation of AI models might be just as effective as building a more complicated robot.

Technical Summary: Baselines Before Architecture

Problem Statement

Recent autonomous penetration testing systems (e.g., MAPTA, PentestGPT V2) report high benchmark scores on the XBOW benchmark by wrapping frontier Large Language Models (LLMs) in complex, multi-component security harnesses. These harnesses include role decomposition, persistent memory, retrieval, and attack-tree search. However, a methodological attribution problem exists: these systems often change both the underlying backbone model and the architectural harness simultaneously. Consequently, it is difficult to determine how much performance gain is derived from the security-specific architecture versus the inherent capabilities of the newer model or the general coding-agent loop. Without a controlled, model-matched baseline, the field cannot distinguish whether a specialized harness performs meaningful work or merely provides a capable LLM with sufficient access to act.

Methodology

This paper presents a controlled study using the 104-task XBOW benchmark to isolate the contribution of security-specific architecture. The study employs a three-phase experimental design using default, minimally modified coding CLI agents (Codex, OpenCode, and Pi) as "plain-agent" baselines.

Experimental Phases:

Plain-Agent Selection (RQ1): Codex, OpenCode, and Pi were evaluated under the same backbone model (GPT-5), budget ($0.75/challenge), target interface, and scoring rules. Each agent ran two full passes over the 104 challenges to identify the strongest and most stable baseline.
Prompt Effect (RQ2): The strongest baseline (Codex) was tested with two security-specific prompt variants (detailed task prompt and whole-system prompt) against its native default prompt to determine if security-specific text improves performance.
Architecture Residual (RQ3): The selected plain baseline was compared against published results from MAPTA (using GPT-5) and PentestGPT V2 (using GPT-5.2). The "architecture residual" was calculated as the difference between the published specialized system score and the closest model-matched plain-agent score.
Model Scaling (RQ4): The plain Codex scaffold was run with newer models (GPT-5.2 and GPT-5.5) to measure performance gains attributable solely to model improvements without adding security-specific architecture.

Controls and Metrics:

Environment: All agents ran in isolated Docker containers with a black-box flag scoring rule.
Metrics: The study distinguishes between pass@1 (single attempt), pass@k (union of $k$ independent passes), and "reliable" solves (solved in both passes).
Cost Analysis: Detailed breakdowns of token usage, runtime, and cost per solved vs. failed task were recorded.

Key Contributions

Attribution Methodology: A framework for separating backbone model capability, general coding-agent scaffolding, and pentesting-specific architecture.
Same-Model Baseline: Identification of Codex as the strongest plain coding-agent baseline under GPT-5, outperforming OpenCode and Pi in both solve rate and stability.
Architecture Residual Quantification: A direct comparison showing that while specialized harnesses provide a measurable lift, the gap narrows significantly when model differences are controlled.
Prompt and Scaling Analysis: Evidence that security-specific prompts did not improve performance over the native default, while newer models (GPT-5.5) substantially improved the plain scaffold, often exceeding the headline scores of older specialized systems.
Reproducible Artifact: A public GitHub repository containing the evaluation harness, prompt files, agent configurations, and analysis scripts.

Results

1. Plain-Agent Performance (RQ1 & RQ2)

Codex was the superior baseline, solving 70/104 tasks in each of two passes (67.3% pass@1 average) and 81/104 in union (pass@2).
Security Prompts: Adding security-specific text to the prompt degraded performance. The detailed prompt and whole-system prompt variants solved fewer challenges and incurred higher costs than the native default Codex prompt.

2. Architecture Residuals (RQ3)

MAPTA (GPT-5): Reported 76.9% vs. Plain Codex GPT-5 average of 67.3%. The architecture residual is +9.6 percentage points. MAPTA also demonstrated higher cost efficiency ($0.206/task vs. $0.266/task for plain Codex).
PentestGPT V2 (GPT-5.2): Reported 85.0% vs. Plain Codex GPT-5.2 average of 79.8%. The residual is +5.2 percentage points.
Coverage: Repeated plain-agent runs (pass@2 union) matched or exceeded the published single-run scores of both specialized systems (Plain Codex GPT-5 pass@2 = 77.9% > MAPTA 76.9%; Plain Codex GPT-5.2 pass@2 = 88.5% > PentestGPT V2 85.0%).

3. Model Scaling (RQ4)

GPT-5.5: Running the plain Codex scaffold with GPT-5.5 achieved a 92.3% pass@1 and 95.2% pass@2 union.
This plain-agent result exceeded the 91% headline score reported by PentestGPT V2 (using Opus 4.5), demonstrating that model scaling alone can close or exceed the gap attributed to specialized architecture.

4. Resource Efficiency

Specialized systems like MAPTA showed better cost efficiency per challenge than the plain GPT-5 baseline, suggesting that better planning reduces wasted exploration.
However, failures in plain agents were significantly more expensive than successes (e.g., GPT-5 failed scans cost $0.505 vs. $0.150 for successful scans), highlighting the cost of unguided exploration.

Significance and Claims

The paper argues for a "baseline-first" evaluation methodology in autonomous penetration testing. Its primary significance lies in the following claims:

Attribution is Critical: High benchmark scores cannot be attributed to architecture design alone without a model-matched plain-agent baseline. The confounding of model progress and architectural design has obscured the true value of security harnesses.
Harness Value is Nuanced: Specialized architectures do provide measurable value, particularly in cost efficiency and single-attempt solve rates (the residual). However, the magnitude of this value is smaller than headline-to-headline comparisons suggest.
Prompting is Not a Harness: Merely adding security-specific text to a prompt does not constitute an architectural improvement and may degrade performance.
Model Progress Masks Architecture: Newer models (e.g., GPT-5.5) can outperform older specialized systems using the same minimal scaffold. Therefore, claims of architectural superiority must be validated against the current state-of-the-art plain agents, not just older baselines.

The authors conclude that while security harnesses can improve efficiency and solve rates, future research must report model-matched plain-agent baselines to accurately attribute performance gains to architectural design rather than model capability.