Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing
This paper systematizes the operational failures of LLM-driven penetration testing tools through a hands-on evaluation, deriving quantitative design laws and a four-dimensional friction index to guide the construction of robust agentic security systems, exemplified by the Inspectra platform.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of computer security, there is a long-standing tradition of hiring skilled humans to break into systems before the bad guys do. These experts, known as penetration testers, spend their days hunting for hidden flaws in software, networks, and cloud infrastructure. They look for weak passwords, unpatched holes, and logic errors that could allow a criminal to steal data or take control. For decades, this work has been a manual, human-driven craft. However, a new force has entered the field: artificial intelligence. Specifically, large language models—systems trained on vast amounts of text that can understand and generate human language—are now being asked to do the work of these security experts. The idea is to build "agentic" systems: software that can not only read code and scan networks but also plan attacks, execute them, and write reports, all without a human holding its hand. This shift promises to make security testing faster and cheaper, but it also introduces a new set of problems. When you hand a complex, dangerous job to a machine that thinks in probabilities rather than certainties, the machine can make mistakes that a human would never make, or worse, it can confidently lie about what it found.
A team of researchers set out to understand exactly how these AI security agents behave when they are left to work alone. They did not just build a theoretical model; they constructed a real, working platform called Inspectra and put it through its paces against a variety of security tools. Their goal was to move beyond the hype of what these agents could do and document the hard engineering realities of what they actually do when things go wrong. What they found was a discipline in its infancy, where the same failures were happening over and over again, not because the technology was broken, but because it was being used in ways its designers never intended. The researchers discovered that the biggest hurdles were not a lack of intelligence in the AI, but rather a lack of structure in how the system was built. They identified specific, predictable patterns of failure that occur when an AI tries to remember too much, when it tries to judge its own work, and when it is given a budget it cannot control.
One of the most immediate problems the team encountered was the issue of memory. Imagine an AI agent that starts a long investigation, reading thousands of lines of code and running dozens of tests. As the investigation continues, the amount of information it needs to keep in its "mind" grows. Eventually, the system runs out of space to hold all that information. In human terms, this is like trying to remember every detail of a long conversation while the person speaking keeps talking; the earliest details simply fade away. The researchers found that when an AI agent works in one long, continuous session, it inevitably forgets the evidence it collected at the beginning of the scan. By the time it reaches the end to write its report, it might confidently describe files it never actually opened or claim to have found vulnerabilities that were recorded hours earlier but have since been erased from its memory. The solution they proposed was to break the work into short, distinct phases. Instead of one long conversation, the system uses a series of short-lived agents. Each agent does a small piece of the work, writes its findings into a permanent, organized file, and then stops. The next agent reads only a brief summary of that file, not the raw data. This approach allows the system to handle much larger tasks without losing its place, effectively extending the horizon of what the AI can remember by compressing the information it needs to carry forward.
Another critical failure mode involved how the system judged its own success. When an AI red-teaming tool tries to break a system, it needs a way to decide if an attack actually worked. The researchers found that many automated tools were too eager to say "yes, this worked" when they saw certain keywords, even if the attack had failed. This led to a flood of false alarms. To fix this, the team designed a two-step verification process. First, a quick, cheap check would filter out obvious failures. Then, a second, more careful judge would review only the cases that passed the first check. Crucially, this second judge had to be a different kind of system than the first one; if both judges were the same type of AI, they would make the same mistakes and the second check would be useless. By using two different systems to verify the results, the team could dramatically reduce the number of false alarms, making the final report much more trustworthy. They also discovered a subtle but dangerous bias: if an attack failed in a way that the system couldn't understand, the software often recorded it as a success. This meant the most dangerous, elusive attacks were the ones most likely to be hidden or mislabeled, giving a false sense of security.
The researchers also tackled the problem of cost and control. Security tools can be unpredictable; some run for seconds, while others can get stuck and run for hours, consuming massive amounts of money and computing power. The team showed that simply telling an AI "don't spend too much time" in a text instruction is not enough. The AI might ignore the instruction, or it might get distracted by the content it is reading and forget the rule. Instead, the system needs a hard, external gatekeeper—a piece of code that sits outside the AI's decision-making process. This gatekeeper enforces strict limits on how long a tool can run and how much it can spend, cutting it off the moment it hits the limit. This ensures that the AI cannot accidentally bankrupt a project or scan a server it is not authorized to touch. The researchers emphasized that the AI should never be the final authority on its own safety or budget; that role must always belong to a separate, unchangeable layer of code.
Finally, the team looked at the tools themselves. They tested ten different security scanners and found that the ones easiest to run by hand were often the hardest to use in an automated system. Many tools were designed for a human to click through a login screen and then hand over the session to the scanner. When an AI tried to do this, it often failed because the login process was too complex or the tool didn't understand how to stay logged in. The researchers developed a pattern where a separate browser automation tool handled the login, and then passed a clean, authenticated list of links to the scanner. This simple separation of duties solved many of the integration headaches. They also noted that the landscape of AI safety policies is volatile; a model that allows a security test today might block it tomorrow due to a change in the provider's rules. This means that any system built on these tools must be flexible enough to swap out the underlying AI models without breaking the entire workflow.
The paper concludes that the future of automated security does not lie in making the AI smarter, but in building a better structure around it. The most effective systems are those that treat the AI as a powerful but fallible worker, surrounded by a rigid framework of rules, short memory cycles, and independent checks. By accepting that the AI will forget, will make mistakes, and will sometimes ignore instructions, engineers can build systems that are robust enough to handle the real world. The researchers' work provides a blueprint for moving from experimental demonstrations to reliable, deployed products, ensuring that when an AI security agent says it has found a flaw, it has actually found it, and that it has done so without breaking the bank or the rules.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.