Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
This paper introduces a cost-aware evaluation framework for security agents that reveals distinct scaling behaviors between offensive and defensive tasks, demonstrating that while offensive performance improves with increased compute, defensive success relies more on disciplined tool usage and operational efficiency than raw reasoning budget.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where artificial intelligence isn't just a chatbot that writes poems, but a digital detective and a digital burglar rolled into one. This is the frontier of cybersecurity, a field where computers are trained to either break into systems to find weaknesses (the "offensive" side) or to investigate digital crime scenes to stop attacks (the "defensive" side). For a long time, scientists testing these AI agents have asked a simple question: "How good is this AI at solving the hardest puzzles if we give it unlimited time and money?" It's like testing a race car by letting it drive until the tank is empty, ignoring the cost of gas. But in the real world, every step an AI takes—every question it asks, every database it checks—costs actual money. If an AI is too expensive to run, it's useless for a real security team, no matter how smart it is. This paper asks a more practical question: "How much can this AI actually do for a specific budget?"
The researchers, Paul Kassianik, Blaine Nelson, and Yaron Singer, decided to stop just counting how many puzzles an AI solved and start counting the price tag. They put their AI agents through two very different types of challenges. First, they sent them on "Capture the Flag" (CTF) missions, which are like digital scavenger hunts where the AI has to hack into a sandboxed computer to find a secret code. Second, they sent them on "Security Operations Center" (SOC) missions, which are like detective work where the AI has to sift through massive piles of security logs to figure out who broke into a company and how.
Here is the twist they discovered: Offense and defense scale in completely different ways. When the AI was playing the hacker (the offensive role), giving it more money to think and try again usually helped. It was like giving a detective more time to re-read a clue; the more resources they had, the more likely they were to crack the case. Some open-source models, when given a decent budget, could even catch up to the most expensive, top-tier proprietary models.
However, when the AI played the detective (the defensive role), throwing more money at the problem didn't necessarily make it smarter. In fact, the best performers were the ones who were disciplined. They knew exactly when to stop, when to ask for help, and when to save their budget. The paper found that for these investigations, success wasn't about how many tools the AI used or how long it thought; it was about using the right tools at the right time. An AI that frantically called every database in the world just because it had a big budget often did worse than one that carefully picked a few key pieces of evidence.
The authors also warned that some of the "perfect" scores on these tests might be misleading. Because the puzzles are public and old, some AIs might have just memorized the answers during their training, like a student who memorized the answer key instead of learning the math. When the researchers tested the AIs without letting them use any tools or look at the clues, the scores dropped significantly for some models, proving that "knowing the answer" isn't the same as "solving the problem."
Ultimately, the paper argues that we need to stop judging security AIs just by their peak performance in a vacuum. Instead, we need to look at their "economic efficiency." A model that solves 90% of problems for $5 is a better tool for a real-world security team than a model that solves 95% of problems for $500. The study suggests that for hacking tasks, more compute power helps, but for defending systems, smart, disciplined tool use is the real superpower. They even built a website to show these cost-versus-success curves, proving that in the world of AI security, being cheap and smart is often better than being rich and reckless.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.