🤖 AI

To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack

This position paper argues that to counter inevitable AI-driven cyber attacks, defenders must fundamentally shift their strategy by responsibly developing and deploying their own offensive AI capabilities within controlled, governed environments to master threats before adversaries do.

Original authors: Terry Yue Zhuo, Yangruibo Ding, Wenbo Guo, Ruijie Meng

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Terry Yue Zhuo, Yangruibo Ding, Wenbo Guo, Ruijie Meng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The Old Rules Don't Work Anymore

For the last decade, cybersecurity has worked like a game of "Whac-A-Mole."

The Old Way: Hackers had to be highly skilled humans. Building a complex attack was like building a custom house; it took months of hard work and deep expertise. Because it was so expensive and slow, hackers only attacked the "big houses" (major companies) or used simple, generic tools to knock on thousands of doors hoping one was open. Defenders assumed hackers couldn't afford to knock on every door.
The New Threat: AI Agents are changing the game. Imagine a hacker who isn't a person, but a swarm of thousands of tiny, tireless robots. These robots don't need to be experts. They can knock on millions of doors in a second. Even if they fail 99% of the time, they only need to succeed 1% of the time to make a profit. They can attack the "small houses" (niche systems) that humans used to ignore because they weren't worth the effort.

The paper argues that defenders are currently trying to stop these robots by putting up "Do Not Enter" signs on the robots themselves. But the paper says this won't work because bad actors can just build their own robots without the signs.

Why Current Defenses Are Failing

The authors say we are trying to stop AI hackers by using three main methods, but they are all like trying to stop a flood with a sieve:

Filtering the Training Data (The "Clean Library" approach):
- The Idea: We remove all bad instructions (like "how to build a bomb") from the AI's training books so it never learns to be bad.
- Why it Fails: Bad hackers don't need to read the "bad books." They can teach the AI to figure out how to hack using basic logic and common sense, just like a human can figure out how to pick a lock without a manual. Also, bad hackers can just download the AI and teach it themselves.
Safety Alignment (The "Good Behavior" approach):
- The Idea: We train the AI to say "No" when asked to do something bad.
- Why it Fails: Bad hackers are clever. They can trick the AI into thinking a bad request is actually a good one (like asking a guard to "test the security" instead of "break in"). Also, if the AI is running on a hacker's own computer, they can just retrain it to ignore the "No" button.
Output Guardrails (The "Bouncer" approach):
- The Idea: We put a bouncer at the door to check every message the AI sends and block anything that looks dangerous.
- Why it Fails: A hacker can break a big attack into a thousand tiny, harmless-looking steps. The bouncer sees each step as safe, but when you put them all together, they build a bomb.

The Proposed Solution: Build Your Own "Red Team" Robots

The paper suggests a radical shift: We must teach our own AI agents how to hack, so we can use them to defend us.

Think of it like a fire department.

Current Defense: We wait for a fire to start, then try to put it out.
The Paper's Idea: We hire a team of professional arsonists (our "Offensive AI") to work inside a secure, walled-off training facility (a "Cyber Range").
- These "Good Arsonists" try to burn down the building using every trick in the book.
- Because they are AI, they can try millions of ways to start a fire in seconds.
- We watch where they succeed, find the weak spots in our building, and fix them before a real bad guy shows up.

The Three Steps to Make This Safe

The authors know this sounds dangerous, so they propose three strict rules to keep the "Good Arsonists" from causing real damage:

Build Better Test Tracks: We need better "driving courses" (benchmarks) to measure exactly how good these hacking robots are. We need to test them on real-world scenarios, not just simple puzzles.
Train, Don't Just Script: Instead of giving the robots a fixed list of steps to follow, we need to train them to learn and adapt, just like a human hacker would. This makes them better at finding new, unknown weaknesses.
The "Sandbox" Rule: This is the most important part. The "Good Arsonists" must never leave the secure training facility.
- They stay inside a digital cage where they can't touch the real internet.
- They find the holes and write a report.
- Then, we take that report and give it to a different, "Safe AI" that only knows how to fix holes, not make them. This "Safe AI" is what we release to the public to protect our systems.

The Bottom Line

The paper concludes that we cannot wait for bad hackers to figure out how to use AI to attack us. By the time they do, it will be too late.

Instead, we must master the weapon first. We need to build our own offensive AI, keep it locked in a safe cage, and use it to find every possible way our systems can be broken. Only by understanding how the enemy thinks and acts at a massive scale can we hope to defend ourselves.

In short: To stop the bad robots, we have to build our own good robots that are even better at breaking things, but we must keep them in a cage so they only help us fix our house.

Technical Summary: To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack

1. Problem Statement

The paper identifies a fundamental shift in the economic and technical balance of cybersecurity driven by the advent of AI agents. For over a decade, the defense industry has relied on human labor scarcity to limit attackers; sophisticated exploits require deep expertise and manual effort, forcing adversaries to either target high-value assets or use generic, automated tools against thousands of targets.

However, AI agents disrupt this equilibrium by automating vulnerability discovery and exploitation across thousands of targets with minimal human guidance. Unlike traditional automated tools that rely on pre-programmed rules for known vulnerabilities, AI agents can mimic human strategic agency, adapt to novel systems, and discover new attack paths. This automation lowers the marginal cost of attacks, making "long-tail" systems (previously ignored due to high manual effort) economically viable targets.

The core problem is that current defensive strategies are reactive and model-centric, focusing on preventing misuse through data filtering, safety alignment, and output guardrails. The authors argue these measures are insufficient because:

Adversarial Adaptation: Attackers can bypass safety mechanisms by controlling open-weight models, self-hosting, or fine-tuning models to remove alignment.
Economic Viability: AI agents do not need to be perfect; they only need a small success rate to be profitable due to the ability to parallelize attacks at near-zero marginal cost.
Defensive Gap: Defenders lack the "offensive security intelligence" to predict how attacks will occur at scale, leaving them unable to anticipate the specific vulnerabilities AI agents will exploit.

2. Methodology and Proposed Framework

The paper does not present a single experimental result but rather proposes a strategic framework for shifting defensive posture. The methodology involves three primary pillars:

A. Threat Modeling

The authors formalize a threat model of a financially motivated adversary with access to state-of-the-art (SOTA) AI agents. The adversary's goal is not necessarily to maximize damage to a single target but to maximize aggregate profit across a heterogeneous population of victims. In this model, failures and hallucinations do not constrain effectiveness because attacks can be attempted at a massive scale.

B. Critique of Current Defenses

The paper systematically evaluates and dismisses the efficacy of current defensive safeguards against adaptive AI agents:

Data Governance: Filtering training data fails because AI agents can reason from first principles and acquire new information at inference time, rather than relying on memorized exploit text.
Safety Alignment: Alignment techniques (RLHF, fine-tuning) are brittle and can be bypassed via jailbreaking, objective distortion, or by composing benign steps into harmful long-horizon trajectories.
Representation Engineering: Internal model edits are difficult to verify and often fail when the context changes or when behavior emerges from extended agent interactions.
Output Guardrails: These fail against agentic workflows where harmful outcomes arise from sequences of seemingly harmless actions, particularly in self-hosted or open-weight environments where monitoring is absent.
Access Controls: These are ineffective once models are widely available as open weights, as they can be leaked, replicated, or fine-tuned to remove restrictions.

C. Proposed Offensive Security Strategy

The authors propose that defenders must develop offensive security intelligence to master the capabilities of AI attackers before adversaries do. This involves three specific actions:

Construct Comprehensive Benchmarks:
- Move beyond static datasets to dynamic evaluation environments covering the full attack lifecycle (reconnaissance, exploitation, post-exploitation).
- Include large-scale, real-world projects and simulate "in-the-wild" conditions (e.g., Docker environments, tool sets like Kali/CodeQL).
- Ensure benchmarks cover fine-grained categories (e.g., exploit chaining, command and control) and provide robust ground truth labels.
Advance from Workflow-Based to Trained Agents:
- Current offensive agents rely on external workflows (prompting + tool calls) which are brittle.
- The paper advocates for training agents directly in cyber ranges (controlled environments) using reinforcement learning. This allows agents to learn adaptive strategies, accumulate experience, and discover zero-day vulnerabilities autonomously, rather than following fixed pipelines.
Implement Governance and Distillation:
- Containment: Offensive agents must be restricted to audited, isolated cyber ranges with strict network isolation.
- Distillation: The findings and attack trajectories of offensive agents should be distilled into defensive-only agents. These defensive agents focus on detection, root cause analysis, and remediation without inheriting the dangerous action space of the attacker.
- Staged Release: Offensive capabilities should be treated as high-risk artifacts, released in stages based on measured competence and version control.

3. Key Contributions

Paradigm Shift: The paper argues that offensive AI capabilities must be treated as essential defensive infrastructure. The choice is not if offensive AI will exist, but whether defenders will master it under responsible governance or be forced to reverse-engineer it after successful attacks.
Limitations of Safety Alignment: It provides a technical argument that safety alignment and data filtering are insufficient against determined adversaries with access to open-weight models, necessitating a shift toward proactive, capability-based defense.
Roadmap for Offensive AI: It outlines a concrete evolution path for offensive security agents (Table 2), moving from Knowledge Models (mature) $\to$ Workflow Agents (current status) $\to$ Trained Agents (future goal for zero-day discovery).
Benchmarking Standards: It highlights the deficiencies in current cybersecurity benchmarks (e.g., lack of dynamic environments, limited lifecycle coverage) and proposes a methodology for building agent-friendly, dynamic, and comprehensive evaluation suites.

4. Results and Evidence

The paper relies on a synthesis of existing literature and preliminary observations rather than new experimental data from the authors.

Benchmark Performance: Table 1 summarizes SOTA agent performance on existing benchmarks (e.g., CyberSecEval-3, CTF, Vul. detection). It notes that while agents perform well on small-scale generative tasks (like patching short functions), they struggle with large-scale analytic tasks (like PoC generation on whole projects) and exploit chaining.
Systemic Risk Analysis: The authors argue that AI agents enable "superhuman" cyber attacks by combining breadth (scanning thousands of targets) and depth (adaptive post-exploitation). They cite that agents can operate faster than humans and adapt to security measures in real-time, creating conditions for widespread breaches.
Economic Logic: The paper posits that the cost structure of attacks has fundamentally changed; low per-attempt success rates are now profitable due to the ability to parallelize attacks, rendering traditional "high-value target" defense assumptions obsolete.

5. Significance and Claims

The paper claims that the current defensive AI safety paradigm is restrictive and disconnected from economic reality. By focusing solely on preventing misuse (safety alignment), the community ignores the necessity of understanding and mastering offensive capabilities to build resilient systems.

The authors assert that:

AI-driven attacks are inevitable. The economic incentives for automation in cyberattacks are too strong to ignore.
Defensive intelligence requires offensive capability. To predict how attacks will occur at scale, defenders must possess the same capabilities as attackers, but deployed within controlled, audited environments.
Responsible development is possible. Through strict governance (cyber ranges, distillation into defensive agents, and staged releases), the risks of offensive AI can be contained while reaping the benefits of accelerated vulnerability discovery and remediation.

The paper concludes that failing to invest in offensive security intelligence leaves defenders unprepared for adversaries who will inevitably explore these capabilities independently. The goal is to close the loop between discovery and repair using machine-scale intelligence before adversaries can exploit the "long tail" of the software ecosystem.