🤖 AI

Builder, Defender, Breaker: The Case Against Removing the Human from the AI-Driven Security Lifecycle

This paper argues that removing humans from the AI-driven security lifecycle is fundamentally flawed because it eliminates the necessary independence between building, defending, and testing roles, thereby creating shared blind spots, eroding accountability, and leaving systems vulnerable to predictable adversarial exploitation.

Original authors: Mohamed Chahine Ghanem

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohamed Chahine Ghanem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes construction project where you need to build a fortress, guard it, and then try to break into it to find weak spots. In the past, you hired three different teams for these jobs: the Builders, the Defenders, and the Breakers.

The paper argues that we are currently rushing to replace all three of these human teams with a single, super-smart AI robot. The idea is that if this robot can write code, it can also check its own work and try to hack its own creation. The author, Mohamed Chahine Ghanem, says this sounds efficient, but it's actually a dangerous trap.

Here is the breakdown of why removing humans from this process is a bad idea, using simple analogies:

1. The "One-Brain" Problem (The Monoculture)

Imagine if the person who designed the fortress, the person guarding the gates, and the person trying to pick the locks were all clones of the same person. They all went to the same school, read the same books, and think exactly the same way.

The Risk: If that one "brain" has a blind spot (a mistake they don't know how to make), all three roles will have that same blind spot.
The Result: The Builder makes a mistake. The Defender (who thinks like the Builder) doesn't see it. The Breaker (who also thinks like the Builder) doesn't know how to find it. The flaw slips through the cracks because everyone is looking at the world through the same pair of glasses. The paper calls this an "algorithmic monoculture."

2. The "Grading Your Own Homework" Problem (The Oracle)

In school, if you write an essay and then grade it yourself, you are likely to give yourself a high score because you know what you meant to say, even if you didn't say it clearly. You are biased toward your own work.

The Risk: When an AI writes code and then writes the test cases to check that code, it is essentially grading its own homework. It checks to see if the code matches its own expectations, not if the code actually works in the real world.
The Result: The system becomes a closed loop where it only confirms what it already believes. It loses the ability to find real errors because it lacks an independent "teacher" (a human) to say, "Wait, that doesn't make sense."

3. The "Speed Trap" (Drifting Out of the Loop)

Imagine a race car driver who is so fast that the human safety officer can't keep up. At first, the officer watches the car. Then, the car goes so fast the officer can only watch a summary later. Finally, the car is so fast the officer is just a passenger who gets a report after the crash.

The Risk: AI works at machine speed. If an AI finds a bug and fixes it in seconds, a human can't possibly stop to check it without slowing everything down. So, we start removing the human "stop" button.
The Result: Humans drift from being "in the loop" (making decisions) to "on the loop" (watching) to "out of the loop" (just watching the results). By the time a human realizes something is wrong, the AI has already made thousands of mistakes.

4. The "Predictable Target" (The Adversary)

Security is a game of cat and mouse. If the mouse (the defender) always thinks the same way, the cat (the hacker) learns exactly how to catch it.

The Risk: If the same AI model is used to build, defend, and test, the entire system becomes predictable. A hacker only needs to figure out how to trick one type of AI, and they can trick the whole system.
The Result: It's like building a castle with only one type of lock. Once a thief picks that lock, the whole castle falls. The paper notes that in real-world tests, AI systems that were great at hacking other machines failed miserably when faced with humans who could think creatively and unpredictably.

5. The "No One to Blame" Problem (Accountability)

If a human builder makes a mistake, you can ask them, "Why did you do that?" and hold them responsible. If a machine makes a mistake, who do you ask?

The Risk: If the AI builds, defends, and tests itself, and something goes wrong (like a massive security breach), there is no human to hold accountable. The machine didn't "choose" to fail; it just followed its code.
The Result: This creates an accountability vacuum. Without a human to sign off on the work, the system loses the discipline that comes from knowing you have to answer for your actions.

The Solution: Keep the Human in the Driver's Seat

The paper doesn't say we should stop using AI. In fact, AI is great at doing the heavy lifting, writing code quickly, and finding obvious bugs.

However, the author argues that humans must remain the "Head" of the operation.

The AI should be the hands and the eyes (doing the work).
The Human must be the brain that provides the independent judgment.

We need humans to be the "external oracle" (the teacher who grades the homework), the "circuit breaker" (the one who can hit the emergency stop), and the person who takes responsibility. If we remove the human entirely, we aren't just automating a job; we are breaking the safety system that keeps the digital world secure.

Technical Summary: Builder, Defender, Breaker: The Case Against Removing the Human from the AI-Driven Security Lifecycle

Problem Statement
The paper addresses the emerging trend in cybersecurity where a single family of generative AI models is increasingly deployed to perform all three critical roles of the security lifecycle: building (code generation), defending (intrusion detection and hardening), and breaking (vulnerability discovery and fuzzing). While partial AI assistance is widely accepted, the prevailing narrative suggests that full autonomy—removing the human from the loop entirely—is the natural endpoint. The author argues that this transition from partial assistance to full autonomy represents a qualitative shift, not merely a quantitative one. The core problem is that when the builder, defender, and breaker are drawn from the same generative distribution (an "algorithmic monoculture"), they inherit a shared set of blind spots. This convergence collapses the independence required for meaningful verification, creates a predictable target for adversaries, and dissolves the locus of accountability.

Methodology and Evidence Base
The paper employs a conceptual and evidentiary analysis rather than presenting new experimental data. It synthesizes findings from four distinct domains to construct its argument:

Autonomous Code Generation: Citing studies on GitHub Copilot and formal analyses of models generating C programs, the paper highlights that AI models reproduce the insecurity patterns of their training data, with high vulnerability rates (40–62%) that are consistent across different models trained on similar corpora.
Adversarial Machine Learning: The analysis draws on NIST taxonomies and research showing that detectors trained on specific distributions can be evaded or poisoned. It emphasizes that a defender sharing the same distribution as the code it protects will be blind to the same edge cases.
Software Fault Tolerance: The paper references the failure of N-version programming (where independently written implementations failed on the same inputs more often than expected) to argue that AI models trained on similar data will exhibit correlated failure modes, rendering redundancy ineffective.
Autonomous Hacking Tournaments: Evidence is drawn from the 2016 DARPA Cyber Grand Challenge and the 2025 AI Cyber Challenge. While these systems demonstrated high success in finding and patching vulnerabilities without human intervention, the paper uses these results to illustrate the "closed loop" phenomenon where the tools testing the system are of the same nature as the system being tested.

Key Contributions and Arguments
The paper articulates four primary hazards associated with a fully autonomous, AI-driven security lifecycle:

Correlated Failure and the Monoculture Problem: In a traditional lifecycle, independence between stages (builder, defender, tester) ensures that if one fails, another catches the error. In an AI monoculture, all stages share the same priors and training data. Consequently, a vulnerability the builder fails to generate is also a vulnerability the defender fails to detect and the tester fails to probe. This creates a "blind-spot set" ( $\mathcal{B}$ ) that traverses the entire lifecycle unseen.
The Oracle Problem: Verification requires an independent oracle. When AI generates code and AI generates tests for that code, the test suite inherits the same assumptions as the code. The paper argues this is equivalent to "marking one's own homework," where the system confirms its own expectations rather than verifying correctness against the real world. Furthermore, human oversight is shown to be ineffective; studies indicate humans using AI assistants produce less secure code while reporting higher confidence in it.
Adversarial Exploitation of Uniformity: Security is inherently adversarial. A homogeneous AI pipeline offers a uniform, predictable target. Adversaries can exploit this by crafting attacks (evasion or poisoning) that transfer across the entire monoculture because the models share structural gradients. Additionally, the dual-use nature of autonomous tools means a system capable of finding and patching vulnerabilities is inherently capable of exploiting them, with no human "circuit breaker" to prevent the tool from being redirected against live targets.
The Accountability Vacuum: Full autonomy creates a "responsibility gap." If an autonomous lifecycle ships a vulnerability or causes an outage, there is no principal to hold accountable. The paper notes that governance frameworks like the EU AI Act explicitly require human oversight for high-risk systems, recognizing that accountability cannot be delegated to a machine. Removing the human removes the mechanism by which the lifecycle is held to standards.

Results and Claims
The paper does not claim that AI has no utility; it acknowledges that autonomous systems have successfully identified zero-day vulnerabilities and patched flaws in competitive environments (e.g., the 2025 AI Cyber Challenge). However, the paper's central claim is that full autonomy is a trap for the security lifecycle. The results of autonomous tournaments demonstrate capability, but they also reveal fragility: when autonomous systems faced human adversaries in the DEF CON Capture-the-Flag tournament, they performed poorly, suggesting that machine-speed reasoning lacks the adaptability required for complex, adversarial reasoning.

The paper concludes that the human role in the security lifecycle is not a temporary scaffold to be removed as models improve, but a permanent structural requirement. The human provides the necessary independence, the external oracle, the ability to intervene at machine speed, and the locus of accountability.

Significance
The significance of this work lies in its challenge to the "full autonomy" narrative in cybersecurity. It argues that the division of labor between humans and machines must be preserved not to limit speed, but to maintain the fundamental properties of security: independence, verification, and accountability. The paper proposes a defensible division of labor where:

Build and test paths use independent sources (different models/data) to prevent correlated failures.
Humans retain "on-the-loop" authority with the ability to intervene effectively at machine speeds.
Provenance and independent oversight (audit logs, third-party red-teaming) are maintained.
Workforce skills are preserved to prevent "deskilling," ensuring humans remain capable of exercising their authority.

Ultimately, the paper asserts that while machines can wear the hats of Builder, Defender, and Breaker, the "head" that answers for them must remain human.