🤖 AI

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

The paper introduces Trident, an agentic LLM red teaming framework that leverages a "Code-as-Policy" RLVR architecture to generate adaptive attack strategies, revealing that existing Deep Reinforcement Learning cyber defenses are fundamentally brittle and easily outperformed by dynamic threats compared to static heuristic baselines.

Original authors: Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi, Sanggeon Yun, Hyunwoo Oh, SungHeon Jeong, Nathaniel D. Bastian, Mahdi Imani, Mohsen Imani

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi, Sanggeon Yun, Hyunwoo Oh, SungHeon Jeong, Nathaniel D. Bastian, Mahdi Imani, Mohsen Imani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes game of digital chess, but instead of moving pieces on a board, two teams are fighting over a sprawling, invisible network of computers. One team, the "Blue Team," builds an AI defender that learns to spot intruders and fix broken computers in real-time. The other team, the "Red Team," tries to break in. For years, scientists have tested these Blue Team defenders against Red Team attackers that are like robots following a strict, unchanging script. They run the same moves over and over, like a clockwork toy. The problem is, real hackers aren't clockwork toys; they are clever, adaptive, and change their tactics the moment they see a defense. Meanwhile, a new kind of super-smart AI, known as a Large Language Model (LLM), has been getting incredibly good at solving puzzles and writing code. The big question scientists are asking is: If we teach these super-smart AIs to be the Red Team, can they finally outsmart the Blue Team's defenses in a way that old, static robots never could?

This is exactly what the researchers behind a project called Trident set out to find. They built a new, dynamic testing ground where a smart AI attacker could learn by doing, rather than just following a pre-written script. They created a massive library of over 13,000 simulated battles between attackers and defenders to train their AI. Instead of making the AI choose one move at a time, they taught it to write its own "attack strategy" as a piece of computer code, which it then runs against the defender.

The results were startling. The Trident AI, trained on just a single 7-billion-parameter model (a relatively small size for modern AI), didn't just beat the defenders; it caused a massive drop in their effectiveness. While the old, static attackers barely made a dent, Trident reduced the Blue Team's defensive performance by an average of 522% compared to the baseline. This suggests that many of today's "smart" cyber defenses are actually much more fragile than we thought, because they were only ever tested against dumb, predictable enemies. Trident didn't just follow rules; it discovered clever new tricks on its own, like realizing when a defender was trying to trick it with a fake target (a "decoy") and learning to ignore it, or figuring out which computers were the most valuable to hack first. The paper suggests that to build truly safe systems, we need to stop testing them against static robots and start testing them against adaptive, learning AIs like Trident.

Technical Summary: Trident – How to Break Deep Reinforcement Learning

Problem Statement

Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) are increasingly deployed to monitor networks and recover compromised hosts in real-time. However, their robustness against adaptive threats remains critically understudied. Current evaluations rely almost exclusively on static, heuristic "red agents" with fixed behavioral signatures. Consequently, DRL defenders develop policies implicitly conditioned on these static patterns, creating systematic blind spots that traditional evaluation methods fail to uncover.

Simultaneously, while Reinforcement Learning with Verifiable Rewards (RLVR) has advanced Large Language Model (LLM) reasoning in domains like mathematics and software engineering, its integration into cybersecurity has been hindered by a lack of suitable benchmark environments. Existing cybersecurity datasets typically offer static corpora or isolated Capture The Flag (CTF) challenges, lacking the continuous, multi-step interactions required to train agents against active, adaptive defenders. There is a critical gap in infrastructure capable of providing stable, repeatable adversarial environments with verifiable reward signals for training LLM-based red agents via RLVR.

Methodology

To address these gaps, the authors introduce Trident, a dynamic red teaming framework comprising a benchmark infrastructure, a dataset, and an agentic architecture.

1. Trident Benchmark and Dataset

Trident operates across two high-fidelity simulators: CybORG CAGE 4 (modeling a segmented military network with cooperative multi-agent defenders) and CyberWheel (grounding attacks in real-world MITRE ATT&CK techniques).

Infrastructure: The framework utilizes isolated, containerized sandbox servers with a client-server architecture. Generated attack policies are transmitted to these sandboxes, executed natively against live DRL defenders, and return deterministic, verifiable execution logs.
Dataset: The authors constructed a dataset of over 13,000 high-fidelity red-blue interaction trajectories. These trajectories are serialized into structured natural language logs, capturing partial observations, differential host intelligence, and action outcomes, serving as the foundation for RLVR post-training.

2. Trident Agentic Architecture

Trident Agentic reformulates the red agent training problem from a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) into a Contextual Bandit problem using a "Code-as-Policies" paradigm. The architecture consists of three specialized modules:

Log Summarizer: A hybrid module using rule-based parsing and a frozen LLM to compress raw execution logs. It preserves critical "milestone" events (e.g., successful exploits, privilege escalations) while sampling failed attempts, synthesizing a dense semantic summary for the planner.
Planner ( $\pi_\theta$ ): The sole trainable component (a 7B parameter LLM). It receives the compressed summary as a context and generates a high-level attack strategy in a single forward pass, outputting a structured JSON payload. It is optimized via Group Relative Policy Optimization (GRPO) using verifiable rewards derived from the sandbox execution.
Coder: A frozen LLM that translates the Planner's JSON strategy into executable Python code. It employs an Abstract Syntax Tree (AST)-based patching mechanism and an iterative self-repair loop (up to $K=3$ attempts) to correct syntax errors before deployment, ensuring the Planner is not penalized for incidental implementation noise.

3. Reward Design

The system utilizes a dense, verifiable reward function derived directly from execution logs:

Positive Milestones ( $R^+$ ): Weighted rewards for killchain progress (e.g., system impact, privilege escalation) augmented by an efficiency coefficient based on the achievement-to-step ratio.
Negative Events ( $R^-$ ): Penalties for failed exploits, invalid actions, and operational noise.
Execution Penalties: Strict penalties are applied for syntax errors, runtime failures, or attempts to access unobservable simulator internals (e.g., decoy flags), ensuring the agent relies on behavioral inference rather than privileged state access.

Key Contributions

Trident Evaluation Benchmark and Dataset: The first cross-environment red teaming benchmark spanning CAGE 4 and CyberWheel that provides isolated sandbox execution for verifiable rewards against live DRL defenders. It includes a dataset of 13,000+ trajectories specifically designed to support RLVR training.
Trident Agentic: A tripartite agentic LLM architecture that leverages "Code-as-Policies" to enable the direct generation of executable attack strategies. By treating the strategy generation as a contextual bandit, it bypasses the computational intractability of step-by-step POMDP planning.
Empirical Evidence of DRL Brittleness: The framework demonstrates that current autonomous defenders are significantly more brittle against adaptive attackers than previously indicated by static evaluations.

Experimental Results

The authors evaluated Trident Agentic against state-of-the-art DRL blue agents (including GNN, HMARL, and MARL variants) in both CAGE 4 and CyberWheel.

Performance Reduction: Trident Agentic reduced the defensive performance of blue agents by an average of 522% compared to default heuristic baselines.
Comparison to Baselines:
- Static Heuristics: Traditional red agents failed to uncover weaknesses in the learned policies.
- Zero-Shot/CoT: While zero-shot prompting and Chain-of-Thought (CoT) approaches showed some improvement over static baselines, they were substantially outperformed by the RLVR-trained Trident Agentic.
- Frontier Models: A 7B parameter model trained via RLVR (Trident Agentic) surpassed the performance of frontier-scale models (e.g., GPT-4o) used in a zero-shot capacity, highlighting the efficacy of task-specific RLVR training over raw model scale alone.
Emergent Behaviors: The trained agents autonomously discovered novel attack vectors that static heuristics could not represent, including:
- Decoy Avoidance: In CyberWheel, the agent learned to abort attacks on hosts exhibiting repeated failures, effectively identifying and avoiding honeypots.
- Adaptive State Prioritization: In CAGE 4, the agent learned to prioritize high-value targets (e.g., Root/User access) over basic reconnaissance, dynamically shifting focus based on the network state.

Significance and Claims

The paper claims that Trident exposes a fundamental evaluation gap in autonomous cyber defense: current practices critically underestimate the vulnerability of DRL defenders to adaptive, generative attackers. By demonstrating that a compact, open-weight 7B LLM can systematically dismantle state-of-the-art defenses, the authors argue that the field must move beyond static adversary evaluation.

Trident is presented not merely as an offensive tool but as a necessary infrastructure for the "co-evolution" of cyber offense and defense. Its simulator-agnostic, RESTful sandbox design allows for the integration of higher-fidelity environments, enabling the training of blue agents that are robust against generative, adaptive threats. The authors emphasize that transparency regarding these capabilities is essential to drive the development of equally adaptive, RL-driven defensive systems before such attack strategies are independently deployed by adversaries.