💻 computer science

A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems

This paper introduces an autonomous red teaming framework that integrates large language models with reinforcement learning to effectively generate adaptive, multi-stage attack campaigns against AI-enabled Security Orchestration, Automation, and Response systems, demonstrating that this hybrid approach outperforms standalone LLM or domain-specific models in evaluating cyber defense resilience.

Original authors: Ayan Javeed Shaikh, Nathaniel D. Bastian, Ankit Shah

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ayan Javeed Shaikh, Nathaniel D. Bastian, Ankit Shah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes game of chess played on a digital battlefield. On one side, you have the Blue Team: a super-smart, automated security system (an AI) designed to protect a company's computer network. On the other side, you need a Red Team: a hacker AI designed to try and break in, not to steal data, but to test if the Blue Team is actually strong enough.

The problem the authors of this paper identified is that the old ways of testing these security systems are broken. Traditional "hacker scripts" are like robots that follow a strict, pre-written list of moves. They can't think on their feet. If the Blue Team changes its strategy, the robot just keeps trying the same move and fails.

The paper proposes a new, smarter way to build a Red Team AI. They call it a Hierarchical LLM-RL Framework. Here is how it works, using simple analogies:

The Problem: Two Flawed Players

The researchers tried two different types of AI players on their own, and both failed miserably:

The "Talker" (LLM only): Imagine a brilliant military strategist who has read every book on war ever written. They can come up with a perfect, complex battle plan. But, they have never actually held a sword or fired a gun. When they try to execute the plan in a real, chaotic battle, they get confused, forget the steps, and keep repeating the same mistake. In the paper's test, these "Talker" AIs could think of good ideas but couldn't actually pull off a long, multi-step attack.
The "Doer" (RL only): Imagine a soldier who is incredibly good at reacting to what happens right in front of them. They learn by trial and error. But, they have no strategy. They don't know why they are fighting or what the ultimate goal is. In the test, this "Doer" AI got stuck in a loop, just poking the enemy walls over and over again because it didn't know how to plan a full invasion. It gave up on the big picture.

The Solution: The General and the Soldier

The authors realized that to win, you need both the strategist and the soldier working together in a single team. They built a two-part system:

The General (The LLM Planner): This is the "Talker." It looks at the big picture. It decides the strategy: "We need to sneak in, find the back door, climb the ladder, and then take over the server." It doesn't touch the controls; it just gives orders. It stays "frozen" (it doesn't learn new things during the game) so it keeps its vast knowledge of how attacks work.
The Soldier (The RL Controller): This is the "Doer." It listens to the General's orders. Its job is to figure out exactly which button to press at every single second to make that order happen. It learns through trial and error, getting better at executing the General's plan as the game goes on.

The Magic Ingredient: The "Reflection" Notebook
To make the General even smarter, they added a "Reflection" step. After every game, the General writes a short note in a notebook: "I tried to climb the ladder, but the Blue Team caught me. Next time, I'll try a different route." The General reads this note before the next game starts. This helps the General improve its strategy without needing to relearn everything from scratch.

The Results: A Winning Team

They tested this new "General + Soldier" team in a highly realistic simulation of a corporate network (called CAGE 4) against the strongest automated defenders available.

The "Talker" alone managed to break in only about 30% of the time (and usually just the first step).
The "Doer" alone managed to break in 0% of the time. It got stuck doing the same useless thing over and over.
The "General + Soldier" team succeeded in breaking in 94% to 100% of the time. They successfully navigated the entire "kill chain" (the full path from finding the network to taking control of it).

The Big Takeaway

The paper concludes that you cannot rely on just one type of AI to test security.

Thinking alone isn't enough: You need someone who can actually do the work.
Doing alone isn't enough: You need someone who knows what to do.

By combining a smart planner (LLM) with a fast learner (RL), they created a Red Team that is robust, adaptable, and capable of testing the limits of modern AI security systems. This helps companies know if their defenses are truly ready for a real, smart hacker.

Technical Summary: A Red Teaming Framework for Evaluating Robustness of AI-enabled SOAR Systems

Problem Statement

As enterprises increasingly adopt AI-enabled Security Orchestration, Automation, and Response (SOAR) systems for autonomous threat detection and response, the resilience of these systems against adaptive, strategic adversaries remains under-explored. Traditional penetration testing and rule-based red teaming fail to capture the dynamic, multi-step nature of real-world attack campaigns. While recent advances in Large Language Models (LLMs) and Reinforcement Learning (RL) offer potential for autonomous red teaming, both paradigms exhibit fundamental limitations when applied independently:

LLMs: Struggle with long-horizon state tracking, action consistency, and adaptation through environment feedback. They often fail to sustain multi-stage attack campaigns against adaptive defenders.
RL: Suffers from sample inefficiency, limited interpretability, and poor generalization across heterogeneous cyber environments. Pure RL agents often converge to degenerate policies (e.g., repeating reconnaissance actions) when faced with sparse rewards and complex state spaces.

There is a critical gap in evaluating the robustness of AI-enabled cyber defense systems against agents that can combine high-level strategic reasoning with adaptive tactical execution.

Methodology

The authors propose a hierarchical hybrid framework that integrates LLM-based strategic planning with RL-based tactical execution for autonomous red teaming. The system is modeled as a Partially Observable Markov Decision Process (POMDP) and operates within the CybORG CAGE Challenge 4 environment, a high-fidelity enterprise network simulation featuring partial observability and coordinated AI defenders.

Architecture

The framework decouples the agent into two distinct levels:

Strategic Planner (LLM): A frozen LLM operates at a coarse temporal resolution (e.g., every 20 steps). It generates structured "intents" based on current observations and optional memory. These intents encode high-level decisions such as action type, MITRE ATT&CK tactic, target selection, and risk posture. The LLM parameters remain fixed during training to preserve external knowledge.
Tactical Controller (RL): A trainable RL controller (using Proximal Policy Optimization, PPO) executes actions at every timestep. It conditions its policy on both the current environment observation and the LLM-generated intent. This allows the controller to learn environment-grounded policies while adhering to high-level strategic guidance.

Key Mechanisms

Hierarchical Reward Shaping: To address sparse and delayed rewards, the authors introduce a multi-component reward function ( $r_t = r_{env} + r_{progress} + r_{constraint} + r_{alignment}$ ). This includes penalties for invalid actions and bonuses for progressing through kill-chain stages (e.g., discovery, exploitation, privilege escalation, impact), aligned with MITRE ATT&CK.
Reflexion (Memory): An optional mechanism allows the LLM planner to generate natural language self-reflections at the end of each episode. These reflections (summarizing successes, failures, and defender behaviors) are stored in a FIFO buffer and fed back into the planner's input for subsequent episodes, enabling strategic adaptation without parameter updates.
State Representation: Observations are transformed into a text summary for the LLM and a numeric feature vector for the RL controller. The LLM's structured intent is encoded into an embedding space compatible with the RL policy network.

Key Contributions

Hierarchical LLM-RL Architecture: The paper presents a novel architecture that decouples strategic planning (LLM) from tactical execution (RL), enabling structured interaction between high-level intent generation and low-level environment-grounded control.
Kill-Chain-Aligned RL Framework: The authors introduce a reward shaping mechanism specifically designed to guide long-horizon adversarial behavior consistent with MITRE ATT&CK stages, stabilizing learning in sparse-reward environments.
Comprehensive Empirical Study: The work evaluates a wide range of LLMs (0.6B to 70B parameters), including general-purpose models (Qwen3, Llama), reasoning specialists (DeepSeek-R1, QwQ), and domain-specialized cybersecurity models (Foundation-Sec, DeepHat).
Rigorous Benchmarking: The framework is evaluated against the strongest published autonomous defender for the CAGE 4 setting (H-MARL Expert policy), providing a robust test of red team capabilities under realistic adversarial conditions.

Results

Experiments were conducted over 200 episodes in the CAGE 4 environment. The primary metric was the Episode Compromise Rate (ECR).

Hybrid Framework Performance: The proposed LLM+RL agents achieved near-perfect success.
- Qwen3 + RL: Achieved a 100% ECR (200/200 episodes).
- Foundation-Sec + RL: Achieved a 94% ECR.
- These agents reliably progressed through multiple attack stages, including privilege escalation and impact.
Standalone LLM Limitations: LLM-only agents (zero-shot) exhibited limited end-to-end capability.
- Only 6 of 14 models achieved any compromise beyond the initial foothold.
- The strongest standalone model (QwQ-32B) succeeded in only 30% of episodes.
- Domain-specialized models (e.g., Foundation-Sec-8B) generated high action volumes but failed to translate this into meaningful attack progression (6.5% ECR), indicating that domain knowledge alone is insufficient without consistent execution.
Standalone RL Limitations: The RL-only agent collapsed to a degenerate policy, selecting a single reconnaissance action in nearly all timesteps and achieving a 0% ECR. It failed to discover the structured sequence of actions required for multi-stage attacks under the given reward structure.
Scaling Observations: Performance in the Qwen3 family was non-monotonic; smaller models (0.6B, 1.7B) achieved some success, while larger variants (4B+) failed to attempt exploitation despite effective exploration.

Significance and Claims

The paper claims that autonomous red teaming against AI-enabled SOAR systems requires the tight coupling of strategic reasoning and learned execution. The authors argue that:

Standalone approaches are insufficient: Neither reasoning-only (LLM) nor learning-only (RL) systems can execute full kill-chain behaviors against adaptive defenders.
Hybrid integration yields multiplicative improvement: Combining LLM planning with RL execution results in substantial improvements in multi-stage attack success and robustness.
Structured planning is more critical than domain specialization: General-purpose models consistently outperformed larger, cybersecurity-specialized models in the hybrid setting. This suggests that output reliability and structured controllability are key requirements for LLMs in hierarchical decision systems, outweighing the benefits of specific domain fine-tuning.

The study concludes that this hybrid framework bridges the gap between the strategic reasoning capabilities of LLMs and the adaptive tactical execution of RL, providing a principled approach to stress-testing the robustness of autonomous cyber defense systems.