⚡ electrical engineering

Learning Red Agent Policy from Observations for Neurosymbolic Autonomous Cyber Agents

This paper proposes an imitation learning technique to enable neurosymbolic autonomous cyber-defense agents to predict unobservable red agent policies and actions from network observations, thereby enhancing their ability to adapt to sophisticated cyber-attacks in partially observable environments.

Original authors: Ankita Samaddar, Sandeep Neema, Daniel Balasubramanian, Xenofon Koutsoukos

Published 2026-06-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ankita Samaddar, Sandeep Neema, Daniel Balasubramanian, Xenofon Koutsoukos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes game of chess played in a foggy room. You are the defender (the "Blue Agent"), trying to protect your castle. Your opponent is the attacker (the "Red Agent"), who is trying to break in.

The problem? You can't see your opponent. You only see the results of their moves: a door is suddenly open, a light flickers, or a piece of furniture is moved. You know your own moves, but you have to guess what your invisible opponent just did based on how the room changed.

This paper describes a new way for the defender to "read the mind" of the attacker in a digital network, even when the attacker is hidden.

The Setting: A Digital Castle

The researchers used a simulated network called CybORG. Think of this network as a castle with three main areas:

The Courtyard (User Subnet): Where regular people work.
The Office (Enterprise Subnet): Where important servers and data live.
The Vault (Operational Subnet): The most critical, sensitive machinery.

The attacker tries to sneak from the Courtyard, through the Office, and finally into the Vault to cause damage. The defender tries to stop them.

The Problem: The "Black Box" Fog

In this game, the defender is "partially observable." This is a fancy way of saying: "I can't see the attacker."

The defender sees the state of the network (e.g., "Server A is now compromised").
The defender knows what they did (e.g., "I scanned the network").
But the defender does not know what the attacker did to cause that change. Did they scan? Did they hack a password? Did they install a virus?

Without knowing the attacker's specific move, it's hard to predict their next step or understand how deep they have penetrated the system.

The Solution: A "Sherlock Holmes" AI

The authors propose a technique called Policy Learning using a method similar to Imitation Learning.

Here is the analogy: Imagine you are watching a magician perform a trick. You see the magician's hands move (the defender's action) and you see the rabbit appear (the network state change). You don't see the secret move the magician made to pull the rabbit out of the hat.

The researchers built an AI that acts like a detective:

Watch and Learn: They let the AI watch thousands of games where the attacker and defender play. Even though the AI doesn't see the attacker's moves during the game, it has a "cheat sheet" (the ground truth) during training to see what actually happened.
Find the Pattern: The AI learns a pattern: "When the defender scans the network and the server suddenly looks different, the attacker must have done X."
Predict the Future: Once trained, the AI can look at a new game in real-time. It sees the defender's move and the resulting change, and it says, "I bet the attacker just did X."

How It Works: The Three-Step Detective Process

The paper breaks this down into three stages, like a detective solving a case:

The "What Happened?" Phase (Inverse Dynamics): The AI looks at the "before" and "after" of the network and the defender's move. It tries to guess the "invisible" move the attacker made that caused the change. It creates a rough, abstract guess (a "latent action").
The "Practice" Phase (Policy Learning): The AI trains itself to be better at guessing. It tries to mimic the invisible moves it figured out in step 1. It's like a student practicing a dance by watching the teacher's footprints.
The "Translation" Phase (Mapping): The AI's guesses are still vague. This step translates the vague guess into a specific, real-world attack name (e.g., "Exploit Remote Service") and a specific target (e.g., "Server 3").

The "Neurosymbolic" Brain

The paper mentions Neurosymbolic AI and Behavior Trees.

Neurosymbolic means combining two types of thinking:
- Neural: Learning from data (like a human learning from experience).
- Symbolic: Following logical rules (like a computer following a strict checklist).
Behavior Trees are like a flowchart or a decision tree. Imagine a security guard with a clipboard. The guard checks: "Is the door open? Yes. Is there an intruder? Yes. Then, sound the alarm."
The researchers plugged their new "mind-reading" AI into this flowchart. Now, the guard doesn't just react to the door being open; the guard can predict that the intruder is likely going to the Vault next, and prepare for it.

The Results: How Good Was It?

The researchers tested this in the simulated "CybORG" environment against two types of attackers:

The "B-Line" Attacker: A direct, straight-line path to the goal. (Like a burglar running straight to the vault).
The "Meander" Attacker: A wandering attacker who checks every room and tries to get into every computer before moving on. (Like a burglar snooping around the whole house).

The Findings:

Direct Attackers: The AI was incredibly accurate (over 99%) at predicting the moves of the direct attacker. Because their path is predictable, the "detective" AI could easily guess the next step.
Wandering Attackers: The AI was still very good (around 95% for the general area, though slightly less precise on the exact computer) at predicting the wandering attacker.
Switching Attackers: They even tested an attacker who switches strategies halfway through. The AI could adapt and continue predicting, though it took a moment to catch up when the strategy changed.

Why This Matters

The main takeaway is that this system allows a defender to see the invisible. By predicting what the attacker is doing right now, the defender can:

Detect an intrusion earlier.
Understand how deep the attacker has gone (e.g., "They have user access, but not admin access yet").
React more effectively because they aren't just guessing; they are making an educated prediction based on the attacker's hidden "policy."

In short, the paper presents a way to turn a foggy, blind defense game into a clearer match where the defender can anticipate the attacker's next move, even without seeing them.

Technical Summary: Learning Red Agent Policy from Observations for Neurosymbolic Autonomous Cyber Agents

Problem Statement

Modern autonomous cyber-defense systems increasingly rely on Reinforcement Learning (RL) agents to enforce security rules and adapt to evolving threats. However, these environments are inherently partially observable. Specifically, the defender (blue agent) cannot directly observe the actions or policies of the attacker (red agent). The defender only perceives partial state transitions and indirect outcomes of attacks. This lack of visibility hinders the ability to reconstruct true system dynamics, predict attacker movements, or assess intrusion levels effectively.

While imitation learning has been applied to infer latent policies in classical control and video domains, existing methods are not directly applicable to autonomous networks. This is because network states are typically discrete (snapshots of host/server activities) and state transitions depend on the joint actions of both the red and blue agents. Consequently, there is a critical need for a technique that can learn red agent policies and predict red actions at runtime using only blue observations and blue actions.

Methodology

The authors propose a Policy Learning Technique based on imitation learning, designed to infer red agent policies ( $\pi'$ ) and predict red actions ( $r_{t-1}$ ) from consecutive blue observations ( $o^b_{t-1}, o^b_t$ ) and the previous blue action ( $b_{t-1}$ ). The approach is structured into three phases and integrated with a neurosymbolic framework.

1. Problem Formulation

The network is modeled as a discrete-time Partially Observable Markov Decision Process (POMDP). The objective is to learn the red policy $\pi'$ given a known blue policy $\pi$ . The input consists of:

Blue observations at $t-1$ and $t$ .
The blue action executed at $t-1$ .
The goal is to predict the red action executed at $t-1$ .

2. Three-Phase Learning Technique

The technique employs a multi-stage dynamics model using Deep Neural Networks (DNNs):

Phase I: Data Generation
- The system interacts with the environment to collect transitions $(o^b_{t-1}, b_{t-1} \to o^b_t)$ .
- A dataset $D$ is generated for training.
- A labeled dataset $D_R$ is generated by extracting the ground-truth red actions ( $r_{t-1}$ ) from the system (unobservable during actual runtime) and preprocessing them into components: action name, target host, and subnet.
Phase II: Policy Learning (Latent Action Inference)
- Encoding: Blue actions and targets are one-hot encoded.
- Latent Action Learning: Two models are trained jointly:
  1. Inverse Dynamics Model (IDM1): Estimates a latent red action ( $l_{t-1}$ ) given observations and blue actions.
  2. Forward Dynamics Model (FDM1): Predicts the current observation ( $\tilde{o}^b_t$ ) given the previous state, blue action, and the inferred latent action.
  - These are trained to minimize the observation error $|\tilde{o}^b_t - o^b_t|$ , forcing the latent action to capture the missing red dynamics.
- Policy Imitation: A second Inverse Dynamics Model (IDM2) is trained to imitate the latent actions learned by IDM1, effectively learning the policy $\pi'$ .
- Mapping to Discrete Actions: A Forward Dynamics Model (FDM2) maps the continuous latent actions back to discrete red action components (action name, host, subnet). These outputs are thresholded to produce binary one-hot encodings for the final predicted red action $\hat{r}_{t-1}$ .

3. Integration with Neurosymbolic Agents

The learned policy is integrated into an Evolving Behavior Tree (EBT), a neurosymbolic architecture combining Behavior Trees (BTs) with Learning-Enabled Components (LECs).

New behaviors are added to the EBT to perform runtime prediction.
A condition node ensures sufficient observation history exists before prediction.
The EBT selects the appropriate learned red policy based on the active blue strategy (e.g., distinguishing between "Meander" and "B line" attacker strategies).
If no policy exists, the system logs transitions for offline analysis.

Key Contributions

The paper outlines three primary contributions:

Policy Learning Technique: A novel method to learn red agent policies and predict red actions in partially observable RL environments with discrete states and actions, using only blue observations and actions.
Neurosymbolic Integration: The successful integration of this technique into a behavior tree-based autonomous cyber-defense agent, enabling runtime prediction of adversarial movements.
Empirical Evaluation: A comprehensive evaluation using the CybORG CAGE Challenge 2 environment, which utilizes MITRE ATT&CK tactics. The approach demonstrates high accuracy across diverse strategies, including dynamic strategy switching.

Experimental Results

The authors evaluated the technique on the CybORG CAGE Challenge 2 environment using two static attacker strategies (Meander and B line) and a dynamic switching strategy (RedSwitch).

Prediction Accuracy:
- B line: Achieved the highest accuracy (99.93% for blue observation prediction, 95.28% for host-level red action prediction) due to its deterministic, goal-oriented nature.
- Meander: Showed lower accuracy (94.96% observation, 77.71% host-level) due to higher variability in behavior.
- RedSwitch: Performance varied based on the active strategy; accuracy increased when the agent switched to the B line strategy.
- Subnet vs. Host: Subnet-level prediction accuracy consistently exceeded host-level accuracy across all strategies, attributed to the inherent randomness in target host selection.
Intrusion Level Estimation:
- The predicted actions were used to calculate intrusion scores based on compromise levels (No, User-Level, Privilege-Level).
- Results indicated that the blue EBT agent was significantly more effective at mitigating attacks under the Meander strategy compared to B line and RedSwitch, as evidenced by fewer timesteps reaching Privilege-Level Compromise.

Significance and Claims

The paper claims that its proposed technique effectively addresses the challenge of partial observability in autonomous cyber defense. By inferring red policies at runtime, the system enables:

Early detection of adversarial movements before deep network compromise.
Estimation of attacker intrusion levels over time.
Refinement of mitigation strategies by identifying network weaknesses exposed by predicted red actions.

The authors note that while the current evaluation is based on a simulated environment (CybORG CAGE Challenge 2) derived from MITRE ATT&CK, the technique is generalizable to any partially observable RL-based agent with discrete states and actions. Future work is identified as validating the approach in realistic emulated environments and leveraging the predicted red actions to train more robust blue agents. The work is presented as a step toward more adaptive and interpretable autonomous defense systems.