🤖 AI

Beyond Rewards in Reinforcement Learning for Cyber Defence

This paper demonstrates that sparse, goal-aligned reward functions outperform dense, engineered rewards in training deep reinforcement learning agents for cyber defence, resulting in more reliable, lower-risk policies that effectively minimize the use of costly defensive actions.

Original authors: Elizabeth Bates, Chris Hicks, Vasilios Mavroudis

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Elizabeth Bates, Chris Hicks, Vasilios Mavroudis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Training a Digital Bodyguard

Imagine you are training a robot to be a security guard for a large office building (the computer network). Your goal is for the robot to stop intruders (hackers) from stealing data or breaking things.

To teach the robot, you use a method called Reinforcement Learning. This is like training a dog: when the dog does something good, you give it a treat (a "reward"); when it does something bad, you scold it (a "penalty"). The robot learns by trial and error, trying to get the most treats possible.

The big question this paper asks is: What is the best way to give treats and scoldings?

The Problem with "Micromanaging" Rewards

Currently, most researchers use what the paper calls "Dense Rewards."

The Analogy: Imagine you are training your security robot by giving it points for every single tiny action.

+1 point for walking left.
-0.5 points for looking at a suspicious person.
-10 points for letting a hacker pass.
+2 points for checking a lock.

This is like a micromanager boss who yells at you for every small mistake and praises you for every small success. The problem? The robot gets confused. It might learn to "game the system." For example, it might realize that not checking a lock gives it a small bonus, so it stops checking locks to maximize its score, even though that makes the building less safe. It becomes obsessed with the numbers rather than the actual goal of keeping the building safe.

The Solution: "Sparse Rewards"

The authors suggest using "Sparse Rewards."

The Analogy: Instead of giving points for every step, you only give a reward at the very end of the shift.

The Goal: Keep the building secure.
The Reward: If the shift ends and no data was stolen, the robot gets a big gold star (+1). If the building is totally compromised, it gets a big zero (-1).

This is much simpler. The robot isn't distracted by tiny, arbitrary point values for every little movement. It focuses entirely on the big picture: Did we keep the bad guys out?

The "Ground Truth" Scorecard

One major issue in previous studies was that researchers judged the robots based on the "points" they earned during training. But as we saw, those points can be misleading.

The authors created a new way to judge the robots, which they call the "Ground Truth Score."

The Analogy: Imagine the robot is playing a video game. The game’s score counter might say you have 1,000 points. But the real question is: Did you actually win the level?
The authors realized that in cyber simulations, a hacker might break into a room during a split second, and the robot might kick them out before the computer registers the "damage." The old scoring system missed these split-second breaches.

The "Ground Truth" score looks at the worst-case reality: "At any point during the shift, how many rooms were actually breached?" It doesn't care about the robot's internal point counter; it cares about the actual security of the building.

What They Found

The researchers tested both "Dense" (micromanaged) and "Sparse" (simple goal-based) rewards in two different simulated network environments. Here is what happened:

Sparse Rewards Worked Better: The robots trained with simple, sparse rewards (just "did we keep the network safe?") were actually better at defending the network than the robots trained with complex, dense rewards.
Less Risky Behavior: The "Dense" reward robots often took risky shortcuts to get points. The "Sparse" reward robots were more cautious and reliable.
Better Training Reliability: When you train a robot with dense rewards, the results are unpredictable. Sometimes it learns well, sometimes it fails. With sparse rewards, the training was much more consistent and reliable.
Smart Use of Resources: Interestingly, the sparse-reward robots didn't waste energy. They didn't constantly "restore" or "patch" systems unnecessarily. They only acted when it truly mattered to keep the network safe. The dense-reward robots often over-reacted because they were trying to maximize small point bonuses.

Why This Matters

The paper argues that in cybersecurity, we shouldn't try to engineer perfect, complex scoring systems for every tiny action. Instead, we should stick to simple, clear goals: Keep the network uncompromised.

By using simple rewards and judging the robots based on the "Ground Truth" (actual breaches) rather than the "Game Score" (training points), we can build AI defenders that are safer, more reliable, and better aligned with what humans actually want: a secure network.

Summary in a Nutshell

Old Way: Give the AI points for every tiny move. Result: The AI gets confused, games the system, and takes unnecessary risks.
New Way: Give the AI a simple reward only if the network stays safe. Result: The AI focuses on the real goal, acts more reliably, and defends better.
Key Innovation: A new scoring method ("Ground Truth") that measures actual security breaches, not just the AI's internal point total.

Technical Summary: Beyond Rewards in RL for Cyber Defence

Problem Statement
The paper addresses a critical limitation in the training of Autonomous Cyber Defence (ACD) agents using Deep Reinforcement Learning (DRL). Current ACD agents are typically trained in "cyber gym" environments using dense, highly engineered reward functions that combine numerous penalties and incentives for various network states and defensive actions. While dense rewards facilitate faster learning by providing frequent signals, they risk biasing agents toward suboptimal and potentially riskier policies. This is particularly problematic in cyber defence, where suboptimal strategies may introduce avoidable weaknesses. Furthermore, existing evaluation methods rely on mean episodic rewards, which assume the Markov Decision Process (MDP) model accurately captures the "ground truth" of the environment. The authors argue that this assumption is flawed because standard cyber gyms overlook intra-step dynamics—specifically, node compromises that occur and are resolved within a single time step, thereby obscuring the true risk and performance of the agent.

Methodology
To investigate the impact of reward structure on agent performance, reliability, and risk, the authors propose a novel evaluation framework and conduct extensive experiments across two established cyber gyms: Yawning Titan (YT) and Cyber Autonomy Gym for Experimentation (CAGE 2, specifically the MiniCAGE implementation).

Ground Truth Scoring ( $Score_{GT}$ ): The authors introduce a ground truth scoring mechanism to evaluate agent performance independently of the reward function used during training. $Score_{GT}$ is defined as the maximum number of compromised nodes observed during both intra-step and end-step evaluations. This metric captures security-relevant events that standard episodic rewards miss, such as transient compromises that are remediated before the step ends.
Reward Functions: The study compares sparse and dense reward structures:
- Sparse Positive (SP): +1 if no nodes are compromised.
- Sparse Negative (SN): -1 if all nodes are compromised.
- Sparse Positive-Negative (SPN): +1 if no nodes compromised, -1 if all compromised.
- Dense Negative (DN): -1 per compromised node.
- Complex Dense Negative (CDN): A heavily shaped reward function with action penalties and state-based penalties, representative of standard cyber gym rewards.
Metrics: Beyond $Score_{GT}$ $S cor e_{GT}$ , the authors evaluate:
- Training Reliability: Measured via Dispersion Variability across time ( $\bar{DT}$ ) and across runs ($DR'$), assessing the stability and reproducibility of training.
- Risk: Measured via Conditional Value at Risk (CVaR) on fixed-policy rollouts ($RF$), quantifying worst-case performance.
Experimental Setup: Experiments were conducted using Policy Gradient (PPO) and Value-Based (DQN) algorithms. The authors varied network sizes (2 to 50 nodes in YT), action spaces (basic vs. extended with decoy actions), and agent ordering (Red-then-Blue, Blue-then-Red, and Random) to test robustness against uncertain attacker timing.

Key Contributions

Ground Truth Evaluation: The proposal of a ground truth scoring mechanism that allows for direct, reproducible comparison between agents trained with different reward functions, revealing nuances in policy behavior obscured by standard metrics.
Comprehensive Evaluation: A thorough assessment of sparse versus dense rewards across two cyber gyms, multiple network topologies, and different DRL algorithms.
Empirical Evidence for Sparse Rewards: Demonstration that sparse, goal-aligned rewards can enhance the effectiveness, reliability, and risk profiles of ACD agents, often outperforming dense rewards without explicit numerical penalties for costly actions.

Results

Performance: Sparse rewards, particularly SP and SPN, consistently achieved the lowest $Score_{GT}$ (fewest compromised nodes) across both YT and MiniCAGE environments. In YT, SPN yielded the best average performance, followed closely by SP. In MiniCAGE, SP achieved the best $Score_{GT}$ .
Reliability: Sparse rewards exhibited significantly lower dispersion variability ( $\bar{DT}$ and $DR'$) compared to dense rewards, indicating higher training reliability and reproducibility. Dense rewards showed $\bar{DT}$ values an order of magnitude larger, suggesting unstable training.
Risk: Agents trained with sparse rewards demonstrated lower worst-case risks ($RF$). In YT, as network size increased, the performance gap between sparse and dense rewards widened, with sparse rewards maintaining superior risk profiles.
Agent Ordering: The random agent ordering (simulating uncertain attacker timing) significantly degraded performance for all reward functions. However, SPN significantly outperformed other rewards in this challenging setting. In the "Blue-then-Red" ordering with extended action spaces, SP agents achieved ideal security (0 compromised nodes) in many runs.
Policy Behavior: Analysis of MiniCAGE policies revealed that sparse-reward agents (SP/SPN) confined adversary impact primarily to user hosts (lowest privilege) and used costly "restore" actions more sparingly than dense-reward agents. This suggests sparse rewards align better with real-world security objectives without requiring explicit numerical penalties for action costs.
Ablation Studies: Experiments confirmed that the superiority of sparse rewards stems from their sparsity and goal alignment rather than the numerical sign of the reward. Constant shifts to sparse rewards did not negate their performance advantages.

Significance and Claims
The paper claims to be the first to address the need for evaluation methods that consider the ground truth of complex cyber environments beyond the constraints of the RL framework. The authors argue that dense rewards, while standard practice, risk constraining agent performance and introducing avoidable risks due to arbitrary numerical equivalences and potential misalignment with human domain knowledge. They conclude that sparse, goal-aligned rewards are preferable for ACD because they offer enhanced training reliability, lower-risk policies, and better alignment with defensive objectives. The proposed ground truth scoring mechanism is presented as a necessary tool for robust evaluation and reward design in cyber defence, highlighting the importance of considering intra-step dynamics and worst-case risks in operationalizing ACD agents. The authors emphasize that their findings motivate a shift toward sparse rewards and more rigorous, risk-aware evaluation practices in the field.