🤖 AI

Safety-Contract Graph Multi-Agent Reinforcement Learning for Autonomous Network Security Response

This paper introduces ACD³-GAT, a safety-contract graph multi-agent reinforcement learning framework that effectively balances autonomous network security response performance with strict operational budget constraints, significantly reducing downtime violations and costs compared to traditional reward-only approaches in the CAGE Challenge 4 benchmark.

Original authors: Jose Luis Lima de Jesus Silva

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jose Luis Lima de Jesus Silva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy Security Operations Center (SOC) as a high-stakes control room for a massive digital city. Every day, thousands of alarms go off. Human guards (analysts) have to decide: Is this a real intruder? Should we lock down a server? Should we reboot a computer?

The problem is that humans get tired, and there are too many alarms. So, researchers tried to build AI robots to do this job automatically. They used a type of AI called Reinforcement Learning, which is like training a dog: if the dog catches a ball, it gets a treat (a reward); if it misses, it gets nothing.

The Problem: The "Treat-Only" Dog

In this paper, the researchers found a major flaw in how these AI robots were being trained.

Imagine you are training a guard dog to stop burglars. You tell the dog: "If you catch a burglar, you get a steak!"
The dog learns quickly. It starts catching burglars. But, in its excitement to get steaks, it also starts biting the mailman, the neighbors, and even the family cat because it thinks they might be burglars.

In the digital world, this is exactly what happened. The AI agents were trained only to maximize "security rewards" (catching the bad guys). To do this, they started:

Rebooting (restoring) computers constantly, even when they weren't infected.
Changing firewall rules wildly, disrupting normal business.
Ignoring the fact that these actions take time and money.

The result? The AI was technically "good" at catching threats, but it was operationally disastrous. It exhausted the company's budget for downtime and annoyed the human analysts so much that the system couldn't be used in real life. It was like a guard dog that saved the house but ate all the furniture.

The Solution: The "Safety Contract"

The authors, Jose Luis Lima de Jesus Silva and colleagues, proposed a new way to train these AI agents. Instead of just giving them a "treat" for catching bad guys, they gave them a Safety Contract.

Think of this contract as a strict budget for the AI's actions:

The Downtime Budget: You can only reboot computers 50 times per day. If you go over, you fail.
The False Alarm Budget: You can only accuse innocent computers of being infected 10 times. If you accuse too many, you fail.
The Firewall Budget: You can only change network rules 20 times.

The AI has to learn to catch the bad guys without breaking these rules.

The New AI: ACD3-GAT

The paper introduces a new system called ACD3-GAT. To understand how it works, imagine a Smart Traffic Controller in a complex city:

The Graph Attention Network (GAT): Instead of looking at the network as a flat list of computers, the AI sees it as a map of connections. It understands that if a virus is on "Server A," it might spread to "Server B" next. It pays attention to the most important connections, just like a traffic controller focusing on the busiest intersections.
The Safety Shield: Before the AI makes a move, a "Safety Shield" checks the contract. If the AI wants to reboot a computer but the "Downtime Budget" is empty, the Shield says, "No! You've used your 50 reboots. You must sleep instead."
The Counterfactual Risk: The AI doesn't just look at now; it simulates the future. It asks, "If I block this traffic, will it cause a bigger problem later?" It uses a "what-if" engine to predict risks before acting.

The Results: From Chaos to Control

The researchers tested this new system in a simulated environment called CAGE Challenge 4.

The Old Way (Reward-Only): The AI violated the downtime budget 100% of the time. It rebooted computers 300+ times when the limit was 50. It was a disaster.
The New Way (Safety Contract):
- One version of the new system (C-MAPPO-GAT) reduced violations to almost 0%. It stayed within the budget perfectly, though it was a bit more conservative (less aggressive).
- The full ACD3-GAT system found a sweet spot. It reduced downtime costs by about 85% compared to the old AI. It violated the budget only 13.8% of the time, which the authors say is a much more realistic and usable position for a real-world system.

The Takeaway

The paper concludes that you cannot build a safe, autonomous security system just by telling the AI to "be good." You must explicitly teach it the rules of the road (the budget).

Without these rules, the AI is like a race car driver with no brakes: fast, but dangerous. With the Safety Contract, the AI becomes a professional driver who knows how to win the race and keep the car (and the passengers) safe.

In short: The paper proves that for AI to be useful in real-world security, it must be trained not just to win, but to stay within the budget. The new system, ACD3-GAT, is the first step toward making autonomous security agents that humans can actually trust and deploy.

Technical Summary: Safety-Contract Graph Multi-Agent Reinforcement Learning for Autonomous Network Security Response

1. Problem Statement

The paper addresses the critical gap between theoretical security performance and operational deployability in autonomous network security response systems. While Multi-Agent Reinforcement Learning (MARL) agents can optimize security rewards in simulation (e.g., CAGE Challenge 4), they often fail to respect the operational constraints of a Security Operations Centre (SOC).

In real-world SOC environments, response actions carry specific costs:

Downtime/MTTR: Restoring a compromised host takes it offline, consuming a "Mean Time to Recover" budget.
False Positives: Restoring hosts without clear malicious evidence burdens analysts and erodes trust.
Firewall Disruption: Modifying traffic zones requires change-management approval and creates audit trails.

The core problem is that standard reward-only MARL agents, when trained to maximize security scores, systematically exhaust these operational budgets. In the CAGE Challenge 4 evaluation, unconstrained agents violated the downtime budget in 100% of episodes, consuming 6–9 times the allowable budget per episode. The paper posits that an autonomous response system is not deployable if it cannot operate within explicit SOC governance constraints, regardless of its security reward.

2. Methodology

The authors propose a Safety-Contract Graph MARL framework, instantiated primarily through two architectures: C-MAPPO-GAT and ACD3-GAT. The approach treats the problem as a Constrained Decentralized Partially Observable Markov Decision Process (Dec-POMDP).

2.1. Core Framework Components

Operational Budgets: Three explicit budget counters are defined per episode:
- $B_{down} = 50$ (Restore actions).
- $B_{fp} = 10$ (False-positive Restore actions).
- $B_{fw} = 20$ (Firewall change actions).
- An episode is only "safe" if the cumulative cost for each constraint remains within its budget.
Graph Observation Encoding: Agents observe binary vectors representing network states. These are parsed into host-subnet graphs. A Graph Attention Network (GAT) encoder processes these graphs to capture topological dependencies, replacing standard MLP encoders.
Lagrangian Constrained Optimization: The framework uses a Lagrangian relaxation approach. A dual variable (multiplier $\lambda_k$ ) is updated after each batch to penalize policies that exceed specific budgets. The advantage function is modified to subtract cost penalties:
$\hat{A}^{Lag}_t = \hat{A}^r_t - \sum_k \lambda_k \hat{A}^{c_k}_t$
Counterfactual Action Screening (Shielding): Before an action is executed, a "shield" evaluates candidate actions.
- Hard Guard: Actions that would immediately exhaust a remaining budget are blocked.
- Graph Counterfactual Risk Propagation (G-CRP): A deterministic model propagates risk through the network graph to estimate the post-action compromise probability. Actions are ranked by a safety score $Q_{safe}$ , which combines predicted risk, cost, and budget constraints.
Temporal Contract Graph Shielding (TCGS): A diagnostic extension that uses a learned recurrent model to predict the probability of future budget violations based on trajectory history, acting as a temporal filter for action proposals.

2.2. Architectures Evaluated

C-MAPPO-GAT: A constrained baseline combining Multi-Agent PPO (MAPPO) with a GAT encoder and Lagrangian cost control. It isolates the effect of explicit operational-cost constraints.
ACD3-GAT (Adaptive Constrained Counterfactual Decisioning with a Graph Attention Network): The full integrated architecture. It adds:
- Budget context vectors to the policy input.
- CVaR (Conditional Value-at-Risk) reweighting to penalize tail-risk (worst-case) episodes.
- Opponent-belief state estimation (via GRU).
- The G-CRP and TCGS shielding mechanisms.
- Override signals to trigger human-in-the-loop fallbacks (e.g., SLEEP action) when confidence is low or budgets are exhausted.

3. Key Contributions

Explicit Safety Contracts: The paper moves operational constraints from implicit side-effects to explicit, auditable "safety contracts" that are optimized and screened during training and execution.
ACD3-GAT Framework: A novel architecture that integrates graph perception, Lagrangian constraint learning, tail-risk accounting, and counterfactual risk propagation into a single MARL system.
Empirical Evidence of Reward-Only Failure: The study demonstrates that reward-only MARL (IPPO, MAPPO-GAT) is fundamentally non-deployable in this domain, violating downtime budgets in 100% of episodes.
Controlled Baselines: The introduction of C-MAPPO-GAT serves as a controlled baseline to prove that the safety improvements come from the constraint machinery (Lagrangian + shielding) rather than just the graph encoder or reward shaping.

4. Results

Experiments were conducted on CAGE Challenge 4 using 5 cooperating "Blue" agents against a "Red" adversary over 500-step episodes.

4.1. Safety Compliance

Unconstrained Methods: All reward-only baselines (IPPO, MAPPO-GAT, etc.) achieved a 100% downtime-budget violation rate. Their mean downtime costs ranged from 311 to 430 against a budget of 50.
C-MAPPO-GAT: Reduced the downtime violation rate to 0.3% (2 violations in 600 episodes) and lowered the mean downtime cost to 15.5. This represents a 95–96% reduction in operational harm compared to unconstrained MAPPO-GAT.
ACD3-GAT: Achieved a mean downtime cost of 48.2 with a 13.8% violation rate. While less conservative than C-MAPPO-GAT, it operates on the "safety-contract frontier," balancing risk and reward more effectively than unconstrained methods.

4.2. Return vs. Safety Trade-off

Constrained methods incurred a penalty in mean security return (e.g., C-MAPPO-GAT returned -6,992 vs. -3,937 for unconstrained MAPPO-MLP).
However, the paper argues this trade-off is necessary: a policy with high security reward but 100% budget violation is operationally useless.
Tail Risk: Graph encoders (GAT) improved the worst-tail return (CVaR-10%) compared to MLP encoders, but only the constrained methods eliminated catastrophic episodes (alert threshold breaches).

4.3. Robustness Stress Tests

Topology Shifts: C-MAPPO-GAT maintained 0% downtime violations on unseen network topologies, whereas unconstrained methods remained at 100% violation.
Adaptive Red Process: When tested against an adaptive Red PPO agent, constrained policies showed significantly lower worst-case degradation in performance compared to reward-only MAPPO-GAT.

5. Significance and Claims

The paper modestly claims that its primary contribution is establishing a deployable decision-support pattern for autonomous SOC agents, rather than claiming a fully autonomous production-ready system.

Operational Discipline: The central finding is that "reward-only learning lacks operational discipline." The paper demonstrates that without explicit safety contracts, MARL agents learn strategies that are operationally destructive (e.g., exhausting MTTR budgets).
Auditability: The framework turns the agent into an auditable component where actions can be checked against local budgets, providing a clear path for human governance and escalation.
Frontier Positioning: The results distinguish between the C-MAPPO-GAT configuration (the most reliable compliance point in the benchmark) and the ACD3-GAT architecture (the general framework for integrating safety, graph perception, and risk).
Limitations: The authors acknowledge that the safety guarantees are empirical (based on violation rates in simulation) rather than formal mathematical proofs for non-convex neural policies. The "safety contract" is an evaluation and training discipline within the CAGE-4 simulator, and real-world deployment would require tuning budgets to specific organizational SLAs.

In conclusion, the paper argues that for autonomous network security to be viable, the learning objective must explicitly optimize for operational acceptability alongside security rewards, using mechanisms like Lagrangian constraints and counterfactual shielding to prevent budget exhaustion.