💻 computer science

Adversarial Self-Play for Adaptive Cyber Deception Against Reinforcement Learning-Based Bots in SDN-Enabled Edge Networks

This paper proposes and validates an adaptive cyber deception framework for SDN-enabled edge networks that utilizes adversarial self-play to train a reinforcement learning-based defender, dynamically orchestrating honeypots and moving target defenses to effectively counter RL-trained bots while minimizing resource costs.

Original authors: Yannick Florian YANKAM, Kerol Kafack Wamba

Published 2026-11-04✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yannick Florian YANKAM, Kerol Kafack Wamba

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital world, security has long relied on a simple, static idea: build a high wall, paint a clear line, and wait for the intruder to hit it. Firewalls and fixed rules act like that wall, designed to stop known threats. But the nature of the threat has changed. Today, attackers are not just following a script; they are using artificial intelligence to learn, adapt, and try again until they find a way through. When a computer program can probe a network, fail, learn from that failure, and immediately try a different approach, a static wall eventually becomes a map for the enemy. This leaves a dangerous gap: how do you defend a system when the enemy is constantly rewriting its own playbook?

The answer lies in making the defense just as adaptable as the attack. A new study by researchers at the University of Dschang in Cameroon explores this challenge within the complex environment of edge computing, where data is processed close to where it is created, such as in smart factories or connected hospitals. These networks are fast and flexible, but their many small, connected devices are also easy targets. The researchers proposed a system where the defender does not just wait to be attacked but actively learns how to mislead the intruder. By treating the conflict as a game between two intelligent opponents, they created a defense that evolves in real time, using a technique called adversarial self-play. In this setup, the defensive computer program trains itself by fighting against a learning attacker, forcing both sides to become smarter and more unpredictable with every round of the game.

The researchers built a simulated version of this network to test their idea. They created a digital environment that mimics the structure of a modern edge network, complete with sensors, servers, and the central controller that manages traffic. Into this environment, they introduced two artificial intelligence agents: one acting as the attacker and one as the defender. The attacker's goal was simple: find a way into the network and steal data as quickly as possible. The defender's job was to stop it, but not by simply blocking the door. Instead, the defender used a strategy known as cyber deception. This involves creating fake digital traps, known as honeypots, and constantly changing the network's layout to confuse the intruder.

To make this work, the researchers used a specific type of learning algorithm for both sides. They let the two agents play against each other thousands of times. In the beginning, the attacker was fast and the defender was slow. But as they played, the defender learned to spot the attacker's patterns. When the attacker tried to scan the network, the defender would react by moving the target's digital address or placing a fake server in its path. Crucially, the defender learned to do this only when necessary. The system was designed to understand that changing the network costs energy and time, so it learned to balance the need for security with the need to keep the network running smoothly. The attacker, in turn, learned to spot these traps and try to avoid them, forcing the defender to come up with new tricks.

The results of these simulations were striking. When the researchers tested the system against a network with no defense, the attacker succeeded quickly, often within ten seconds. When they used a traditional, static defense that relied on fixed rules and unchanging traps, the attacker eventually learned the pattern and bypassed it, succeeding in about the same amount of time. However, when the adaptive, learning defense was active, the outcome changed completely. The time it took for the attacker to compromise the network jumped to between fifty and eighty seconds, depending on the target. More importantly, the defender successfully stopped the attack in nearly every single attempt, achieving a success rate of about ninety-eight percent for sensor networks and ninety-seven percent for the main servers.

The study also showed how the attacker's behavior changed under this pressure. Without a smart defense, the attacker would spend most of its time trying to break into known targets. But when faced with the adaptive system, the attacker was forced to spend more time just looking around, trying to figure out where the targets were, because the map it was using kept becoming outdated. The defender managed to keep the network running without noticeable delays, with the time it took to reconfigure the system staying below fifty milliseconds. This speed is critical, as it means the defense can happen without slowing down the real services running on the network.

The researchers emphasized that their findings come from a simulated environment, not a physical one. They built their testbed using software that mimics real hardware, which allows for precise control and repetition but does not capture every real-world variable like hardware failures or wireless interference. They noted that while the system performed exceptionally well in the simulation, the next step would be to test it on actual physical equipment to see how it holds up against the unpredictability of the real world. They also pointed out that their current model assumes the central controller is safe and that the detection of an attack is handled by a separate, reliable system.

Ultimately, this work demonstrates that the best way to fight a learning enemy is with a learning defense. By treating the conflict as a dynamic game where both sides are constantly evolving, the researchers showed that it is possible to keep attackers off-balance. The system does not rely on knowing every possible attack in advance; instead, it learns to react to what is happening right now. This approach offers a promising path forward for protecting the increasingly complex and connected networks that power our modern world, turning the very act of defense into a continuous, intelligent adaptation.

Technical Summary: Adversarial Self-Play for Adaptive Cyber Deception in SDN-Enabled Edge Networks

Problem Statement
The convergence of Edge Computing and Software-Defined Networking (SDN) has created latency-sensitive architectures but introduced significant security vulnerabilities. Peripheral nodes (sensors, IoT devices) are numerous, resource-constrained, and physically accessible, making them ideal entry points for adversaries. Traditional static defenses (firewalls, signature-based detection, fixed-rule honeypots) fail against modern adversaries equipped with Reinforcement Learning (RL). RL-based bots autonomously probe, learn from failures, and adapt their strategies, eventually bypassing static countermeasures that rely on fixed profiles. Furthermore, existing deception systems are often static or rule-driven, allowing adaptive attackers to fingerprint and avoid decoys. There is a critical gap in designing a deception framework where the defender learns and adapts in real-time to counter an adaptive RL attacker within the constraints of edge networks (bandwidth, energy, and latency).

Methodology
The authors propose a dynamic deception framework where the defender learns via Adversarial Self-Play. The system is built on a three-plane SDN architecture (Control, Data, and Edge Application) using the Ryu controller and Mininet for emulation.

Game Formulation: The confrontation is modeled as a two-agent stochastic game, decomposed into two coupled Markov Decision Processes (MDPs) for the attacker and the defender.
- Attacker: An RL agent that scans, exploits vulnerabilities, pivots, and waits. Its state includes scan results, identified services, and a history of actions. Its reward is based on successful exfiltration and information gain, penalized by time.
- Defender: An RL agent observing the network via OpenFlow statistics (throughput, flow counts, source entropy). Its state includes honeypot status, Moving Target Defense (MTD) activation, alert levels, and resource consumption. Its action space includes deploying honeypots, triggering MTD mutations (IP/port changes), or observing.
- Reward Structure: The defender's reward explicitly charges a cost ($c(at)$) for every defensive action to ensure policies remain sustainable on constrained edge nodes. The reward also prioritizes preventing successful attacks (heavy penalty for breach) over merely delaying them.
Learning Algorithm: Both agents are trained using Proximal Policy Optimization (PPO). The authors selected PPO for its stability in multi-agent settings (avoiding policy collapse), its ability to handle large discrete action spaces, and its on-policy nature which suits co-evolving agents.
- Adversarial Self-Play: Agents are trained symmetrically against a pool of the five most recent versions of their opponent. This prevents overfitting to a single static strategy and ensures robustness against a variety of evolving tactics.
- Graduated Response: The defender's actions are governed by alert thresholds. Low alerts trigger observation; moderate alerts trigger honeypot deployment; critical alerts trigger MTD mutations. This minimizes unnecessary resource consumption.
Deception Mechanisms:
- Moving Target Defense (MTD): Reactively invalidates the attacker's network map by mutating IP addresses or ports. A mapping table ( $T_{MTD}$ ) ensures legitimate traffic is transparently redirected to real addresses while the attacker is left with stale information.
- Honeypots: Proactively redirect suspicious traffic to fake servers with fictitious databases to reveal attacker techniques without exposing real assets.
- Flow Isolation: Silently redirects malicious flows to a sinkhole.

Key Contributions

Formalization: The authors formalize the defender-attacker interaction as a two-agent stochastic game with explicit states, actions, and antagonistic rewards, decomposed into coupled MDPs.
Architecture: A deception architecture where the SDN controller acts as both sensor and actuator, orchestrating honeypots, MTD, and flow isolation while maintaining a "transparency invariant" that preserves legitimate traffic.
Cost-Aware Learning: The reward function and observation vector explicitly include operational costs (bandwidth, energy, reconfiguration latency), ensuring learned policies are deployable on constrained edge nodes.
Evaluation Protocol: A three-phase evaluation protocol (No Defense, Static Defense, Dynamic Defense) on an emulated SDN/Edge/IoT testbed using a realistic RL attacker, isolating the specific contribution of online adaptation.

Experimental Results
The framework was evaluated on two scenarios: intrusion into an IoT subnetwork and intrusion into Edge Servers.

Time to Compromise (TTC): Dynamic defense significantly extended the time required for an attacker to succeed. In the IoT scenario, TTC increased from ~10 seconds (no defense) to ~50 seconds. In the Edge Server scenario, it increased from ~10 seconds to 65–80 seconds.
Defensive Success Rate: The dynamic defense achieved a success rate of approximately 98% for the IoT subnetwork and 97% for Edge servers, compared to the collapse of static defenses (which dropped to near zero as attackers learned fixed patterns).
Attacker Behavior: Under dynamic defense, the attacker's behavior shifted from exploitation-focused (74% exploitation in no-defense) to reconnaissance-focused (60% scanning), as the attacker was forced to constantly re-map the network due to MTD invalidations.
Operational Overhead: The system maintained reconfiguration latency below 50ms (approx. 40–45ms), meeting the operational ceiling for edge applications. Resource consumption remained moderate, with the cost-aware reward preventing unnecessary over-deployment of decoys.
Comparison to Baselines: Static defenses initially performed better than no defense but degraded rapidly as the RL attacker learned the fixed mutation schedules. Dynamic defense maintained high performance throughout training.

Significance and Claims
The paper claims to be the first work to combine symmetric PPO agents, SDN orchestration, and an Edge/IoT context to address four specific research gaps:

Static Defenses: Replaces fixed rules with a learning defender that adapts to mutation patterns.
Stationary Adversary: Moves beyond training against fixed targets by using adversarial self-play against a learning attacker.
Operational Cost: Integrates resource costs directly into the learning objective, ensuring deployability on edge nodes.
Benchmarking: Provides a benchmark against a realistic RL attacker in an instrumented SDN/Edge environment, rather than scripted opponents.

The authors conclude that dynamic defense outperforms both the absence of defense and static baselines, effectively raising the time to compromise and defensive success rate while keeping latency within operational limits. They acknowledge limitations, noting that the study relies on emulation (Mininet/Ryu) rather than physical deployment, uses a simplified detection model (Bernoulli draws rather than learned detectors), and that the quadratic growth of the attacker's action space may limit scalability to very large networks without hierarchical approaches. Future work is suggested to include physical testbed validation, scaling via hierarchical policies, and integrating cyber threat intelligence.

Technical Summary: Adversarial Self-Play for Adaptive Cyber Deception in SDN-Enabled Edge Networks

More like this