💻 computer science

Network and Device Level Cyber Deception for Contested Environments Using RL and LLMs

This paper reviews AI-based solutions for dynamic network and device-level cyber deception in contested environments, specifically focusing on the fusion of Large Language Models (LLMs) and Reinforcement Learning (RL) to optimize cost-effective strategies against stealthy attacks on Operational Technology (OT) systems.

Original authors: Abhijeet Sahu, Shuva Paul, Richard Macwan

Published 2026-03-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Abhijeet Sahu, Shuva Paul, Richard Macwan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the security guard for a massive, high-tech power plant. Your job is to keep the lights on and the turbines spinning. But there's a problem: hackers are trying to sneak in.

In the old days, security was like building a tall, thick wall. You tried to stop the hacker from getting in. But hackers are smart; they keep finding new ways to climb over or dig under the wall.

This paper proposes a smarter, trickier strategy: Cyber Deception. Instead of just building a wall, you build a giant, elaborate maze of fake doors and fake rooms inside your plant. You want the hacker to walk into a fake room, think they've found the treasure, and waste all their time there while you watch them from the shadows.

Here is how the authors of this paper are making that maze even smarter using two types of Artificial Intelligence (AI): Reinforcement Learning (RL) and Large Language Models (LLMs).

1. The Two Main Characters in Our Story

To understand the solution, think of the defense system as a team of two specialized robots:

Robot A: The Traffic Cop (The RL Agent)
- What it does: This robot controls the digital roads (routers) inside the power plant.
- Its job: When it sees a suspicious car (the hacker) driving down the street, it quickly changes the traffic signs. Instead of letting the hacker drive to the real control room, it gently (or not so gently) redirects them to a Fake Control Room (a honeypot).
- How it learns: It uses Reinforcement Learning. Think of this like training a dog. Every time the robot successfully tricks the hacker into the fake room, it gets a "treat" (a reward). If the hacker escapes to the real room, it gets a "scolding" (a penalty). Over time, it learns the perfect traffic patterns to keep the hacker lost in the maze.
Robot B: The Method Actor (The LLM Agent)
- What it does: This robot lives inside the Fake Control Room.
- Its job: When the hacker finally walks into the fake room, they expect to see a real computer screen with real data. If the fake room looks like a cheap cardboard cutout, the hacker will realize it's a trap and run away.
- The Magic: This robot uses a Large Language Model (like the tech behind ChatGPT). It doesn't just play back a pre-recorded script. It acts like a Method Actor. If the hacker asks, "What is the voltage on the main line?", the actor instantly generates a realistic, technical answer that sounds exactly like a real engineer would say it. It remembers what it said five minutes ago so it doesn't contradict itself.

2. The Problem with the Old Way

In the past, building these fake rooms was hard and expensive.

Static Traps: You had to build a fake server that looked exactly like a real one. If the hacker asked a weird question, the fake server would freeze or give a robotic answer, giving the game away.
Human Heavy: You needed a team of humans to constantly update these fake servers to make them look real.
Too Slow: By the time humans updated the fake server, the hacker had already figured out it was a trap.

3. The New "Smart Maze" Solution

The authors combined the Traffic Cop and the Method Actor to create a dynamic, living trap.

The Teamwork: The Traffic Cop (RL) decides where to send the hacker. The Method Actor (LLM) decides what to say once the hacker arrives.
The Feedback Loop:
1. The hacker enters the fake room.
2. The Method Actor talks to them.
3. The system checks: "Did the hacker believe us? Did they stay longer?"
4. If the actor did a great job (the hacker stayed), the Traffic Cop gets a huge reward and learns to send more hackers to that specific fake room.
5. If the actor sounded fake (the hacker left), the Traffic Cop learns to stop sending people there and tries a different route.

4. Why This Matters for Power Plants (OT Systems)

The paper specifically talks about Operational Technology (OT), which is the stuff that runs physical things like power grids, water treatment plants, and factories.

The Stakes are Higher: In a regular office, if a hacker steals a password, it's annoying. In a power plant, if a hacker messes with the controls, the lights go out, or worse, equipment explodes.
The Timing is Critical: Real power plant computers talk to each other in split seconds. If your fake computer takes too long to answer, the hacker knows it's a fake.
The Solution: The "Method Actor" (LLM) is trained to answer instantly and with the exact timing of a real machine. It's not just a chatbot; it's a chatbot that knows how to mimic the heartbeat of a power grid.

5. The Results (The "Happy Ending")

The researchers tested this in a simulated power grid (based on a standard model called the IEEE 123-bus system).

Without the AI: Hackers figured out the fake rooms quickly and moved on to the real ones.
With the AI Team: The hackers got stuck in the maze. They spent a lot of time talking to the fake robots, thinking they were hacking the real system.
The Benefit: While the hacker was busy wasting time in the fake room, the real system stayed safe. The defenders got to watch the hacker's every move, learn their tricks, and gather intelligence without ever risking the actual power grid.

Summary Analogy

Imagine a burglar trying to rob a bank.

Old Defense: A big steel door. The burglar picks the lock and gets in.
New Defense (This Paper): The bank has a secret tunnel system.
- The Traffic Cop (RL) sees the burglar and instantly opens a door that leads to a fake vault room.
- The Method Actor (LLM) is inside that room. When the burglar asks, "Where is the money?", the actor says, "It's in the safe behind the painting," and acts exactly like a nervous bank teller.
- The burglar spends 3 hours trying to find the safe behind the painting, completely unaware that the real money is safe in the basement, and the police (the defenders) are recording everything.

In short: This paper shows how we can use smart AI to build a "honey trap" that is so convincing and adaptive that hackers waste their time on fake targets, keeping our critical infrastructure safe.

1. Problem Statement

The paper addresses the inherent asymmetry in cybersecurity, where attackers have ample time for reconnaissance and lateral movement, while defenders must react instantly. This challenge is particularly acute in Operational Technology (OT) environments (e.g., power grids), where delayed responses can cause physical system disruption.

Limitations of Current Deception: Traditional deception methods (honeypots, IP randomization) are often static, expensive to maintain, and require significant human intervention. They are easily detected by skilled adversaries through timing analysis or protocol inconsistencies.
The OT Specific Challenge: Existing Reinforcement Learning (RL) approaches for deception struggle with OT protocols like DNP3 (Distributed Network Protocol 3). Modeling DNP3 as a standard Markov Decision Process (MDP) is infeasible due to the high-dimensional, stateful nature of Industrial Control System (ICS) protocols, which require strict timing semantics, master-outstation state consistency, and unsolicited messaging logic.
The Gap: There is a lack of frameworks that combine adaptive network-level routing with realistic, protocol-aware host-level emulation specifically for OT systems.

2. Methodology

The authors propose a hierarchical, composite RL-LLM architecture that integrates network-level traffic redirection with device-level protocol emulation.

A. System Architecture

The framework consists of three tightly coupled layers:

Network Control Layer (RL Agent): A Multi-Agent Reinforcement Learning (MARL) system where individual routers act as independent agents. They dynamically modify routing policies to redirect suspicious traffic from legitimate RTUs (Remote Terminal Units) to decoy honeypots.
Host Emulation Layer (LLM Agent): A Large Language Model (LLM) driven honeypot that emulates DNP3 outstations. Unlike static honeypots, this agent generates context-aware, protocol-consistent responses to attacker queries.
Evaluation & Feedback Layer: A mechanism that quantifies the "deception quality" of the LLM responses and feeds this signal back into the RL reward function.

B. Key Technical Components

MDP Formulation: The network deception problem is modeled as an MDP ( $M = S, A, P, R, \gamma$ $M = S, A, P, R, γ$ ).
- State: Routing configuration, packet drop rates, interface utilization.
- Action: Rerouting traffic to specific interfaces.
- Reward: A composite cyber-physical reward function designed to prevent "reward hacking." It balances:
  - Cyber Reward: Maximizing traffic to honeypots ( $N_{pot}$ ) and minimizing traffic to real RTUs ( $N_{act}$ ).
  - Physical Reward: Penalizing the loss of critical load buses and rewarding the restoration of power.
LLM Integration & RAG:
- The LLM is not used naively; it is constrained via Retrieval-Augmented Generation (RAG) and Personality Injection (using a personality.yml file) to ensure responses adhere to DNP3 structural compliance and timing constraints.
- Accuracy Metric ( $P_{dec}$ ): The quality of the LLM's response is measured using Perplexity ( $M_{perp}$ ) against a reference sequence. This score is converted into a probability ( $P_{dec}$ ) using a Sigmoid function, which modulates the RL reward. If the LLM generates unrealistic responses, the RL agent is penalized for routing traffic to that node.
Adversary Model: The attacker is assumed to have an initial foothold and operates as a Man-in-the-Middle (MITM), capable of passive traffic inspection, active querying (e.g., circuit breaker status), and malicious command injection.

C. Training Phases

Training Phase: Joint optimization of RL and LLM using simulated attacker interactions. The LLM's personality file is updated iteratively based on interaction data.
Deployment Phase: RL inference only (no online policy updates) with bounded response latency constraints to ensure real-time feasibility.

3. Key Contributions

Formal RL Framework: A single- and multi-agent RL framework for adaptive network-level deception with a bounded composite reward design that accounts for both cyber and physical outcomes.
First Protocol-Aware LLM Honeypot for DNP3: The first implementation of an LLM-driven honeypot specifically for DNP3 outstations that preserves ICS timing and state semantics, moving beyond simple SSH/HTTP emulation.
Composite RL-LLM Architecture: A novel integration where the LLM's deception quality directly influences the RL agent's routing decisions, creating a closed feedback loop.
Computational Analysis: An analysis of overhead and deployment considerations for real-time OT environments, highlighting the trade-offs between simulation fidelity and training time.

4. Results and Analysis

The authors evaluated their approach using an IEEE 123-bus feeder model with DNP3 traffic.

RL Performance (Network Only):
- Multi-Agent vs. Single-Agent: Multi-Agent RL (using PettingZoo) significantly outperformed single-agent approaches in terms of convergence speed and stability. Multi-agent agents achieved shorter episode lengths (approx. 5 steps vs. 14-24 steps) and higher rewards, demonstrating better scalability.
- Algorithm Comparison: Proximal Policy Optimization (PPO) generally showed faster convergence and better stability compared to Advantage Actor-Critic (A2C) in complex scenarios.
LLM Performance:
- Initial LLM responses had low accuracy (high perplexity). Through iterative updates to the personality file and RAG context, the Perplexity score ( $M_{perp}$ ) decreased from 5.07 to 0.12, and the deception probability ( $P_{dec}$ ) increased from 0.06 to 0.90 over 6 time steps.
Combined RL + LLM (Cyber-Physical):
- Integrating the LLM accuracy metric ( $P_{dec}$ ) into the RL reward function significantly improved performance.
- Episode Length: The combined approach reduced the average episode length to 40.2 (compared to 59.25 for CP-only and 54.4 for A2C CP-only), indicating the agent learned to prioritize high-quality deception nodes faster.
- Reward: The combined approach achieved a positive average reward (45.6), whereas some baseline configurations remained negative, indicating successful deception and physical system protection.

5. Significance and Future Work

Significance: This work bridges the gap between AI-driven network defense and realistic OT protocol emulation. By proving that LLMs can be constrained to generate protocol-compliant responses, the authors demonstrate a path toward dynamic, low-cost, and highly effective deception strategies for critical infrastructure. The composite reward function ensures that deception does not come at the cost of physical system stability.
Limitations & Future Scope:
- Scalability: Current multi-agent implementation uses uniform action spaces; future work aims to support heterogeneous action spaces using RLlib.
- Fidelity: The current evaluation relies on simulated environments. Future work plans to deploy in NREL's Cyber Range using Minimega virtualization and HELICS co-simulation to test with real SCADA traffic.
- Sim-to-Real Transfer: Addressing the computational cost of training in high-fidelity environments by exploring Sim-to-Real transfer techniques.

In conclusion, the paper presents a robust, adaptive defense mechanism that leverages the generative capabilities of LLMs and the decision-making power of RL to create a "moving target" defense specifically tailored for the complex constraints of Operational Technology.