💻 computer science

Multi-Agent LLM Governance for Safe Two-Timescale Reinforcement Learning in SDN-IoT Defense

This paper proposes a self-reflective, two-timescale defense framework for SDN-IoT that integrates fast-timescale, controller-aware PPO agents with slow-timescale, multi-agent LLM governance to dynamically evolve auditable safety policies, thereby significantly improving detection accuracy while preventing control-plane destabilization and QoS degradation under adversarial attacks.

Original authors: Saeid Jamshidi, Negar Shahabi, Foutse Khomh, Carol Fung, Mohammad Hamdaqa

Published 2026-04-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Saeid Jamshidi, Negar Shahabi, Foutse Khomh, Carol Fung, Mohammad Hamdaqa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, bustling city where millions of smart devices (like thermostats, cameras, and sensors) are constantly talking to each other. This is the Internet of Things (IoT). To keep traffic flowing smoothly and stop bad guys from clogging the streets, the city uses a central traffic command center called Software-Defined Networking (SDN).

However, there's a problem: if the command center tries to stop too many bad cars at once, the center itself gets overwhelmed, the traffic lights freeze, and the whole city grinds to a halt. This is exactly what happens in computer networks when hackers launch attacks.

This paper proposes a new, smarter way to manage this city using a "Two-Timescale" system that combines a fast-reacting street cop with a slow-thinking city planner.

The Two Characters in Our Story

1. The Street Cop (Fast Timescale - The RL Agents)

Who they are: These are tiny, independent AI agents sitting on every single street corner (each network switch).
What they do: They react instantly. If they see a suspicious car (a cyberattack), they immediately wave it down, slow it down, or send it to a holding pen.
The Problem: If every cop acts too aggressively at the same time, they flood the central command center with too many reports. The center gets so busy processing these reports that it can't even turn the traffic lights on time. The system crashes.
The Fix: These cops are trained to be "controller-aware." They know that if the command center is already stressed, they shouldn't call for heavy help. They have a rulebook that tells them when to hold back.

2. The City Planner (Slow Timescale - The LLM Governance)

Who they are: This is a team of highly intelligent, reflective AI "planners" (using Large Language Models) who don't run the streets. Instead, they watch the whole city from a high tower.
What they do: They don't make split-second decisions. Instead, they look at the history of the last hour or day. They ask: "Why did the system almost crash? Did the cops get too aggressive? Did the rulebook need changing?"
The Magic: Instead of rewriting the cops' brains (which is risky and slow), the planners rewrite the rulebook. They add a small, safe note like: "If the command center is 80% full, no cop is allowed to call for a heavy arrest; they must just slow traffic down instead."
The Safety Check: Before this new rule goes into effect, the planners run a "stress test." They simulate a massive attack to make sure the new rule actually helps and doesn't make things worse. If it fails the test, the rule is thrown out.

The Creative Analogy: The Orchestra Conductor

Think of the network as a giant orchestra.

The Musicians (The Switches): They are playing their instruments (handling data).
The Conductor (The Controller): They keep the tempo.
The Attack: A chaotic noise trying to drown out the music.

Old Way: The musicians try to play louder and faster to drown out the noise. But they play so fast that the Conductor can't keep up, the sheet music gets lost, and the whole orchestra stops playing.

This Paper's Way:

The Musicians (RL Agents): They play fast and adapt to the noise immediately, but they have a strict instruction: "If the Conductor looks stressed, play softer."
The Composer (LLM Governance): Every few minutes, the Composer listens to the recording of the last hour. If they hear the Conductor struggling, they don't tell the musicians to play a new song. Instead, they write a small, specific note on the sheet music: "When the Conductor is stressed, switch to a quieter rhythm."
The Safety Check: Before the new sheet music is distributed, the Composer runs a simulation to ensure the new rhythm won't cause a crash.

Why This Matters

No More Crashes: By separating the "fast reaction" from the "slow planning," the system prevents the central brain from getting overloaded.
Safe Evolution: The system learns and gets better over time, but it does so carefully. It never makes a huge, risky change without testing it first.
Better Results: In the experiments, this method caught more attacks (9.1% better) and kept the network running smoothly (42% less backlog) compared to older methods that just tried to be "smarter" without thinking about the system's limits.

The Bottom Line

This paper teaches us that in complex, high-speed systems, you can't just rely on speed. You need a fast reaction team and a slow, thoughtful governance team that updates the rules of the game. By letting the "planners" tweak the "rulebook" rather than the "musicians' brains," the system stays safe, stable, and secure even when under heavy attack.

1. Problem Statement

The paper addresses a critical gap in securing Software-Defined Networking (SDN) for Internet of Things (IoT) environments. While SDN offers centralized control and programmability, defense mechanisms often treat mitigation as an isolated detection task, ignoring the closed-loop coupling between mitigation actions and the network's control plane.

The Core Challenge: Aggressive mitigation (e.g., dropping flows, rate limiting) consumes controller resources (CPU, memory) and increases rule-installation delays. In high-traffic or attack scenarios, these actions can overwhelm the SDN controller, causing queue backlogs, increased latency (QoS degradation), and even controller collapse.
Limitations of Existing Approaches:
- Detection-Centric: Focus on accuracy but treat mitigation as a static reaction, ignoring feedback loops.
- Standard RL: Often optimize short-term rewards, leading to unstable behaviors (oscillations) or catastrophic controller overload under complex traffic dynamics.
- Lack of Governance: Existing systems lack structured, auditable mechanisms for policy evolution. Adapting to new threats usually requires retraining models or manual rule tuning, which is opaque and operationally costly.

2. Methodology: Self-Reflective Two-Timescale Architecture

The authors propose a novel framework that separates fast decentralized mitigation from slow reflective governance.

A. Fast Timescale: Decentralized RL Agents

Mechanism: Independent Proximal Policy Optimization (PPO) agents are deployed on each SDN switch.
Input: Agents observe local telemetry (queue occupancy, flow-table pressure, controller backlog, traffic rates) rather than full global state.
Action: Agents select mitigation actions (Allow, Alert, Mirror, Rate Limit, Drop Flow, Quarantine).
Safety Constraint: Actions are filtered through a Safety Filter based on the global policy entity $\Pi$ . If an action violates safety constraints (e.g., would overload the controller), it is masked or replaced with a fallback (e.g., switching from "Drop" to "Rate Limit").
Objective: Optimize a risk-sensitive, multi-objective reward function using Conditional Value at Risk (CVaR) to minimize worst-case tail risks (e.g., controller collapse) rather than just average performance.

B. Slow Timescale: Multi-Agent LLM Governance

Mechanism: A multi-agent Large Language Model (LLM) ensemble operates on a slower timescale (every $K$ steps) to analyze system trajectories and evolve the Global Policy Constitution ( $\Pi$ ).
The Policy Entity ( $\Pi$ ): Instead of updating neural network weights, the system updates an explicit, machine-parsable policy structure containing:
- Safe action masks ( $S_\Pi$ ).
- Safety thresholds ( $\tau$ ).
- Reward weights ( $w$ ).
- Operational caps ( $\chi$ ).
The LLM Pipeline:
1. Critic: Analyzes trace evidence (e.g., backlog spikes, false positives) to diagnose failure modes.
2. Compiler: Converts diagnoses into structured policy deltas ( $\Delta\Pi$ ) adhering to a strict JSON schema.
3. Red-Team: Generates targeted adversarial stress campaigns to test candidate policies.
4. Judge: Validates candidate policies against hard safety and non-regression constraints before deployment.
Governance Logic: Updates are only deployed if they pass Hard-Safety checks (no violation of critical limits) and Non-Regression checks (no degradation in Macro-F1 or RTT). This ensures "fail-closed" and auditable evolution.

C. Closed-Loop Dynamics

The system explicitly models the delay in rule activation and controller backlog ( $d_t$ ). The transition kernel accounts for the fact that mitigation actions today alter the distribution of future observations (queues, delays) for all agents, creating a coupled control problem that standard RL often fails to solve.

3. Key Contributions

Two-Timescale Control Framework: A novel architecture separating fast, decentralized RL for immediate response from slow, structured LLM-based governance for long-horizon policy evolution.
Auditable Policy Constitution ( $\Pi$ ): Introduction of a machine-readable policy entity that encodes safety constraints and reward priorities. The LLM edits this constitution via structured deltas ( $\Delta\Pi$ ) rather than modifying RL parameters, ensuring interpretability and reproducibility.
Controller-Aware Risk Sensitivity: The framework explicitly models SDN control-plane coupling (backlog, delayed actuation) and optimizes for tail-risk (CVaR) to prevent catastrophic system failures.
Structured Multi-Agent LLM Pipeline: A governance loop (Critic, Compiler, Red-Team, Judge) that validates policy changes through deterministic schema checks and stress testing, preventing unstable or unsafe policy drift.

4. Experimental Results

The solution was evaluated in a closed-loop SDN-IoT emulation environment under heterogeneous traffic and adversarial stress.

Performance Gains:
- Macro-F1: Improved by 9.1% over unconstrained PPO and 15.4% over static threshold-based mitigation.
- Worst-Case Stability: Worst-case agent degradation was reduced by 36.8%.
- Controller Health: Controller backlog peaks decreased by 42.7%.
- QoS: Round-Trip Time (RTT) p95 inflation remained below 5.8% even during high-intensity attacks.
Stability: No oscillatory instability or controller collapse was observed across independent runs. The system maintained bounded operation even under synchronized burst attacks.
Governance Efficiency: Policy evolution converged within five reflection cycles. Catastrophic overload events were reduced from 11.6% to 2.3% of episodes without retraining RL parameters.
Trade-off Management: The system successfully navigated the security-cost trade-off, preventing the "diminishing returns" trap where aggressive detection leads to disproportionate QoS degradation.

5. Significance

This paper represents a paradigm shift in SDN-IoT security by reframing defense as a policy-management problem rather than just a detection or parameter-optimization problem.

Safety & Reliability: It demonstrates that separating fast reactive control from slow structural governance allows for adaptive defense without sacrificing system stability.
Operational Viability: By using structured, auditable policy edits ( $\Delta\Pi$ ) instead of opaque neural weight updates, the approach aligns with industrial change-control principles, making it suitable for safety-critical infrastructure.
Scalability: The decentralized nature of the RL agents combined with the centralized but infrequent governance layer offers a scalable solution for large, heterogeneous IoT networks.
Resilience: The focus on tail-risk minimization (CVaR) ensures the system remains robust against rare but catastrophic failure modes, which are the primary concern in critical network operations.

In summary, the proposed solution effectively bridges the gap between adaptive AI-driven defense and the rigid safety requirements of real-world network infrastructure, offering a robust, auditable, and stable framework for SDN-IoT security.