💻 computer science

Proximal Policy Optimization-Based Intrusion Detection with Deep Latent Feature Learning

This paper proposes a novel intrusion detection framework that combines a two-stage stacked autoencoder for deep feature extraction with a Proximal Policy Optimization-based deep reinforcement learning agent, demonstrating competitive performance and efficiency across multiple benchmark datasets.

Original authors: Brahim El malki, Nidal Nasser, Ahmed El Ouadrhiri, Khalid EL FAZAZY, Hamid Tairi, Jamal Riffi

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Brahim El malki, Nidal Nasser, Ahmed El Ouadrhiri, Khalid EL FAZAZY, Hamid Tairi, Jamal Riffi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your computer network is a massive, bustling city. Every day, millions of cars (data packets) drive through the streets. Most are normal commuters going to work or school, but occasionally, a thief in a stolen car tries to break into a bank or crash into a building.

For a long time, security guards (traditional Intrusion Detection Systems) have tried to catch these thieves by checking a "Wanted List." If a car matches a face on the list, they stop it. But what if the thief wears a mask, changes their car, or drives a vehicle no one has seen before? The old guards get confused, miss the bad guys, or accidentally stop innocent people (false alarms).

This paper introduces a new, smarter security system that learns on the job, rather than just memorizing a list. Here is how it works, broken down into simple steps:

1. The "Compression Suit" (Deep Latent Feature Learning)

First, the system looks at the traffic. The city is chaotic, with too much data to process at once. Imagine trying to describe a 100-page novel to a friend in one sentence. You'd have to strip away the fluff and keep only the most important plot points.

The researchers built a "Two-Stage Stacked Autoencoder" to do exactly this.

Stage 1: It takes the messy, high-dimensional traffic data and squishes it down, like folding a giant map into a small pocket. It keeps the "important stuff" (the shape of the car, the speed, the route) but throws away the noise.
Stage 2: It folds that already-small map even tighter.
The Result: Instead of looking at thousands of data points, the system now looks at a tiny, 16-point summary of the traffic. This makes the job much faster and clearer.

2. The "Smart Guard" (Deep Reinforcement Learning with PPO)

Once the data is compressed, it goes to the "Smart Guard." This isn't a guard following a rulebook; it's a Deep Reinforcement Learning (DRL) agent. Think of this agent as a video game character learning to play a level for the first time.

The Game: The agent sees the compressed traffic summary (the state).
The Choice: It has to decide: "Is this a normal car (0) or a thief (1)?"
The Reward System: This is the secret sauce. The agent learns through trial and error using a specific scoring system called Proximal Policy Optimization (PPO):
- If it correctly spots a thief: +5 points (Great job!).
- If it correctly lets a normal car pass: +1 point (Good job).
- If it stops an innocent car (False Alarm): -1 point (Oops, sorry).
- If it lets a thief slip by (False Negative): -10 points (Huge penalty!).

Because the penalty for missing a thief is so much higher than the penalty for a false alarm, the agent learns to be very careful. It keeps playing the game over and over, adjusting its strategy to get the highest score possible.

3. The Training Grounds

To make sure this new system actually works, the researchers tested it in three different "simulated cities" (datasets) that represent different eras of cyber threats:

NSL-KDD: An older, classic dataset (like a 1990s city).
UNSW-NB15: A modern city with new types of traffic.
CIC-IDS2017: A very complex, real-world city with heavy traffic and sophisticated thieves.

The Results: How Well Did It Do?

The paper claims this new "Compression Suit + Smart Guard" combo beat almost every other method they tested (like Random Forests, SVMs, and standard Deep Learning models).

Accuracy: On the most complex dataset (CIC-IDS2017), it got 98.9% accuracy. That means out of 1,000 cars, it only made about 11 mistakes.
Catching the Rare Thieves: One of the hardest jobs for security systems is catching the "rare" thieves (attacks that happen very rarely, like U2R or R2L). Traditional systems often miss these. This new system improved its ability to catch these rare attacks by 13.8% to 22.1% compared to other models.
Speed: It can process traffic fast enough to be used in real-time (about 2.3 milliseconds per check), making it suitable for high-speed networks.

In Summary

The paper presents a framework that first simplifies the complex world of network traffic into a small, manageable summary, and then uses a learning agent that is heavily punished for missing bad guys. This combination allows the system to adapt to new threats and catch intruders more accurately than older, static methods, all while running fast enough for real-world use.

Technical Summary: Proximal Policy Optimization-Based Intrusion Detection with Deep Latent Feature Learning

Problem Statement
Modern Intrusion Detection Systems (IDS) face significant challenges due to the escalating sophistication of cyber threats, the proliferation of heterogeneous technologies (IoT, cloud, 5G), and the limitations of conventional signature-based and anomaly-based methods. Existing machine learning (ML) approaches often suffer from high false-positive rates, computational inefficiency when processing high-dimensional traffic, and a reliance on historical data that hinders their ability to detect zero-day attacks. There is a defined research gap for adaptive, efficient, and intelligent detection paradigms capable of learning optimal security policies in dynamic environments.

Methodology
The authors propose a novel hybrid framework that integrates deep feature extraction with adaptive decision-making. The architecture consists of two primary components:

Two-Stage Stacked Autoencoder (2S-SAE):
- Purpose: To perform unsupervised feature extraction and dimensionality reduction on high-dimensional network traffic data while preserving critical characteristics.
- Structure: The system employs a hierarchical compression process.
  - Stage 1: A deep autoencoder maps raw input features ( $R^d$ ) to a 64-dimensional latent space using an encoder-decoder symmetry ( $R^d \to 128 \to 64 \to 128 \to R^d$ ).
  - Stage 2: The weights of the Stage 1 encoder are frozen. A second, more compact autoencoder refines the 64-dimensional latent vectors into a final 16-dimensional representation ( $R^{64} \to 32 \to 16 \to 32 \to R^{64}$ ).
- Outcome: This process achieves a compression ratio of $d:16$ , generating compact latent representations ( $z_{16}$ ) that serve as the state input for the reinforcement learning agent.
Proximal Policy Optimization (PPO) Agent:
- Framework: The compressed features are fed into a Deep Reinforcement Learning (DRL) agent operating on an Actor-Critic mechanism.
- Formulation: The detection task is modeled as a Markov Decision Process (MDP) where the state is the 16-dimensional latent vector, and the action is a binary classification ($0$: Normal, $1$: Intrusion).
- Reward Mechanism: A cost-sensitive reward function is designed to prioritize critical detections:
  - True Positive: $+5$ (or $+1.0$ in implementation details)
  - True Negative: $+1$ (or $+1.0$ )
  - False Positive: $-1$ (or $-1.5$)
  - False Negative: $-10$ (or $-3.0$)
- Optimization: The agent uses the PPO-clip objective with Generalized Advantage Estimation (GAE) to ensure stable policy updates, allowing the system to iteratively refine its detection strategy through reward-guided learning.

Key Contributions
The paper outlines four main contributions:

Novel Framework: A DRL-based IDS that effectively combines Stacked Autoencoders for feature representation with Actor-Critic learning for adaptive classification.
Comprehensive Evaluation: Robust performance validation across three distinct benchmark datasets (NSL-KDD, UNSW-NB15, and CIC-IDS2017) representing different generations of network threats.
Efficient Training Environment: The development of a pseudo-environment for DRL training that overcomes sequential dependency limitations inherent in standard network traffic processing.
Optimization for Constrained Environments: Strategies and results demonstrating the framework's suitability for resource-constrained environments, such as IoT networks, via efficient feature compression and real-time inference capabilities.

Experimental Results
The framework was evaluated on NSL-KDD, UNSW-NB15, and CIC-IDS2017 using five independent runs with different random seeds. Key findings include:

Performance Superiority: The proposed 2S-SAE+PPO framework consistently outperformed baseline models (Random Forest, SVM, XGBoost, MLP, LSTM, AE-DNN) across all metrics.
- On CIC-IDS2017, the model achieved 98.9% accuracy, 99.0% precision, 98.8% recall, and an F1-score of 0.989.
- On NSL-KDD, it achieved 98.6% accuracy and an AUC of 0.998.
- On UNSW-NB15, it achieved 96.3% accuracy and an AUC of 0.990.
Rare Attack Detection: The framework showed significant improvements in detecting rare attack categories (U2R, R2L), with F1-score gains of 13.8–22.1 percentage points compared to baselines, attributed to the asymmetric reward structure penalizing false negatives heavily.
Statistical Significance: Paired t-tests confirmed that the improvements over baselines (e.g., LSTM) were statistically significant ( $p < 0.01$ ).
Efficiency: The model demonstrated an inference latency of 2.3 ms and a throughput of 111,300 samples/second on an NVIDIA RTX 3090 GPU, meeting real-time requirements for high-speed networks (10 Gbps).
Ablation Study: Comparisons confirmed that the two-stage compression (hierarchical feature learning) and the PPO decision-making mechanism (vs. static classifiers) were the primary drivers of performance gains.

Significance and Claims
The authors claim that this work addresses the limitations of static ML models by introducing an adaptive system capable of learning optimal security policies in complex, dynamic environments. The significance lies in the successful integration of deep latent feature learning with reinforcement learning, resulting in a system that:

Reduces computational overhead through hierarchical feature compression.
Adapts to evolving attack patterns via reward-guided learning.
Achieves superior detection accuracy and lower false-positive rates compared to traditional deep learning models.
Operates efficiently enough for near-real-time deployment in modern network infrastructures.

The paper concludes that while the RL component increases training complexity, the performance gains (1.2–4.3% improvement in F1-score) justify the cost, particularly for detecting minority classes and zero-day threats. Future work is identified as exploring online continual learning, federated deployment, and adversarial robustness.