🤖 machine learning

Multi-Level Distributional Entropy for Explainable Network Intrusion Detection

This paper introduces Multi-Level Distributional Entropy (MDE), an interpretable framework that derives entropy-based features directly from flow-level statistics to enable effective and reproducible network intrusion detection without requiring raw packet data or training, while also revealing critical performance failures hidden by aggregate metrics.

Original authors: Mohamed Aly Bouke, Md Shohel Sayeed, Swee-Huay Heng, Azizol Abdullah, Mohamed Othman

Published 2026-06-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohamed Aly Bouke, Md Shohel Sayeed, Swee-Huay Heng, Azizol Abdullah, Mohamed Othman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Blind" Security Guard

Imagine a security guard at a busy airport. Their job is to spot terrorists (hackers) among thousands of regular travelers (normal internet traffic).

For years, this guard has been using a checklist of totals: "How many bags did this person carry?" "How long were they in the line?" "How heavy was their luggage?" These are like the standard statistics used in computer security (packet counts, data volume).

The problem is that these totals miss the pattern. A terrorist might carry the exact same number of bags as a tourist, but they might be carrying them in a very rigid, robotic way, while the tourist moves naturally. The old checklist ignores these subtle behavioral differences.

This paper introduces a new tool called MDE (Multi-Level Distributional Entropy). Instead of just counting bags, MDE analyzes the rhythm and variety of the traffic to see if it looks "human" or "robotic."

Part 1: How MDE Works (The Three Levels)

The researchers created a way to calculate "entropy" (a measure of randomness or chaos) without needing to see every single piece of data. They do this in three creative ways:

Level 1: The "Rhythm Check" (Gaussian Differential Entropy)
- The Analogy: Imagine listening to a drummer. A human drummer has natural variations; sometimes they hit the snare a little harder, sometimes a little softer. A robot drummer hits with perfect, identical force every time.
- The Science: MDE looks at the size of data packets. If they are all the exact same size (like a robot), the "entropy" is low. If they vary naturally (like a human), the entropy is high. MDE calculates this mathematically just by looking at the average and the spread of the sizes, without needing to see the individual packets.
Level 2: The "One-Way Street" Check (Jensen-Shannon Divergence)
- The Analogy: A normal conversation is a two-way street: you talk, I talk back. A shouting match or a megaphone blast is one-way: I scream at you, you say nothing.
- The Science: Many cyberattacks (like DDoS floods) send a massive amount of data in one direction and get almost nothing back. MDE measures how "lopsided" the traffic is. If the traffic is perfectly balanced, it's likely human. If it's a one-way street, it's likely an attack.
Level 3: The "Uniformity Check" (Flag-Pattern Entropy)
- The Analogy: Think of traffic lights. A normal day has a mix of red, yellow, and green lights. A glitched system might be stuck on just "Red" or just "Green" for hours.
- The Science: Internet traffic uses "flags" (tiny control signals) to manage connections. Normal traffic uses a diverse mix of these flags. Attack tools often get stuck using just one type (like only "SYN" flags for a flood). MDE checks if the traffic is using a diverse "language" or a repetitive, broken one.

Part 2: The "Black Box" Problem (Explainability)

In the past, advanced computer models were like black boxes. They would say, "This is an attack," but no one knew why. Security analysts couldn't trust a system they couldn't understand.

This paper uses a tool called SHAP (a "translator" for AI).

The Analogy: If the AI says "Arrest this person," SHAP prints a receipt showing exactly which clues led to that decision. "We arrested them because their rhythm was robotic (Level 1) and they only shouted one way (Level 2)."
The Result: The researchers found that MDE features are very consistent. The AI reliably uses these "rhythm" and "one-way" clues to make decisions, and it does so in a way that makes sense to human experts.

Part 3: The Reality Check (Why "Average" Scores Lie)

The most important finding of this paper isn't just about the new tool; it's about how we test security systems.

The researchers argue that looking at a single "average score" (like an F1 score) is like looking at a class average grade without seeing the individual test results.

The Analogy: Imagine a class where 99% of students get 100% on a test, and 1% get 0%. The class average is 99%. That looks like a perfect class! But if the 1% who got 0% were the ones who failed a critical safety exam, the "99% average" is a lie.

What the paper found:

On one dataset (CICIDS-2018), the system looked great with a score of 0.74. But when they looked closer, they realized the system was missing 52% of the actual attacks. The "average" score hid the fact that the system was failing half the time.
On another test, the system was trained on Monday–Thursday traffic and tested on Friday. When the "Friday" traffic changed slightly (a new type of attack), the system's "average" score looked okay, but in reality, it stopped detecting attacks entirely (0% detection rate). The math said the system was still "ranking" attacks correctly, but the alarm didn't go off because the threshold was wrong.

Part 4: The Conclusion

What MDE actually does:
It doesn't necessarily make the computer "smarter" at finding attacks than the old methods (the scores are often the same). Instead, it gives the computer a better, more logical way of looking at the data.

It works without needing raw, messy data (it uses summaries).
It explains why it made a decision.
It reveals when the system is actually failing, even when the "average score" looks good.

The Limitations:
The paper admits that if the "robot" changes its behavior completely (a new type of attack never seen before), the system will fail, just like any other security system. Also, the math assumes traffic behaves somewhat like a bell curve (Gaussian), which isn't always true for encrypted or complex traffic.

In short: The paper builds a better "rhythm detector" for internet traffic that is transparent, easy to understand, and honest about when it is failing, rather than hiding behind a single, misleading number.

Technical Summary: Multi-Level Distributional Entropy for Explainable Network Intrusion Detection

Problem Statement
Current Machine Learning-based Network Intrusion Detection Systems (IDS) rely heavily on aggregate flow statistics (e.g., packet counts, byte volumes, connection duration) extracted by tools like CICFlowMeter. While effective for magnitude-based detection, these aggregate descriptors discard the distributional and sequential structure of traffic, which often differs systematically between benign sessions and attacks. Traditional information-theoretic measures (e.g., Shannon entropy, Approximate Entropy) are known to capture this structural difference but require ordered sequences of raw packet data. Since most IDS benchmarks and operational pipelines utilize pre-aggregated flow datasets where raw packets are unavailable, there is a practical barrier to integrating entropy-based features. Furthermore, existing high-performing ensemble classifiers are often opaque, creating compliance and audit challenges in regulated environments. Finally, prevailing evaluation protocols often rely on single-dataset, aggregate F1 scores, which can conceal critical operational failure modes such as low detection rates (DR) or poor generalization under temporal shifts.

Methodology: The MDE Framework
The authors propose Multi-Level Distributional Entropy (MDE), an analytical framework that derives interpretable entropy features directly from pre-aggregated flow summary statistics without requiring raw packet access or training data. MDE operates on three complementary levels:

Level 1 (Within-Flow Gaussian Differential Entropy - ADE): Assumes packet sizes and inter-arrival times follow a Gaussian distribution. It computes differential entropy ( $h(X) = \frac{1}{2}\ln(2\pi e\sigma^2)$ ) using the mean and standard deviation available in flow records. This captures within-flow complexity; attack tools often produce low-variance (regular) traffic, resulting in lower entropy compared to variable benign traffic.
Level 2 (Cross-Directional Jensen-Shannon Divergence - JSD): Measures the distributional asymmetry between forward (source-to-destination) and backward (destination-to-source) traffic. It approximates the JSD between two Gaussian distributions parameterized by flow statistics. This captures directional imbalances common in attacks (e.g., DDoS floods or port scans) versus the bidirectional symmetry of benign sessions.
Level 3 (TCP Flag-Pattern Shannon Entropy): Computes Shannon entropy over the distribution of TCP control flags (FIN, SYN, RST, etc.) within a flow. Low entropy indicates flows dominated by a single flag type (e.g., SYN floods), a signature of specific attack patterns.

Key Contributions

Analytical Feature Construction: MDE constructs 7–12 interpretable entropy features analytically from standard flow statistics (means, standard deviations, counts). It requires no raw packet data, no training data for feature extraction, and no retraining when applied to different datasets with varying schemas.
Leakage-Free Operational Evaluation Protocol: The authors introduce a rigorous evaluation framework that avoids data leakage (using fold-local preprocessing) and reports a full suite of operational metrics (Detection Rate, False Alarm Rate, MCC, PR-AUC) alongside F1. This protocol is designed to expose failure modes that aggregate scores conceal, including temporal shifts and unseen attack families.
Explainability and Stability Analysis: The study applies SHapley Additive exPlanations (SHAP) to MDE-augmented classifiers. It quantifies the fold-stability and domain-coherence of entropy attributions, demonstrating that entropy features receive reproducible and theoretically grounded explanations across heterogeneous environments.

Results

Performance: Across four benchmarks (NSL-KDD, CICIDS-2017, CICIDS-2018, UNSW-NB15), entropy-only features achieve weighted F1 scores ranging from 0.708 to 0.989. When combined with conventional features, MDE does not degrade performance; in-distribution F1 scores remain statistically indistinguishable from conventional-only baselines.
Operational Failure Modes: The full metric reporting revealed significant gaps hidden by aggregate F1:
- On CICIDS-2018, an F1 of ~0.74 masked a Detection Rate (DR) of only 0.48, meaning fewer than half of attacks were detected.
- On CICIDS-2017 under temporal shift (training on Mon–Thu, testing on Fri), the DR collapsed to 0.082 despite an AUC of 0.87, indicating that while the model could rank attacks higher than benign traffic, fixed thresholds failed completely.
- In unseen-attack-family tests (Infiltration and Bot families held out), DR dropped to 0.000 for all models, with F1 remaining >0.998 solely due to the benign majority class.
Robustness and Transfer: MDE features showed graceful degradation under additive Gaussian noise. However, zero-shot cross-dataset transfer (e.g., training on CICIDS-2017, testing on CICIDS-2018) resulted in significant performance drops (F1 from 0.955 to 0.591), confirming that entropy signatures are attack-family specific and require domain adaptation.
Interpretability: SHAP analysis confirmed that entropy features (particularly JSD and ADE) are consistently among the top contributors to predictions. Fold-stability analysis showed high Spearman rank correlations (0.80–0.95) for all-feature attributions, confirming that the model relies on domain-coherent entropy signals.

Significance and Claims
The paper claims that MDE's primary contribution is not a dramatic increase in aggregate classification accuracy over conventional features, but rather a shift in representation and evaluation methodology.

Representation: MDE provides compact, theoretically grounded features that capture distributional structure (regularity, asymmetry, protocol diversity) which aggregate statistics discard. These features are natively interpretable via SHAP, addressing the "black box" issue of ensemble classifiers.
Evaluation: The study demonstrates that standard aggregate F1 scores are insufficient for operational security assessment. By exposing threshold-ranking divergence (where AUC remains high but DR collapses) and the failure to detect unseen attack families, the paper argues for evaluation protocols that prioritize full operational metrics and temporal/unseen-family testing.
Applicability: MDE offers a practical solution for environments where raw packet data is unavailable, enabling the integration of information-theoretic insights into standard flow-based IDS pipelines without the overhead of deep learning or raw data storage.

The authors conclude that while entropy features do not solve the fundamental challenge of generalizing to unseen attack taxonomies without retraining, they provide a robust, interpretable, and computationally efficient layer of analysis that complements conventional statistics and reveals operational risks often masked by standard metrics.