Individual Packet Features are a Risk to Model Generalisation in ML-Based Intrusion Detection

1. Problem Statement

The paper addresses a critical flaw in current Machine Learning (ML)-based Intrusion Detection Systems (IDS) for the Internet of Things (IoT). While many studies report high detection accuracy using Individual Packet Features (IPF)—attributes extracted from single packets such as size, timing, source/destination IP, and port numbers—the authors argue these models suffer from severe generalization issues.

The core problem is that IPF-based models often rely on information leakage and low data complexity rather than learning genuine attack patterns. Consequently, these models achieve artificially high performance metrics on specific datasets but fail completely when deployed in diverse, real-world environments or on different datasets.

2. Methodology

The authors employed a mixed-method approach combining a systematic literature review with extensive experimental validation:

• Literature Review: They analyzed 68 ML-based IDS studies (2019–2023), focusing on the 10 studies that utilized IPFs. They examined the datasets used (e.g., Edge-IIoTset, MQTTset, CICIDS2017) and identified common preprocessing flaws, such as the inclusion of identifying features (IPs, MACs) and the lack of session-based data splitting.
• Experimental Design:
  • Dataset: They utilized the IoT-NID dataset, which is unique because it records attacks as distinct sessions, allowing for rigorous separation of training and testing data.
  • Information Leakage Simulation: They compared two evaluation scenarios:
    1. Cross-Validation (CV): Random splitting of packets, which often mixes packets from the same session into both training and test sets.
    2. Isolated Sessions: Training on one session and testing on a completely different session to simulate real-world generalization.
  • Complexity Analysis: They calculated "complexity scores" (based on 22 analytical methods) for attack data regarding packet size and time to determine if low complexity was driving high accuracy.
  • Feature Isolation: They tested models using single features (e.g., only packet size or only timestamp) to demonstrate how simple attributes can act as proxies for attack labels in low-complexity datasets.

3. Key Contributions

The paper provides the first empirical evidence demonstrating that IPF-based approaches are fundamentally flawed for robust intrusion detection. Key contributions include:

• Identification of Information Leakage: The authors demonstrate that many high-performing studies inadvertently use identifying features (e.g., Source IP, Destination Port, Sequence Numbers) that remain constant within a specific attack session. When training and test sets are split randomly (rather than by session), these features allow the model to "memorize" the attacker's identity rather than learning the attack behavior.
• The "Low Complexity" Trap: The study reveals that many attacks (e.g., UDP floods, Mirai botnets) produce highly uniform packet sizes or temporal patterns. In datasets with low complexity, even basic features (like packet size) become perfect identifiers for the specific dataset, leading to overfitting.
• Session-Based Generalization Failure: The authors prove that models trained on IPFs fail to generalize when the data is split by session. A model trained on Session A often performs near-randomly on Session B, even if both contain the same type of attack, because the specific identifying features or uniform patterns differ.
• Reproducibility: The authors made their scripts and code publicly available to facilitate reproducibility of these findings.

4. Key Results

• Discrepancy in Performance:
  • Cross-Validation (Leaky): Models achieved near-perfect accuracy (e.g., 99–100% F1 scores) when using CV on datasets like IoT-NID.
  • Isolated Sessions (Realistic): When tested on unseen sessions, performance collapsed. For example, a model trained on Session 2 of a UDP attack achieved 100% accuracy on Session 4 (similar characteristics) but dropped to ~3% accuracy on Session 3 (different characteristics) when using only packet size as a feature.
• Feature Analysis:
  • Features like dport (destination port) and IP_id often acted as perfect classifiers in CV scenarios because they were constant within a session but varied between sessions.
  • Decision trees trained on these features were found to be extremely shallow and simple, indicating they were learning shortcuts (identifiers) rather than complex behavioral patterns.
• Complexity Correlation: There was a strong inverse correlation between data complexity and model success. Attacks with low complexity (e.g., uniform packet sizes) yielded perfect detection in CV but failed in isolation. As complexity increased, the reliance on simple IPFs diminished, and detection rates dropped significantly.

5. Significance and Implications

• Rethinking Feature Engineering: The paper argues that relying solely on IPF is insufficient for IoT security. Researchers must move toward Flow-based (aggregated statistics) or Window-based (temporal sequences) features that capture packet interactions and context.
• Methodological Rigor: The study highlights a critical need for session-based data splitting in IDS research. Random packet-level splitting is insufficient and leads to misleadingly optimistic results. Future evaluations must ensure that entire sessions (flows) are kept entirely within either the training or testing set.
• Security Resilience: By exposing the fragility of current IPF-based models, the paper urges the community to develop more robust detection systems that can adapt to new environments and evolving attack vectors, rather than simply memorizing dataset-specific identifiers.

Conclusion: The paper concludes that while IPF-based models may appear effective in controlled, low-complexity experiments, they are unreliable for real-world deployment due to information leakage and an inability to generalize. Robust IoT security requires holistic approaches that consider the context and interaction of network packets.