IoTGeM: Generalizable Models for Behaviour-Based IoT Attack Detection

1. Problem Statement

The rapid proliferation of Internet of Things (IoT) devices has expanded the attack surface for cyber threats. While Machine Learning (ML) is widely used for behavior-based Intrusion Detection Systems (IDS), existing research suffers from critical methodological flaws that undermine the reliability and real-world applicability of their models:

• Data Leakage and Overfitting: Many studies use the same dataset for training, validation, and testing, or perform feature selection on the entire dataset before splitting. This leads to models that memorize dataset-specific artifacts (e.g., specific IP addresses, session timestamps, or packet size distributions) rather than learning generalizable attack patterns.
• Outdated or Non-IoT Data: Many studies rely on legacy datasets (e.g., KDD99, NSL-KDD) or non-IoT datasets (e.g., UNSW-NB15), which do not reflect the heterogeneity and resource constraints of modern IoT environments.
• Feature Limitations: Traditional approaches often rely on flow-based features (aggregated statistics over a complete connection) or individual packet features. Flow-based methods require waiting for a flow to terminate (causing latency), while individual packet features often include identifying information (like IPs) that prevents generalization across different networks.
• Evaluation Metrics: Over-reliance on "Accuracy" in imbalanced datasets often masks poor performance on minority attack classes.

2. Methodology: The IoTGeM Framework

The authors propose IoTGeM, a holistic framework designed to produce robust, generalizable models through strict data isolation and novel feature engineering.

A. Data Strategy and Isolation

To prevent data leakage, the authors employ a rigorous strict isolation methodology:

• Training/Cross-Validation (Train/CV): Used exclusively for model training.
• Hyperparameter Optimization (HPO) & Validation: Separate datasets used for feature selection and tuning. Crucially, the fitness function for feature selection is evaluated on a completely separate, unseen dataset (exogenous feedback).
• Testing: Models are evaluated on two distinct scenarios:
  1. Session Test: An independent session from the same dataset.
  2. Dataset Test: A completely different dataset (e.g., training on IoT-NID, testing on CIC-IoT-2022 or IoT-23).

B. Feature Extraction: Rolling and Expanding Windows

Instead of waiting for flows to end or analyzing single packets, IoTGeM uses a Window-Based Approach:

• Rolling Windows (RW): Analyzes a fixed number of recent packets (e.g., last 2, 6, or 9 packets) to capture immediate behavioral changes.
• Expanding Windows (EW): Starts with a small window and grows, recalculating statistics as new packets arrive.
• Feature Types: The method extracts multi-dimensional statistical features including packet size, payload entropy, TCP flag statistics, and time-based metrics (inter-arrival times) within these windows. This captures complex attack behaviors without relying on flow termination.

C. Multi-Step Feature Selection

A novel three-step process is used to eliminate non-generalizable features:

1. Voting Elimination: Features are tested across three scenarios (CV, Session, and External Validation). A feature must pass a Cohen's Kappa threshold (>0) in at least two scenarios, including the external validation, to proceed. This filters out features that only work on specific sessions.
2. Genetic Algorithm (GA) with Exogenous Feedback: A GA optimizes the feature subset. Unlike traditional GAs that evaluate fitness on the training set, this GA evaluates the fitness (F1 score) of each feature combination on a separate, unseen dataset. This forces the algorithm to select features that are inherently generalizable.
3. SHAP Analysis: SHapley Additive exPlanations (SHAP) is used post-training to interpret models, ensuring that selected features correlate with actual attack mechanics rather than spurious correlations.

D. Threat Model

The system is designed as a Network-based IDS (NIDS) deployed at a network chokepoint (gateway/router). It assumes an external attacker attempting to compromise IoT devices via flooding (DoS), scanning, brute-force, and reconnaissance attacks.

3. Key Contributions

1. Novel Feature Extraction: Introduction of a window-based feature extraction method that generates rich, multi-dimensional statistical features (size, entropy, flags) from raw packet streams, avoiding the latency of flow-based methods and the overfitting risks of packet-based methods.
2. Exogenous Feedback Feature Selection: A unique GA-based selection process where the "fitness" of a feature set is determined by its performance on an unseen, independent dataset. This is a significant departure from standard cross-validation methods and directly targets generalizability.
3. Strict Evaluation Protocol: The use of completely isolated datasets for training, tuning, and testing, including cross-dataset evaluation, to provide a realistic measure of model robustness.
4. Comprehensive Benchmarking: Evaluation against 8 different ML algorithms (LR, DT, NB, SVM, RF, KNN, MLP, XGB) across 10 distinct attack types using both window-based and traditional flow-based features for direct comparison.

4. Results

The models were evaluated on 10 attack types (ACK Flood, HTTP Flood, SYN Flood, UDP Flood, ARP Spoofing, Brute Force, Port Scan, Host Discovery variants, and OS Detection).

• Superior Generalization: In cross-dataset tests (the most stringent scenario), IoTGeM's window-based models significantly outperformed traditional flow-based models.
  • F1 Scores: Achieved 99% for ACK, HTTP, SYN, MHD, and PS attacks, and 94% for UDP attacks.
  • Flow-Based Failure: Traditional flow-based models often failed completely on cross-dataset tests (e.g., F1 scores dropping to near 0 for SYN and UDP floods), indicating severe overfitting to network-specific statistics.
• Feature Analysis (SHAP):
  • Window Approach: Relied on features related to the core mechanics of attacks, such as port diversity, payload entropy, and TCP flag anomalies.
  • Flow Approach: Relied heavily on Inter-Arrival Times (IAT) and flow duration. The authors argue these are highly sensitive to network dynamics and specific to the training environment, leading to poor generalization.
• Exceptions: The flow-based approach performed slightly better for ARP Spoofing (ARPS) because the attack relies on subtle time fluctuations between packets, which flow-based tools capture well. However, for all other attacks, the window-based approach was superior.

5. Significance and Conclusion

The paper demonstrates that generalizability in IoT attack detection is achievable but requires a fundamental shift in methodology:

• Avoiding Overfitting: By using exogenous feedback for feature selection and strictly isolating datasets, IoTGeM prevents models from learning "dataset artifacts" (like specific IP ranges or timing patterns unique to a simulation).
• Real-World Applicability: The window-based approach allows for earlier detection (no need to wait for flow termination) and produces models that maintain high performance when deployed in new, unseen network environments.
• Reproducibility: The authors provide public access to their code and scripts, addressing the reproducibility crisis in ML-based security research.

In conclusion, IoTGeM establishes that while deep learning and complex architectures are popular, a carefully engineered feature extraction pipeline combined with rigorous, leakage-free validation is the key to building truly robust and generalizable IoT intrusion detection systems.