Assessing Generalisation Capability of Machine Learning Models for Intrusion Detection

Technical Summary: Assessing Generalisation Capability of Machine Learning Models for Intrusion Detection

Problem Statement
The rapid expansion of networked and IoT systems has intensified cybersecurity threats, exposing the limitations of traditional signature-based intrusion detection systems (IDS). While machine learning (ML) models frequently report high accuracy on specific benchmark datasets, this study identifies a critical gap: high performance within a single dataset does not guarantee reliability in unseen network environments. Existing evaluations often fail to account for dataset diversity, traffic generation differences, and feature distribution shifts, leading to models that may not generalize to real-world scenarios.

Methodology
The study evaluates the generalisation capability of supervised ML models using two publicly available intrusion detection datasets: UNSW-NB15 (general network traffic) and TON_IoT (IoT/IIoT traffic).

• Models: Three supervised classifiers were selected to represent different learning paradigms: Random Forest (ensemble), Logistic Regression (linear), and Naive Bayes (probabilistic).
• Experimental Settings:
  1. Same-Dataset Evaluation: Models were trained and tested on the same dataset using a 70/30 train-test split.
  2. Cross-Dataset Evaluation: Models trained on one dataset were tested on the other to assess generalisation across different network environments.
• Feature Engineering: For cross-dataset experiments, only the seven common features shared by both datasets were retained: duration, proto, service, src_bytes, dst_bytes, src_pkts, and dst_pkts. This was necessary due to structural differences between the datasets.
• Preprocessing: Data underwent cleaning (removing duplicates, handling infinite/missing values), normalization (Min-Max), and label encoding. Feature correlation analysis was conducted to identify redundancy.

Key Results

• Same-Dataset Performance: All models achieved strong performance when trained and tested on the same data.
  • Random Forest emerged as the top performer, achieving 95.08% accuracy on UNSW-NB15 and 99.79% accuracy on TON_IoT.
  • Logistic Regression and Naive Bayes also showed competitive results, particularly in recall, though with lower overall accuracy than Random Forest.
• Cross-Dataset Performance: A significant "generalisation gap" was observed. When models were trained on one dataset and tested on the other, performance dropped precipitously.
  • Random Forest: Dropped to 38.07% accuracy (UNSW-NB15 $\to$ TON_IoT) and 39.54% accuracy (TON_IoT $\to$ UNSW-NB15).
  • Logistic Regression: Performed relatively better in the UNSW-NB15 $\to$ TON_IoT direction (77.79% accuracy) but fell to 38.24% in the reverse direction.
  • Naive Bayes: Showed the weakest generalisation, with accuracies as low as 25.85% and 36.58%, and extremely low recall in some configurations.
• Feature Analysis: Correlation heatmaps revealed redundant traffic-volume features. Feature importance analysis on the TON_IoT dataset indicated that source-side byte/packet volumes and DNS-related features were critical for the Random Forest model, suggesting the model learned dataset-specific patterns rather than universal attack signatures.

Key Contributions

1. Empirical Evidence of Generalisation Failure: The study provides concrete evidence that high benchmark accuracy on individual datasets (UNSW-NB15, TON_IoT) does not translate to cross-dataset viability.
2. Cross-Dataset Benchmarking: By systematically evaluating models across two distinct datasets using a restricted common feature set, the paper highlights the fragility of current IDS models when faced with domain shifts.
3. Comparative Analysis: The results are compared against existing literature, showing that while the study's same-dataset results are competitive or superior to prior works, the cross-dataset collapse is a distinct finding often overlooked in standard evaluations.

Significance and Claims
The paper claims that the primary contribution is not merely a comparison of classifiers, but the exposure of a notable generalisation problem in cybersecurity machine learning. The authors argue that the sharp decline in performance during cross-dataset evaluation proves that high benchmark scores often mask a failure to adapt to diverse network traffic.

The study frames this challenge within the context of affective computing and human-centric AI. It draws a parallel between the difficulty of inferring latent states from noisy, context-dependent behavioral signals in affective computing and the challenge of identifying malicious activity from shifting network behaviors. The authors suggest that techniques from these fields—such as behavioral signal analysis, domain adaptation, and uncertainty-aware modelling—could be leveraged to develop more robust, adaptive, and generalisable intrusion detection systems capable of operating across evolving IoT and enterprise environments.

Limitations
The authors acknowledge that the poor cross-dataset performance may be partly attributed to the limited number of common features (only seven) used for evaluation, which removed dataset-specific information available in same-dataset tests. Additionally, differences in traffic generation environments and attack behaviors between the two datasets contributed to the domain shift.