Technical Summary: Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks

Problem Statement

Lightweight machine learning models are increasingly proposed for Intrusion Detection Systems (IDS) in Industrial Internet of Things (IIoT) networks due to their suitability for resource-constrained edge deployment. While existing literature reports near-perfect detection accuracy for these models, such evaluations are almost exclusively conducted within the same dataset used for training (in-domain). This creates a critical gap: it remains unverified whether these models retain performance when deployed on unseen industrial networks with different device compositions, traffic patterns, and attacker behaviors. Furthermore, prior work often relies on artificially balanced class distributions and fails to test specific mechanisms of failure, such as reliance on dataset-specific shortcuts (e.g., raw port numbers) or the capacity for recovery through limited adaptation.

Methodology

This study conducts a systematic cross-dataset evaluation to address these gaps.

Datasets and Feature Engineering:
The authors utilized three distinct IIoT/IoT datasets:

1. Edge-IIoTset: Used as the source (training) domain.
2. Gotham 2025: Used as the first target domain (packet-level capture).
3. WUSTL-IIoT-2021: Used as the second target domain (flow-level Argus export).

To ensure comparability across these structurally distinct sources, a minimal common schema was constructed using only features available in all three: source/destination ports, transport protocol, and four TCP flag bits. Crucially, raw port numbers were not used. Instead, ports were mapped to four coarse categorical buckets (none, well-known, registered, dynamic) and one-hot encoded. This design explicitly aimed to prevent models from learning "raw port memorization," a known shortcut in intrusion detection.

Models:
Four representative lightweight architectures were trained on the Edge-IIoTset source domain and evaluated without retraining on the two target domains:

1. DecisionTree: A depth-limited tree (max_depth = 10).
2. SmallMLP: A three-layer feed-forward neural network.
3. Small1DCNN: A compact 1D convolutional neural network.
4. SmallLSTM: A single-layer LSTM treating the feature vector as a sequence.

Evaluation Protocols:

• Cross-Domain Transfer: Models were evaluated on target datasets in a zero-shot setting (no retraining).
• Class Distribution: Evaluations were performed under both artificially balanced class distributions (common in prior work) and the natural, heavily benign-skewed distributions found in real deployments.
• Explainability: SHAP values and native feature importances were analyzed to identify reliance on specific features.
• Adversarial Robustness: The HopSkipJump black-box attack was applied to assess vulnerability to evasion.
• Few-Shot Recovery: Models were fine-tuned on small fractions (0% to 25%) of labeled target-domain data to assess the capacity for performance recovery.

Key Results

1. Collapse of Cross-Domain Performance
All four architectures achieved high in-domain F1 scores (~0.97) on Edge-IIoTset. However, when applied to the target datasets without retraining, performance collapsed drastically.

• On Gotham, F1 scores dropped to a range of 0.18–0.28 under natural class distributions.
• On WUSTL-IIoT-2021, F1 scores dropped further to 0.09–0.13.
• No model retained more than 29% of its in-domain F1 score when applied outside its training network.

2. The Port-Bucket Shortcut Persists
Explainability analysis revealed that despite the removal of raw port numbers, models still relied overwhelmingly on the coarse port-bucket categories.

• For the DecisionTree, the top feature (dst_port_none) accounted for a significant portion of importance.
• Crucial Finding: The prevalence of this specific port bucket in source-domain attack traffic was 96 times higher than in Gotham's attack traffic and 435 times higher than in WUSTL-IIoT-2021's.
• This confirms that coarsening feature resolution relocates the shortcut reliance rather than removing it. The models learned to associate specific port buckets with attacks in the source domain, a correlation that did not hold in the target domains.

3. Impact of Class Distribution on Evaluation
The choice of evaluation protocol significantly altered the perceived difficulty of the target networks.

• Balanced vs. Natural: Under balanced evaluation, models appeared to perform better (e.g., F1 ~0.53 on WUSTL-IIoT-2021). Under natural distribution, performance collapsed (F1 ~0.13) due to a massive drop in precision caused by false positives accumulating against the large pool of benign traffic.
• Reversal of Difficulty: Balanced evaluation suggested WUSTL-IIoT-2021 was easier to generalize to than Gotham. Natural evaluation reversed this, showing WUSTL-IIoT-2021 as significantly harder. This indicates that comparative claims about generalization challenges are protocol-dependent.

4. Architecture-Dependent Adaptability
The ability to recover performance through limited fine-tuning varied by architecture:

• DecisionTree: Showed no improvement until 25% of the target data was used, after which F1 nearly quadrupled.
• SmallLSTM: Improved immediately with small data (5%) but performance declined as more data was added.
• SmallMLP: Showed modest, gradual improvement.
• Small1DCNN: Failed to improve at any data fraction.
  This demonstrates that adaptability is a property of the specific architecture, not a general trait of lightweight models.

5. Independence of Properties
Cross-domain generalization, adversarial robustness, and computational efficiency were found to be independent axes. For instance, SmallLSTM was among the best at cross-domain transfer on Gotham but the least robust to adversarial attacks, while DecisionTree was robust and adaptable but required more data to recover.

Significance and Claims

The paper argues that the current standard for evaluating lightweight IIoT intrusion detection—relying on high in-domain accuracy under balanced, single-dataset conditions—is insufficient for assessing deployment readiness.

• Generalization Failure is Systemic: Strong in-domain performance does not imply cross-network capability. The failure is driven by reliance on feature distributions (specifically port buckets) that do not transfer, even when raw identifiers are removed.
• Evaluation Bias: Artificially balanced evaluation protocols substantially overstate real-world performance and can misleadingly reverse the ranking of target network difficulty.
• Adaptability is Not Universal: The capacity to recover performance via few-shot learning is architecture-dependent. Some models (like DecisionTree) can recover with sufficient local data, while others (like Small1DCNN) cannot.
• Recommendation: The authors conclude that cross-network evaluation under realistic, imbalanced class distributions should be treated as a standard requirement for lightweight IIoT IDS research, rather than an optional extension. They caution that deployment readiness cannot be inferred from within-domain accuracy alone.