💻 computer science

When Unseen Attacks Look Normal: Open-Set Evaluation, Feature Observability, and Protocol-Invariant Detection in Mobile Ad Hoc Networks

This paper demonstrates that while standard machine learning models fail to detect unseen attacks in Mobile Ad Hoc Networks due to their reliance on variable distributions, a simple protocol-invariant feature—counting routing neighbors with no decoded frames—achieves perfect detection of wormhole attacks by identifying structural violations rather than statistical anomalies.

Original authors: Amruth V, Devaraj Verma C

Published 2026-09-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amruth V, Devaraj Verma C

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the chaotic world of mobile ad hoc networks, devices talk to one another without any central tower or fixed infrastructure to guide them. Imagine a group of hikers in a dense forest, each carrying a radio. To send a message across the group, one hiker must pass it to a neighbor, who passes it to another, until it reaches its destination. This system is incredibly useful for disaster relief or military operations where no cell towers exist, but it is also fragile. Because every device acts as both a sender and a relay, a single dishonest device can sabotage the entire conversation. It can pretend to be a helpful neighbor, steal the messages, or simply drop them in the dirt. For years, researchers have tried to build digital watchdogs to spot these traitors. They train computer programs to recognize specific types of bad behavior, like a device that suddenly stops forwarding messages or one that floods the network with fake requests. These programs are usually tested by seeing how well they can sort known bad actors from good ones. If the program can tell the difference between a thief and a hiker, it is considered successful.

However, a new study suggests that this way of testing is dangerously incomplete. The researchers, working with a detailed computer simulation of a hundred moving devices, discovered that a program might be excellent at spotting the five specific types of attacks it was trained on, yet be completely blind to a sixth type it has never seen before. In their experiments, they created a scenario with five different kinds of sabotage: a "black hole" that swallows all traffic, a "grey hole" that drops half the messages, a "sinkhole" that tricks devices into sending traffic to a dead end, a "flooding" attack that clogs the network with noise, and a "wormhole" that creates a secret tunnel between two distant devices. They trained seven different detection methods to recognize these five threats. When the researchers tested these methods only on the attacks they had already seen, the programs performed similarly well, with high accuracy scores that made them all look like winners. But the real test came when they hid one attack type from the training data and asked the programs to find it in a sea of normal traffic.

The results were startling. The most sophisticated programs, which used complex neural networks similar to those that power image recognition, failed spectacularly. When faced with unseen flooding, sinkhole, and wormhole attacks, these neural networks unanimously labeled the bad actors as normal, harmless devices. They were so sure of their wrong answers that their confidence scores were indistinguishable from their correct answers. In fact, for black hole and flooding attacks, these smart programs performed worse than random guessing. The only method that showed any real ability to spot the unknown threats was a much simpler approach based on a forest of decision trees. This method worked because, unlike the neural networks, it could sense when it was looking at something it didn't understand. When the data didn't fit its training, the trees in its "forest" disagreed with each other, creating a signal of uncertainty that the other programs lacked.

The study also uncovered a critical flaw in how such simulations are often built. One of the attacks, the wormhole, was initially detected with near-perfect accuracy, but the researchers realized this was an illusion. The simulation had given the detectors access to the exact physical coordinates of every device, allowing them to measure the true distance between neighbors. In the real world, a device cannot know its own exact location or the exact location of others; it can only estimate distance based on the strength of the radio signal. When the researchers removed this "shortcut" and forced the detectors to rely only on observable information, the performance for the wormhole attack collapsed. The detectors could no longer tell the difference between a normal device and a wormhole endpoint.

To solve this, the researchers found a different kind of clue that does not require knowing where devices are. They noticed that in a wormhole attack, two devices act as tunnel endpoints, appearing as neighbors in the network routing table even though they have never actually heard each other's radio signals. In a normal network, a device only lists a neighbor if it has successfully received a message from it. The researchers built a simple rule: if a device sees a neighbor in its list that it has never heard speak, that is a wormhole. This single observation, which relies on a fundamental rule of how the network is supposed to work, separated the wormhole attackers from all other devices with perfect accuracy in every test.

The paper concludes that the current way of evaluating security systems is misleading. A system that scores highly on known attacks may be useless against new ones. The study shows that the choice of how a system expresses its confidence matters more than the complexity of the model itself. A simple check for a broken rule in the network protocol proved more effective than a complex learning model for the wormhole attack, while a different type of model was needed to catch the flooding attack. The researchers argue that security testing must move beyond just checking if a system knows the old tricks. It must also test how the system reacts when it encounters a completely new kind of trouble, ensuring that the digital watchdogs can actually bark when they see something they have never seen before.

Technical Summary: When Unseen Attacks Look Normal

Problem Statement

Mobile Ad Hoc Networks (MANETs) lack fixed infrastructure and central authority, making routing protocols the primary attack surface. Current intrusion detection systems (IDS) for MANETs are predominantly evaluated as closed-set classification problems, where models are trained and tested on a fixed, known set of attack classes. This framing assumes that deployed detectors will only encounter attacks they were explicitly trained to recognize.

However, in real-world deployment, a model trained to separate five known classes must handle a sixth, unseen attack. The paper investigates what happens when a model encounters an attack it has never seen. Specifically, it questions whether standard closed-set metrics (like accuracy or F1 score) can reliably predict a model's behavior on unseen threats, and whether standard confidence signals (like maximum softmax probability) can distinguish between a known class and an unknown attack.

Methodology

The authors conducted a controlled evaluation using ns-3 simulations involving 120 runs with 100 nodes moving via random waypoint in a 1500x1500m field. The network utilized AODV routing over IEEE 802.11b.

Experimental Design:

Attacks: Five routing attacks were implemented: Black Hole, Grey Hole, Sinkhole, Flooding, and Wormhole.
Models: Seven detectors were evaluated on identical features and data splits: Decision Tree, k-NN, SVM, Random Forest (300 trees), CNN-1D, LSTM, and GRU. Additionally, a GRU-based Autoencoder was trained only on benign traffic.
Evaluation Protocol:
1. Closed-Set: Standard 4-fold cross-validation on all known attacks.
2. Open-Set (Leave-One-Attack-Out): Each attack was held out from training in turn. The model was then tested on the unseen attack to measure its ability to detect novelty.
3. Uncertainty Scoring: Three novelty scores were compared:
  - Maximum Softmax Probability (standard baseline).
  - Ensemble Vote Disagreement (for Random Forest).
  - Reconstruction Error (for Autoencoder).
Feature Observability: The study distinguished between features observable by a deployed node (protocol state, RSSI) and simulator-derived features (true geometric coordinates).

Key Contributions

Decoupling Closed-Set and Open-Set Performance: The study demonstrates that high closed-set performance does not correlate with open-set capability. While seven detectors achieved a narrow macro F1 range (0.878–0.944) on known attacks, their performance on unseen attacks varied wildly (AUROC 0.10 to 0.98).
Failure of Neural Confidence Signals: Neural architectures (CNN, LSTM, GRU) consistently failed to identify unseen attacks. They assigned unseen attacks to the "normal" class with high confidence (unanimous assignment in many cases), and their maximum softmax probabilities fell below chance (AUROC 0.10–0.25) for several attacks.
Ensemble Disagreement as a Novelty Signal: The Random Forest model was the exception. Its vote disagreement (the fraction of trees voting for the winning class) served as a robust uncertainty estimate, achieving AUROC scores of 0.982 on Black Hole, 0.934 on Grey Hole, and 0.965 on Flooding.
The "Wormhole" Trap and Protocol Invariants:
- Initial results for Wormhole detection (AUROC 0.950) relied on geometric distance features derived from true simulator coordinates, which are unavailable in deployment.
- Removing these simulator-only features dropped performance to chance (AUROC 0.571).
- The authors identified a deployable protocol invariant: the count of routing neighbors from which a node has never decoded a frame ( $\nu$ ). In honest traffic, this is always zero. In a wormhole tunnel, endpoints register neighbors they have never "heard."
- Thresholding this single observable statistic achieved AUROC 1.000 for Wormhole detection, outperforming all learned models.
Limitations of Reconstruction-Based Detection: The study explains why autoencoders failed on Wormhole (and Grey Hole) when using observable features. Reconstruction error detects deviations from a distribution; if a feature is constant in benign traffic (e.g., $\nu=0$ ), the model learns to reconstruct that constant perfectly. The anomaly is a violation of a hard constraint, not an unusual draw from a distribution, rendering reconstruction error ineffective without direct thresholding.

Results

Closed-Set: Seven detectors spanned a macro F1 range of only 0.066 (0.878 to 0.944). The Random Forest performed best, but the differences were not statistically significant across folds, suggesting closed-set metrics cannot reliably rank detectors for this problem.
Open-Set (Unseen Attacks):
- Neural Networks: Assigned unseen attacks to the "normal" class unanimously. For example, 19/19 Flooding nodes and 20/20 Sinkhole nodes were classified as normal.
- Random Forest: Successfully separated unseen Black Hole, Grey Hole, and Flooding attacks from normal traffic using vote disagreement.
- Autoencoder: Successfully detected Flooding (AUROC 0.996) and Sinkhole (AUROC 0.996) but failed on Grey Hole (AUROC 0.374) and Wormhole (AUROC 0.571 without geometry).
- Wormhole: No discriminative classifier achieved a usable score (best was LSTM at 0.637). Only the protocol invariant ( $\nu > 0$ ) achieved perfect separation.
Feature Observability: Removing simulator-derived geometric features caused the Wormhole detection score to drop from 0.950 to 0.571, highlighting the risk of overfitting to simulation artifacts.

Significance and Claims

The paper argues that the current evaluation paradigm for MANET intrusion detection is insufficient. The primary significance lies in three findings:

Closed-set accuracy is a poor proxy for open-set robustness. A model can be highly accurate on known attacks while completely blind to unseen ones.
The choice of uncertainty mechanism is more critical than the model architecture. A Random Forest's vote disagreement provided a usable novelty signal where neural network softmax probabilities failed catastrophically.
Protocol invariants can outperform learned models. For attacks that violate hard constraints (like the Wormhole tunnel), a simple check of a protocol invariant is superior to modeling the distribution of normal traffic.

The authors conclude that intrusion detection results for MANETs should be reported alongside leave-one-attack-out evaluations to reveal blind spots. Furthermore, the selection of a confidence or novelty score should be treated as a primary design decision, not an implementation detail, as it fundamentally determines whether a system can detect the "unseen."