💻 computer science

Detection of Anomalous Network Nodes via Hierarchical Prediction and Extreme Value Theory

This paper proposes a two-stage method combining hierarchical time series prediction and Extreme Value Theory to detect anomalous network nodes based on unusual ARP call behavior, demonstrating significantly reduced false positives on a real-world dataset.

Original authors: Sevvandi Kandanaarachchi, Mahdi Abolghasemi, Hideya Ochiai, Asha Rao, Conrad Sanderson

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sevvandi Kandanaarachchi, Mahdi Abolghasemi, Hideya Ochiai, Asha Rao, Conrad Sanderson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy office building with hundreds of employees (the network nodes). Every day, they send out little notes to each other asking, "Hey, who has this specific email address? What's their desk number?" This is like the ARP calls in a computer network—a constant chatter of devices asking, "Who are you, and where are you?"

Usually, everyone follows a predictable routine. The accountant sends a few notes in the morning, the designer sends a burst at noon, and the server sends a steady stream all day.

Now, imagine a spy (the malware) sneaks into the building. Instead of working quietly, the spy starts running around frantically, shouting out to every desk in the building, "Who are you? What's your number? Give me your keys!" This chaotic behavior is a huge red flag, but it's hard to spot because the building is so big and noisy.

This paper presents a clever two-step system to catch that spy without raising a false alarm every time someone just drops a pen.

Step 1: The "Team Huddle" Prediction (Hierarchical Prediction)

Most security systems look at each employee in isolation. They might say, "John usually sends 5 notes an hour. Today he sent 6. Is that weird?" This is like trying to predict the weather by looking at a single tree.

The authors propose a smarter approach: The Team Huddle.

Instead of looking at one person, they look at the whole building's rhythm and each person's rhythm together.

The Big Picture: They know the total number of notes sent in the building usually follows a pattern (e.g., busy in the morning, quiet at lunch).
The Individual: They know how much John, Sarah, and the Server usually send.

By combining these two views, the system creates a "Team Huddle" prediction. It asks: "Given that the whole building is usually quiet at 2 PM, and John usually sends 2 notes, does it make sense if John suddenly sends 50?"

This method uses a mathematical technique called Minimum Trace to make sure the predictions for the individuals add up perfectly to the prediction for the whole building. It's like a coach adjusting the players' stats so the team total makes sense, ensuring no one gets blamed for a mistake that was actually just a misunderstanding of the team's overall flow.

Step 2: The "Extreme Weather" Alarm (Extreme Value Theory)

Once the system predicts what a node should be doing, it looks at the residuals (the difference between what happened and what was predicted).

The Old Way (The Threshold Trap): Imagine a security guard who says, "If anyone sends more than 10 notes, sound the alarm!" The problem is, sometimes a legitimate employee just has a really busy day (a "heavy tail" in the data). If you set the bar too low, you get a million false alarms. This leads to Alert Fatigue, where the security team gets so tired of false alarms that they ignore the real ones.
The New Way (Extreme Value Theory - EVT): Instead of a simple "if/then" rule, the system uses Extreme Value Theory. Think of this as a meteorologist studying "1-in-100-year storms."

EVT understands that internet traffic is like weather: it's mostly calm, but sometimes there are huge, rare spikes that don't fit a normal bell curve.

If a node sends 12 notes, that's just a "sunny day with a few clouds."
If a node sends 10,000 notes, that's a "hurricane."

EVT is specifically designed to model these rare, extreme events. It asks: "Is this deviation so extreme that it belongs to the category of 'once-in-a-lifetime' events, or is it just a normal busy day?" By focusing on the shape of the extreme data, it can distinguish between a legitimate busy day and a malicious attack with much higher accuracy.

The Results: Catching the Spy Without the Noise

The researchers tested this on a real network with 362 nodes and over 10 million "notes" (ARP calls).

The Competition: They compared their method against a popular "Autoencoder" (a type of AI that tries to memorize patterns). The Autoencoder was like a paranoid guard who screamed "INTRUDER!" at almost everything. It caught the bad guys (high Recall) but also accused innocent employees constantly (low Precision, high False Positives).
The Winner: Their new method, especially when using a tool called LightGBM (a smart decision-making algorithm), was the star.
- It caught the bad guys almost as well as the paranoid guard.
- Crucially, it reduced the false alarms by a massive margin. It was like having a guard who only screams when a hurricane is actually hitting the building, not when it's just raining.

In a Nutshell

This paper solves the problem of "Alert Fatigue" in cybersecurity.

Don't look at individuals in a vacuum: Use the behavior of the whole network to predict what a single device should be doing.
Don't use simple rules: Use advanced math (EVT) to understand that "busy" looks different for different devices, and only flag the truly impossible, extreme outliers.

The result is a security system that is smart enough to know the difference between a busy employee and a spy, keeping the security team focused on the real threats.

1. Problem Statement

Industrial networks face evolving cyber-attacks that often bypass traditional signature-based detection methods (e.g., firewalls at gateways or antivirus on individual nodes). Once malware infiltrates a network via an unsecured device, it can spread to other nodes, often exhibiting unusual behavior in Address Resolution Protocol (ARP) calls.

The core challenges addressed in this work are:

Invisibility of Internal Threats: Conventional Intrusion Detection Systems (IDS) often monitor incoming/outgoing traffic (TCP/UDP) but miss internal LAN traffic like ARP calls, where lateral movement and spoofing occur.
Alert Fatigue: High rates of false positives in anomaly detection systems lead to "alert fatigue," causing security professionals to ignore critical incidents.
Data Distribution Complexity: Network traffic often follows heavy-tailed distributions (power laws) rather than normal (Gaussian) distributions. Standard thresholding methods fail here, misclassifying legitimate heavy-tail events as anomalies.

2. Methodology

The authors propose a two-stage, unsupervised approach to detect anomalous nodes by modeling and predicting ARP call behavior.

Stage A: Hierarchical Time Series Prediction

Instead of modeling each node in isolation, the method uses Hierarchical Time Series to leverage information across the entire network.

Structure: A two-level hierarchy is defined:
- Bottom Level: Individual network nodes (362 nodes).
- Top Level: The aggregate sum of all ARP calls in the network.
Reconciliation: The system generates base predictions for individual nodes and then reconciles them with the top-level aggregate using the Minimum Trace (MinT) technique. This adjusts individual predictions to ensure they align with the global network pattern, extracting latent information that improves accuracy.
Prediction Models: Four distinct time-series modeling algorithms are evaluated:
1. ETS (Error, Trend, Seasonal): State-space exponential smoothing.
2. TSLM (Time Series Linear Regression): Log-linear regression using trends and lagged observations.
3. ZINB (Zero-Inflated Negative Binomial): Specifically handles the high frequency of zero values and large spikes in the data.
4. LightGBM: A gradient boosting framework optimized via grid search.

Stage B: Anomaly Detection via Extreme Value Theory (EVT)

Once predictions are made, the system calculates residuals (the difference between actual and predicted ARP calls).

The Challenge: Simple thresholding of residuals leads to high false positives because network traffic residuals often follow heavy-tailed distributions.
The Solution: The authors apply Extreme Value Theory (EVT).
- Residuals are modeled using a Generalized Pareto Distribution (GPD).
- The GPD accommodates various tail behaviors (Gumbel, Weibull, or Fréchet) without assuming a fixed distribution.
- A threshold is set on the probability of observing a residual value. If the probability of a residual exceeding a certain value is below a user-configured alpha (set to $\alpha=0.05$ ), the node is flagged as anomalous.

3. Key Contributions

Novel Combination: This is the first known application combining Hierarchical Prediction with Extreme Value Theory specifically for computer network security.
Robustness to Heavy Tails: By utilizing EVT, the method explicitly handles the heavy-tailed nature of network traffic, significantly reducing false positives compared to standard statistical thresholds.
Hierarchical Advantage: The approach demonstrates that modeling nodes collectively (via reconciliation) yields better predictions than modeling nodes in isolation, accounting for global network patterns.
Real-World Validation: The method was tested on a massive, real-world dataset rather than synthetic data.

4. Experimental Results

Dataset: A real-life dataset containing 10.3 million ARP calls from 362 nodes over approximately one year. The dataset includes a ground truth of ~1% anomalies (nodes interacting with a monitoring device in a way that indicates an attack).
Evaluation Metrics: Precision, Recall, F-measure, and False Positives per window.
Comparisons: The proposed variants were compared against a popular Autoencoder baseline.

Key Findings:

Best Performance: The LightGBM + EVT variant achieved the highest overall performance with an F-measure of 0.341 and the lowest False Positives (1.0 per window).
Precision vs. Recall: While the Autoencoder achieved the highest recall (0.890), it suffered from extremely low precision (0.018) and a massive number of false positives (~59 per window).
False Positive Reduction: The proposed methods drastically reduced false positives. For instance, ZINB + EVT had only 3.34 false positives, and LightGBM + EVT had 1.0, compared to the Autoencoder's 59.0.
Conclusion: The LightGBM variant provided the best balance, significantly outperforming the autoencoder in precision and false positive rates while maintaining competitive recall.

5. Significance and Impact

Mitigating Alert Fatigue: By drastically reducing false positives, the system addresses a critical pain point for security operations centers (SOCs), ensuring that alerts are more likely to be genuine threats.
Internal Threat Detection: The focus on ARP calls allows for the detection of lateral movement and malware propagation that traditional perimeter defenses miss.
Adaptability: The use of EVT allows the system to adapt to the complex, non-Gaussian statistical properties of real-world internet traffic, making it more robust than methods relying on normal distribution assumptions.
Future Work: The authors suggest extending the hierarchy to three levels (grouping similar devices) and exploring alternative reconciliation methods to further refine the model.

In summary, the paper presents a highly effective, data-driven framework for detecting compromised network nodes by combining the structural insights of hierarchical time series with the statistical rigor of Extreme Value Theory, resulting in a practical solution that minimizes false alarms in industrial network security.