💻 computer science

ML Defender (aRGus NDR): An Open-Source Embedded ML NIDS for Botnet and Anomalous Traffic Detection in Resource-Constrained Organizations

This paper introduces ML Defender (aRGus NDR), an open-source, cost-effective embedded ML network intrusion detection system built on eBPF/XDP and a hybrid rule-based/Random Forest architecture that achieves near-perfect accuracy (F1=0.9985) and drastically reduces false positives on commodity hardware, making advanced botnet and anomaly detection accessible to resource-constrained organizations.

Original authors: Alonso Isidoro Román

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alonso Isidoro Román

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your home network is like the circulatory system of a small town. The data packets flowing through it are the blood, keeping everything alive and working. Usually, this blood is healthy. But sometimes, a virus (like a botnet or ransomware) tries to sneak in, clog the arteries, or poison the whole town.

For years, the only way to stop these viruses was to hire a giant, expensive security firm with a massive army of guards (enterprise security). If you were a small hospital, a school, or a local business, you couldn't afford them. You were left vulnerable.

ML Defender (aRGus NDR) is the open-source, low-cost solution designed to give those small towns their own immune system. It's a piece of software that runs on a cheap computer (about the cost of a nice gaming console, $150–$ 200) and acts as a real-time bodyguard.

Here is how it works, broken down into simple concepts:

1. The Two-Step Security Check

Imagine a bouncer at a club who has two ways of deciding who to let in:

The Fast Detector (The Rulebook): This bouncer has a list of obvious "bad guys." If someone is wearing a mask and carrying a gun, he stops them immediately. It's fast, but sometimes he gets too jumpy and stops a regular guy just because he's wearing a hat (this is called a "False Positive").
The ML Detector (The Smart Detective): This bouncer is a brilliant detective who has studied thousands of criminal patterns. He doesn't just look at the mask; he looks at how the person walks, how they talk, and their history. He is very good at telling the difference between a criminal and a regular guy.

The Magic Trick: ML Defender uses both. If the Rulebook says "Stop!" but the Detective says "It's fine," the system listens to the Detective. This stops the system from panicking and blocking innocent traffic, while still catching the real bad guys.

2. How It Sees the Invisible

Modern hackers often encrypt their messages (putting them in a locked box). Traditional security tools can't see inside the box, so they miss the attack.

ML Defender is different. It doesn't try to open the box. Instead, it watches how the box is being carried.

Is the person running? (High speed = bad).
Are they knocking on every single door in the neighborhood? (Port scanning = bad).
Are they trying to talk to a server they've never visited before? (Suspicious = bad).

By analyzing these "behavioral patterns" rather than the content of the message, it can catch ransomware and botnets even if the messages are locked tight.

3. The "Council of Wise Men" (How it was built)

Usually, building a system this complex requires a team of 20 experts. The author of this paper was working alone. To solve this, he invented a new way of working called the "Consejo de Sabios" (Council of Wise Men).

He didn't just ask one AI for help; he asked seven different AIs (like Claude, ChatGPT, Gemini, etc.) to act as a board of directors.

He proposed an idea.
The AIs argued against it, found holes in the logic, and suggested fixes.
The human author acted as the final judge, picking the best advice.

It's like having a roundtable of seven brilliant professors arguing over your homework, but you get to keep the best parts of their arguments.

4. The Results: A Miracle for Small Towns

The researchers tested this system on a famous dataset of computer viruses (the CTU-13 dataset).

Accuracy: It caught 100% of the bad traffic.
Mistakes: It made almost no mistakes. Out of 12,000+ normal data flows, it only flagged two as suspicious. (And those two were actually just weird glitches caused by the testing software itself, not real viruses).
Speed: It is incredibly fast. It makes a decision in less than a microsecond (a millionth of a second). It's faster than you can blink.
Cost: It runs on a $150 computer. No expensive servers, no cloud fees, no monthly subscriptions.

5. Why This Matters

The paper was written because the author saw a hospital in Barcelona get paralyzed by ransomware, and he remembered a friend whose small business was destroyed by the same thing. He realized that the "rich" get the best security, and the "poor" get nothing.

ML Defender is the equalizer. It proves that you don't need a billion-dollar budget to have a world-class immune system for your network. You just need a cheap computer, some open-source code, and a smart way of thinking.

In short: It's a tiny, super-fast, free digital immune system that learns to spot bad guys by how they act, not just what they look like, making the world safer for the small organizations that usually get left behind.

1. Problem Statement

Small and medium-sized organizations (SMEs), schools, and hospitals often lack the budget for enterprise-grade Network Detection and Response (NDR) solutions, leaving them vulnerable to ransomware and DDoS attacks.

The Gap: Existing open-source alternatives (e.g., Snort, Suricata) rely heavily on signature matching, which fails against novel attack variants. Conversely, machine learning (ML) based solutions in research often require GPU infrastructure, cloud offloading, or dedicated security teams, making them impractical for resource-constrained environments.
The Goal: To create a real-time, embedded ML-based NIDS that runs on commodity hardware (approx. $150–200 USD), requires no external dependencies, and provides automated blocking capabilities without sacrificing detection accuracy.

2. Methodology and Architecture

The system, named ML Defender (aRGus NDR), is built in C++20 and utilizes a six-component pipeline designed for low latency and high throughput.

Core Pipeline Components

eBPF/XDP Sniffer: Captures packets at the XDP hook (before the kernel networking stack) using zero-copy mechanisms to minimize overhead.
Ring Consumer: Reconstructs bidirectional flow states and extracts 28 behavioral features (e.g., connection frequency, port diversity, TCP flag distributions) per flow. It handles missing features using a sentinel value (-9999.0f) to ensure deterministic classification.
Fast Detector (Heuristic Layer): A rule-based engine using four heuristics (External IP velocity, SMB connection count, Port diversity, TCP RST ratio) within a 10-second sliding window. It acts as a high-speed filter but has a higher False Positive Rate (FPR) on benign traffic (~6.61%).
ML Detector (Embedded Classifier): An embedded Random Forest classifier implemented in pure C++20 (transpiled from scikit-learn). It classifies traffic into four threat categories: Ransomware, DDoS, Traffic anomalies, and Internal threats (using ONNX Runtime for the latter).
RAG Subsystem: Uses FAISS and SQLite with a local TinyLlama instance to provide semantic query capabilities over live event streams, ensuring no sensitive metadata is sent to the cloud.
Firewall ACL Agent: Automatically blocks attacker IPs via ipset/iptables upon confirmed detection.

Detection Policy: "Maximum Threat Wins"

The system employs a Dual-Score Detection architecture:
$Score_{final} = \max(Score_{fast}, Score_{ml})$

Both detectors output continuous confidence scores [0, 1].
The ML Detector's primary role is False Positive suppression. While the Fast Detector flags many benign flows, the ML layer refines these decisions, reducing real production blocks to near zero.
The system uses a "fail-closed" approach: if the maximum score exceeds a threshold, the traffic is blocked.

Cryptography and Transport

Transport: ZeroMQ for asynchronous messaging; Protocol Buffers for serialization.
Encryption: ChaCha20-Poly1305 (selected for performance on hardware without AES-NI, including ARMv8).
Integrity: HMAC-SHA256 for log verification.
Key Management: Currently uses etcd for seed distribution (single-node), with plans for peer-to-peer key negotiation.

3. Key Contributions

A. Technical System (ML Defender)

Embedded Inference: Achieves sub-microsecond inference latency (0.24 µs to 1.06 µs) entirely in-process on commodity hardware, eliminating the need for GPUs or cloud offloading.
Cost-Effective Deployment: The full pipeline runs on a $150–200 Linux box, making enterprise-grade NDR accessible to hospitals and schools.
Dual-Score Architecture: Successfully combines heuristic speed with ML precision, reducing the False Positive Rate from 6.61% (heuristic only) to 0.0002% in production scenarios.

B. Methodological Contribution: "Consejo de Sabios" (Council of Wise Men)

Human-AI Collaborative Engineering: The author developed the system using a structured methodology involving seven Large Language Models (LLMs) as intellectual peers.
Adversarial Validation: Models were tasked with identifying failure modes and challenging assumptions rather than averaging outputs.
Test-Driven Hardening (TDH): A new methodology emerged from this collaboration where tests are written before fixes to prove failure mechanisms (e.g., exposing a latent cryptographic context symmetry bug that static analysis missed).

4. Evaluation Results

Experimental Setup

Environment: Virtualized (Vagrant/VirtualBox) on a consumer laptop (Intel i9, 32GB RAM).
Datasets:
- CTU-13 Neris (2011): Used for detection performance (Ground Truth: 646 malicious flows).
- BigFlows: Used for throughput stress testing (38,064 flows).
Training: Models trained exclusively on synthetic data derived from CTU-13, CIC-IDS2017, and MAWI datasets. The CTU-13 Neris dataset was held out entirely for evaluation.

Performance Metrics

Detection Accuracy (CTU-13 Neris):
- F1 Score: 0.9985
- Precision: 0.9969
- Recall: 1.0000
- False Positive Rate: 0.0002% (2 false positives in 12,075 benign flows).
- Note: The two false positives were identified as artifacts of the VirtualBox host-only adapter (mDNS multicast and broadcast) and are expected to be absent in bare-metal deployments.
False Positive Reduction: The ML layer reduced the heuristic layer's FPR by approximately 500-fold.
Latency:
- DDoS: 0.24 µs
- Ransomware: 1.06 µs
- Total pipeline latency is dominated by network I/O, not inference.
Throughput:
- Processed up to 34–38 Mbps (limited by VirtualBox NIC emulation, not the pipeline logic).
- Zero packet drops or errors over 2.37 million packets.
- Resource usage: ~3.2 CPU cores for the ML detector; stable RAM usage at ~1.28 GB.

5. Significance and Limitations

Significance

Democratization of Security: Proves that high-fidelity, real-time ML-based NDR is achievable on low-cost hardware, addressing a critical gap for under-resourced sectors like healthcare and education.
Architectural Feasibility: Demonstrates that embedded Random Forest inference can replace heavy cloud-based or GPU-dependent solutions without sacrificing speed.
Innovation in Development: The "Consejo de Sabios" methodology offers a blueprint for independent researchers to access multi-expert peer review, while "Test-Driven Hardening" provides a robust framework for securing distributed systems against subtle protocol bugs.

Limitations & Future Work

Scope: Evaluation is currently limited to the 2011 CTU-13 Neris botnet scenario. Generalization to modern ransomware (post-2020) and encrypted C2 traffic is not yet empirically verified.
Feature Set: Only 28 of 40 planned features are implemented; 11 use sentinel values.
Deployment: Currently validated in virtualized environments; bare-metal throughput characterization is pending.
High Availability: The current etcd configuration is single-node; distributed HA clustering is planned.
Encryption: Seed distribution via etcd is not recommended for production; peer-to-peer negotiation is a future milestone.

Conclusion

ML Defender represents a significant step toward democratizing cybersecurity. By combining a highly efficient, embedded C++20 architecture with a novel human-AI collaborative development process, it delivers a system capable of detecting botnet and ransomware traffic with near-perfect accuracy on commodity hardware. While further validation against modern threats is required, the system successfully bridges the gap between academic ML research and deployable, resource-constrained security solutions.