💻 computer science

Behavioral Anomaly Detection and Malicious Service Identification in the Tor Network Using Deep Learning

This paper proposes MBRD-Tor, a three-stage deep learning framework that addresses the challenges of protocol homogenization, encryption noise, and extreme class imbalance in the Tor network by fusing multi-scale behavioral signatures, employing an ensemble-based quality estimation for adaptive weighting, and utilizing GAN-generated synthetic features to achieve superior malicious service detection performance.

Original authors: Amirhossein Saviz

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amirhossein Saviz

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling city where everyone is free to walk around, but some people want to wear masks to hide their faces. This is the Tor network, a digital tool designed to protect privacy by bouncing messages through a series of secret tunnels so no one can see who sent what. However, just like in a real city, bad actors sometimes use these masks to hide their crimes, sending secret commands to viruses or running illegal shops. The problem for the city's security guards is that Tor is so good at its job that it makes every single message look exactly the same: a uniform, 512-byte block of data. It's like trying to spot a pickpocket in a crowd where everyone is wearing the exact same gray suit, standing in the exact same pose, and walking at the exact same speed. Traditional security tools, which usually look for differences in size or shape, are completely confused. They can't tell the difference between a harmless tourist and a criminal because the "uniforms" are identical. This creates a massive headache for defenders: they need to find the tiny needle of a malicious attack hidden in a haystack of billions of identical-looking blocks, and the bad guys are getting smarter at hiding their tracks.

This is where a new study by Amirhossein Saviz comes in, proposing a clever new way to catch these digital criminals without ever needing to peek inside their locked suitcases. The paper introduces a system called MBRD-Tor, which acts less like a rigid security guard checking IDs and more like a detective who pays attention to the rhythm and behavior of the crowd. Instead of trying to read the content of the messages (which is encrypted and impossible to see), the system watches how the messages move: how fast they arrive, which direction they are going, and the order in which they appear.

The researchers realized that even if the "suits" are identical, the way a pickpocket moves through a crowd is different from a normal tourist. A criminal might send a signal out and then wait for a big package to come back in, creating a specific pattern. To catch this, the team built a three-step process. First, they created a "behavioral fingerprint" by combining timing, position, and direction into a single, compact summary of the traffic. Second, they faced a tricky problem: in the real world, bad traffic is incredibly rare (sometimes less than 1% of all traffic), and some of the data is just "noisy" or messy. To fix this, they used a panel of five different "judges" (an ensemble of algorithms) to vote on how trustworthy each piece of data is, giving more weight to the clean, clear examples and ignoring the messy ones. Finally, to solve the problem of having too few examples of bad traffic to learn from, they used a special kind of AI called a Generative Adversarial Network (GAN). Think of this as a digital art forger that creates perfect, fake copies of the rare criminal patterns, but only if a strict committee of judges agrees the fake copies are indistinguishable from the real thing. This allows the system to learn what the criminals look like without needing millions of real-world examples.

When the team tested their new system on a dataset of real Tor traffic, the results were promising. Under normal conditions, where bad traffic is a bit more common, their system achieved an F1-score (a measure of overall accuracy) of 73.70%, which was significantly better than the previous best methods. But the real magic happened when they simulated the extreme conditions of the real internet, where malicious traffic makes up only 1% of the total. In this scenario, most other systems completely failed, dropping their ability to catch criminals to near zero. However, MBRD-Tor held its ground, maintaining a 21.1% recall rate (the ability to actually find the bad guys) and an F1-score of 33.3%. The study suggests that by focusing on the subtle behavioral rhythms of the data and using a smart, multi-step process to clean up the training data, it is possible to spot malicious activity even when it's hiding in plain sight among millions of identical-looking blocks. The authors conclude that this approach offers a robust, privacy-friendly way to keep the digital city safe without needing to break the encryption that protects everyone's privacy.

Technical Summary: MBRD-Tor Framework for Malicious Tor Traffic Detection

1. Problem Statement

The Tor (The Onion Router) network, while essential for user privacy, is increasingly exploited by malicious actors for Command and Control (C2) communication, ransomware coordination, and hosting illegal dark web services. Detecting these activities faces three fundamental technical challenges:

Protocol-Level Homogenization: Tor's fixed 512-byte cell architecture strips away size-based fingerprinting vectors, creating feature homogeneity that standard traffic analysis cannot easily distinguish.
Heterogeneous Sample Quality: Multi-layer encryption, adaptive padding defenses, and network jitter result in highly variable discriminability across training samples, making uniform treatment of data suboptimal.
Extreme Class Imbalance: In real-world perimeter traffic, malicious flows often constitute less than 1% to 5% of total traffic. Standard Empirical Risk Minimization (ERM) models optimized on such data suffer from catastrophic recall drops on rare malicious behaviors.

2. Methodology: The MBRD-Tor Framework

The paper proposes MBRD-Tor (Multi-scale Behavioral Representation and Data-centric refinement for Tor), a three-stage, data-centric framework designed to shift focus from complex classifier training to enhancing representation engineering and training-data quality.

Stage 1: Multi-Scale Representation Learning and Compression

To overcome the fixed-size cell homogenization, the framework treats a Tor flow as a sequence of cells ( $n=50$ ), extracting three metadata attributes per cell: inter-arrival time ( $\Delta t$ ), position, and direction.

Fused Embeddings: Three complementary embeddings are generated:
- Temporal Encoding: Log-normalized and z-scored inter-arrival times.
- Positional Encoding: Sinusoidal functions encoding sequence ordering.
- Directional Encoding: One-hot representation of cell direction (inbound/outbound).
Contextualization: These vectors are combined and enriched with local (sliding window) and global context vectors to capture directional patterns, such as periodic beaconing.
Autoencoder Compression: A four-layer feedforward Autoencoder compresses the comprehensive flow representation into a compact latent space ( $z \in \mathbb{R}^{64}$ ). This model is trained using a joint loss function that combines reconstruction error with an auxiliary classification loss to ensure the latent space retains discriminative power.

Stage 2: Quality Estimation and Data Refinement

This stage addresses sample quality variance and extreme class imbalance through an ensemble-based approach.

Ensemble Quality Estimation: Five complementary anomaly and density estimators (Isolation Forest, Local Outlier Factor, One-Class SVM, Variational Autoencoder, and Masked Autoencoder for Density Estimation) are fitted on the latent representations of each class.
Adaptive Weighting: The outputs of these five models are normalized and aggregated into a unified "compatibility score." These scores are converted into class-normalized training weights using a temperature-controlled softmax, allowing the model to prioritize high-quality samples and down-weight noisy or ambiguous ones.
Multi-GAN Synthesis: To counter extreme class skew without blurring decision boundaries (a common issue with SMOTE), the framework trains three structurally distinct Generative Adversarial Networks (GANs) per class:
- GAN-A (Shallow): Captures coarse-grained distributions.
- GAN-B (Deep): Captures fine-grained behavioral structures.
- GAN-C (Residual): Preserves input noise and stabilizes gradients.
Strict Consensus Filtering: Synthetic samples generated by the GANs are subjected to a strict acceptance test. Only synthetic latent vectors that exceed a safety threshold based on the ensemble consensus are injected into the training pool, ensuring they remain "on-manifold."

Stage 3: Classifier-Agnostic Deployment

The framework outputs validated latent embeddings that are "classifier-agnostic." This allows security operators to plug in various downstream classifiers (e.g., lightweight decision forests for edge devices or deep neural networks for central servers) without retraining the feature extraction pipeline.

3. Key Contributions

Multi-Scale Behavioral Signatures: A novel method for fusing temporal, positional, and directional cell embeddings to bypass the limitations of Tor's fixed packet size.
Ensemble-Based Instance Weighting: A robust mechanism using five heterogeneous estimators to calculate sample compatibility, ensuring training stability even with noisy data.
Consensus-Gated GAN Synthesis: A multi-generator synthesis approach where synthetic data is strictly filtered by an ensemble consensus, preventing the introduction of boundary-diluting artifacts common in standard oversampling techniques.
Privacy-Compliant Architecture: The system operates solely on observable metadata (timing, position, direction) without requiring Deep Packet Inspection (DPI) or decryption, adhering to international privacy mandates.

4. Experimental Results

The framework was evaluated on the CCS'22 Tor malware dataset, which contains authenticated traffic from 157 malware binaries and benign Tor Browser flows.

Natural Class Distribution: Under a natural benign-to-malicious ratio (~6.5:1), MBRD-Tor achieved an F1-score of 73.70%. This represents a 15.53% relative improvement over the second-best baseline (LightGBM) and significantly outperforms other state-of-the-art deep learning and traditional machine learning models.
Extreme Class Imbalance: In a simulated environment where malicious traffic comprised only 1% of the total, MBRD-Tor maintained a recall of 21.1% and an F1-score of 33.3%. In contrast, generic classifiers (including LightGBM and AN-Net) saw their recall drop below 10%, effectively collapsing on the minority class.
Ablation Studies:
- Removing the multi-scale representation layer (Stage 1) resulted in a 15.69% drop in F1-score, confirming the necessity of capturing timing-directional dependencies.
- Replacing the specialized GAN synthesis with standard SMOTE interpolation led to a 7.76% drop in F1-score and a significant loss in precision, demonstrating the superiority of the consensus-filtered synthesis.

5. Significance and Claims

The paper claims that MBRD-Tor successfully bridges the gap between theoretical deep learning models and the practical constraints of Tor traffic analysis. Its primary significance lies in its ability to maintain high detection capabilities under extreme class skew (1% malicious traffic) where standard models fail.

By shifting the focus from purely training complex classifiers to enhancing data quality and representation, the framework offers a modular solution that is:

Robust: Resilient to the homogenization of Tor cells and the noise of encrypted traffic.
Scalable: The classifier-agnostic design allows for flexible deployment across different hardware constraints (edge vs. central).
Privacy-Preserving: It detects threats without violating user anonymity through decryption or payload inspection.
Novelty-Aware: The ensemble compatibility scoring allows the system to flag "unknown anomalies" for manual inspection, providing a mechanism for detecting zero-day threats.

The authors conclude that MBRD-Tor provides a reliable, high-performance detection mechanism that preserves critical threat-detection capabilities in environments where standard deep learning models suffer complete minority-class collapse.