💻 computer science

A Hybrid Machine Learning Framework for Robust Detection of Malicious DNS-over-HTTPS Tunneling Traffic

This paper proposes a statistically validated, density-aware hybrid framework combining XGBoost with unsupervised anomaly scoring to robustly detect malicious DNS-over-HTTPS tunneling, demonstrating significant improvements in recall and false negative reduction over baselines while confirming that deep generative components offer no additional benefit.

Original authors: Enas Selem¹, Rania Salama¹

Published 2026-08-10

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Enas Selem¹, Rania Salama¹

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling city where every device is constantly shouting out requests to find its way around. For decades, these shouts were like open letters passed between neighbors: anyone walking by could read them, steal the secrets inside, or even forge fake letters to send people to the wrong places. To fix this, engineers built a "secure tunnel" called DNS over HTTPS (DoH). Think of it like putting those open letters inside a locked, unbreakable steel box and sending them through a special delivery truck that looks exactly like a million other legitimate trucks. It's brilliant for privacy because no one can peek inside the box, but it creates a new problem for security guards: if a criminal wants to smuggle stolen data out of the city, they can just hide it inside one of these locked boxes. The guards can't see what's inside, and the box looks exactly like a normal delivery, making it incredibly hard to spot the bad guys.

This is the challenge tackled in a new study by researchers Enas Selem and Rania Salama. They are asking a critical question: How do we catch criminals hiding in locked boxes without breaking the locks or slowing down the city? The paper proposes a clever "hybrid" detective system. Instead of relying on just one way to spot a criminal, they combine a sharp-eyed supervisor (a machine learning model called XGBoost) with two "anomaly detectors" (Isolation Forest and Local Outlier Factor). Imagine the supervisor as a veteran cop who knows what a typical criminal looks like based on past cases. The anomaly detectors are like a pair of sensors that scream "Something is weird here!" whenever a person's behavior doesn't fit the pattern of normal, honest citizens, even if the cop has never seen that specific type of criminal before. The researchers tested this team-up against a massive dataset of internet traffic, including tricky scenarios where the criminals used tools the system had never seen before. They found that while the hybrid team didn't always beat the veteran cop in easy situations, they were incredibly effective at catching the sneaky, unseen criminals that the cop missed, reducing the number of missed attacks by about 70% to 74% in the toughest tests.

The Story of the Hybrid Detective

The researchers built a system that acts like a security checkpoint for internet traffic, but with a twist. Instead of trying to read the locked steel boxes (which is impossible because they are encrypted), the system watches how the boxes move. It looks at the timing of the deliveries, the size of the packages, and how often they arrive.

The core of their invention is a "hybrid" team. The main character is XGBoost, a powerful machine learning model that acts like a seasoned detective. It has studied thousands of past cases and knows the specific "signatures" of known bad guys. But the researchers knew that if a criminal changes their outfit or uses a new trick, the detective might get confused. So, they added two sidekicks: Isolation Forest and Local Outlier Factor.

Think of these sidekicks as "weirdness detectors." They don't care about specific criminal signatures. Instead, they are trained only on what "normal" citizens look like. If a person walks into the station acting strangely—maybe they are pacing too fast, or their package is the wrong weight compared to everyone else's—these detectors raise a red flag. They don't need to know who the criminal is; they just know that this person doesn't fit the crowd.

The researchers combined the detective's knowledge with the weirdness detectors' alerts into a single "super-feature" list. Then, they fed this list back to the detective to make the final call. They deliberately chose not to use a more complex, deep-learning "brain" (like an autoencoder) because their tests showed it made the system slower and more complicated without actually catching more criminals. They found that the simpler, lighter team of the detective plus the two weirdness detectors was the perfect balance.

The Big Test: Catching the Unseen

The real magic of this paper isn't just that the system works; it's how well it handles the unknown. The researchers put their system through a series of grueling tests called "Leave-One-Attack-Out." Imagine a game where the detective is trained on three types of thieves: those who steal with a crowbar, those with a lockpick, and those with a crowbar made of plastic. But then, the test throws a thief at them who uses a screwdriver—a tool the detective has never seen before.

In these tests, the researchers also removed the "DoH flag" (a simple clue that says "this is a locked box") to make the job even harder. They wanted to see if the system could still catch the screwdriver thief just by looking at how weird the thief's behavior was.

The results were impressive. In every single one of the fifteen different test configurations (using different random seeds to ensure fairness), the hybrid team caught significantly more criminals than the detective working alone.

For the "null" type of unseen attack, the hybrid team missed only 52.2 criminals on average, while the detective alone missed 178.6. That's a 70.8% reduction in missed attacks.
For the "srv" type, they missed 39.4 instead of 127.6 (a 69.1% reduction).
For the "txt" type, they missed 26.2 instead of 101.2 (a 74.1% reduction).

The statistical tests confirmed this wasn't just luck; the difference was real and significant. The hybrid system acted like a safety net, catching the criminals who slipped through the detective's fingers because they were doing something slightly "off" that the detective hadn't memorized yet.

Where It Shines and Where It Stumbles

The paper is very honest about where this system works best and where it doesn't.

The Sweet Spot: When the system faces a new type of attack it has never seen before, the "weirdness detectors" are heroes. They spot the anomaly in the traffic flow and help the detective make the right call.
The Neutral Zone: When the system is tested on data that looks exactly like what it learned from (like the "Standard" or "Full Data" scenarios), the hybrid team does slightly better or ties with the detective alone. In one specific scenario called "Cross-Scenario," where the normal traffic itself changed slightly between training and testing, the two systems were essentially tied. In fact, the weirdness detectors sometimes raised false alarms here, flagging normal but slightly different traffic as suspicious.
The Speed Limit: The researchers also checked how fast the system could run. They found that checking one single package at a time (streaming) was about 16 times slower for the hybrid team than for the detective alone. However, if they checked packages in batches (like a whole truckload at once), the slowdown was only about 2.3 times. This means the system is great for analyzing traffic in short bursts (batch processing) but might be too slow for checking every single packet the instant it arrives.

The "Feature Gap" Problem

There was one tricky case where the system struggled: a tool called dns2tcp. Even with the hybrid team, the system only caught about 35% of the dns2tcp attacks. The researchers dug deep and found a "feature-domain gap." It turns out that dns2tcp traffic in this dataset is just fundamentally different—it's shorter, faster, and moves less data than the other tools. It's like trying to spot a thief who is running on a different planet; the "weirdness detectors" couldn't tell if the behavior was weird because it was a crime or just because it was from a different world. The system couldn't fix this just by adding more rules; the data itself was too different from what the system was trained on.

The Takeaway

This paper shows that when it comes to catching digital criminals hiding in encrypted tunnels, having a "second pair of eyes" that looks for weirdness rather than just matching fingerprints is a game-changer. The hybrid framework doesn't replace the expert detective; it gives them a safety net for the unknown. While it's not a magic bullet that solves every problem instantly (and it does cost a bit more in computing power), it proves that combining different types of intelligence—supervised learning for known threats and unsupervised learning for the weird and unknown—creates a much more robust defense. The researchers suggest that this approach, along with their rigorous testing methods, should be the new standard for building security systems that can handle the next generation of sneaky attacks.

Technical Summary: A Hybrid Machine Learning Framework for Robust Detection of Malicious DNS-over-HTTPS Tunneling Traffic

1. Problem Statement

DNS over HTTPS (DoH) enhances user privacy by encrypting DNS queries within standard HTTPS traffic (port 443). However, this encryption renders Deep Packet Inspection (DPI) ineffective, allowing attackers to weaponize DoH for covert data exfiltration and command-and-control (C2) communication via tunneling. While machine learning (ML) approaches have emerged to detect such traffic using flow-level statistics (e.g., packet timing, session length), existing literature exhibits critical limitations:

Over-reliance on specific features: Many models depend heavily on the "DoH protocol indicator," which attackers can suppress.
Lack of generalization: Most evaluations train and test on the same attack distributions, failing to assess resilience against unseen tunneling variants or tools.
Limited interpretability: Few systems provide decision-level explanations necessary for security analysts to act on alerts.
Unnecessary complexity: Some frameworks incorporate deep generative components (e.g., autoencoders) without demonstrating measurable performance gains over lighter alternatives.

2. Methodology

The authors propose a density-aware hybrid framework designed as a batch-processing analyzer rather than an inline real-time filter. The architecture integrates supervised classification with unsupervised anomaly scoring to create a "safety net" for unseen threats.

Core Architecture

Supervised Baseline: An XGBoost classifier trained on 32 normalized flow-level statistical features.
Unsupervised Anomaly Scoring: Two independent detectors fitted only on benign traffic to learn the density of normal behavior:
- Isolation Forest (IF): Provides a global anomaly score based on how quickly a point is isolated in a tree structure.
- Local Outlier Factor (LOF): Provides a local anomaly score based on the density deviation of a point relative to its $k=20$ nearest neighbors.
Feature Fusion: The raw features are concatenated with the iso_score and lof_score to form a hybrid feature vector.
Classification: The final XGBoost model classifies traffic as benign or malicious.
Interpretability: SHAP (SHapley Additive exPlanations) is applied to explain feature contributions for both the baseline and hybrid models.

Design Decisions

Ablation Study: The authors explicitly excluded deep generative components (e.g., denoising autoencoders). Testing showed that adding these components increased architectural complexity and training time without yielding measurable improvements in detection metrics.
Thresholding: A fixed threshold of 0.5 is used for robustness evaluations (Leave-One-Attack-Out) to ensure fair comparison, while F1-maximizing thresholds are used for standard scenarios.

3. Key Contributions

The paper outlines five primary contributions:

Rigorous Evaluation Protocol: Introduction of a multi-seed Leave-One-Attack-Out (LOAO) protocol (3 attack variants × 5 seeds = 15 configurations) and a Leave-One-Tool-Out (LOTO) evaluation across structurally different tools (iodine, dns2tcp, dnscat2). This exceeds standard single-split evaluations.
Statistical Validation: Formal significance testing using McNemar's exact test for primary scenarios and Wilcoxon signed-rank/exact sign tests for robustness evaluations, ensuring claims are not due to random variance.
Mechanistic Explanation: Identification that the hybrid model's advantage is not uniform; it specifically "rescues" flows sitting near the XGBoost decision boundary (ambiguous cases) rather than improving detection across all traffic.
Transparent Reporting: Honest disclosure of limitations, including a genuine feature-domain gap with the dns2tcp tool and the computational overhead of the hybrid approach.
Optimized Hybrid Framework: A validated architecture combining XGBoost with IF and LOF, avoiding unnecessary deep learning components.

4. Experimental Results

Primary Scenarios (Standard, Full Data, Cross-Scenario)

Evaluated on the CIRA-CIC-DoHBrw-2020 dataset:

Standard & Full Data: The hybrid framework outperformed the baseline XGBoost. In the Full Data scenario, Recall improved from 97.95% to 98.00% (F1: 95.96% vs. 96.11%), a statistically significant difference ( $p=0.024$ ).
Cross-Scenario: When training and test data came from different capture files (simulating distribution shift), the models were tied in Recall (96.59%). The hybrid model showed a slight, non-significant drop in Precision due to increased false positives, indicating that anomaly scoring offers no benefit when benign traffic distributions shift significantly.

Robustness Evaluations

Leave-One-Attack-Out (LOAO): With the DoH indicator removed and one attack variant held out from training:
- The hybrid framework reduced False Negatives by 70.8% to 74.1% across all three held-out variants (null, srv, txt) compared to the baseline.
- This improvement was consistent across all 15 random seed configurations and statistically significant ( $p=0.031$ , the maximum attainable with 5 pairs).
Leave-One-Tool-Out (LOTO): Evaluated across iodine, dns2tcp, and dnscat2:
- The hybrid model consistently outperformed the baseline in Recall and F1 for all tools.
- Limitation: While statistically superior, the absolute performance for dns2tcp remained low (Recall ~35.4%). The authors attribute this to a "feature-domain gap," where dns2tcp traffic statistics (shorter duration, lower byte counts) differ fundamentally from the training distribution, a limitation no classifier could overcome without feature engineering.

Computational Overhead

Single-Flow Latency: The hybrid approach is ~16x slower (38.53 ms vs. 2.39 ms) due to the per-call overhead of IF and LOF.
Batch Throughput: In batch processing (2,000 flows), the overhead reduces to ~2.3x slower (130k vs. 301k flows/sec), making it viable for batch-based security operations but less suitable for strict real-time, single-flow streaming.

5. Significance and Claims

The paper argues that the primary value of this framework lies in operational robustness rather than marginal gains in ideal conditions.

Safety Net for Unseen Threats: The integration of unsupervised anomaly scoring provides a statistically validated safety net. When the strongest signal (the DoH indicator) is removed or the attack variant is unseen, the hybrid model maintains high detection rates by relying on the deviation of traffic from learned benign densities.
Explainability as a Necessity: The use of SHAP allows analysts to verify that alerts are driven by genuine anomalies (e.g., elevated lof_score and byte-volume features) rather than just the DoH flag, facilitating better triage in Security Operations Centers (SOCs).
Realistic Expectations: The authors emphasize that while the framework is robust, it is not a panacea. It does not universally outperform baselines (e.g., in Cross-Scenario settings) and cannot bridge fundamental feature-domain gaps (e.g., dns2tcp).

In conclusion, the paper posits that combining supervised classification with density-aware unsupervised scoring creates a more resilient system for detecting encrypted tunneling, provided it is deployed in batch-processing environments and its limitations regarding specific tool architectures are understood.