💻 computer science

Quantum Autoencoders for Anomaly Detection in Cybersecurity

This paper demonstrates that Quantum Autoencoders, specifically an 8-feature model with Dense-Angle encoding and a RealAmplitude ansatz, outperform Classical Autoencoders in cybersecurity anomaly detection on the BETH dataset by achieving a higher F1 score (0.87 vs. 0.77) even when trained on substantially fewer samples.

Original authors: Rohan Senthil, Swee Liang Wong

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rohan Senthil, Swee Liang Wong

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Finding the "Odd One Out" in a Sea of Normalcy

Imagine you are a security guard at a massive, busy train station. Every day, millions of people walk through the doors. 99.9% of them are just regular commuters doing normal things: buying tickets, checking watches, and walking to their trains. But every now and then, someone tries to sneak in a bomb or steal a bag.

Your job is Anomaly Detection: spotting the bad guy among the millions of good guys.

The problem? The bad guys are rare. If you try to teach a computer to spot them by showing it thousands of pictures of bad guys, you'll fail because you don't have enough pictures of them. You only have plenty of pictures of normal people.

This is where the researchers, Rohan Senthil and Swee Liang Wong, stepped in. They asked: "Can we use the strange, powerful rules of Quantum Physics to build a better security guard than the ones we have today?"

The Old Guard: Classical Autoencoders (CAEs)

Before this paper, security systems used "Classical Autoencoders" (CAEs). Think of a CAE like a photocopier that only knows how to copy normal people.

Training: You feed the photocopier thousands of photos of normal commuters. It learns exactly what a "normal" person looks like.
The Test: When a new person walks in, the photocopier tries to recreate their image based on what it learned.
The Result:
- If it's a normal person, the copy looks great.
- If it's a bad guy (an anomaly), the photocopier gets confused. It tries to force the bad guy's face into a "normal" shape, and the result looks terrible and distorted.
- The Alarm: The system sees the bad distortion and sounds the alarm.

The Flaw: The paper notes that this "photocopier" struggles when you don't have many training photos. If you only show it a few normal people, it gets confused and misses the bad guys.

The New Guard: Quantum Autoencoders (QAEs)

The researchers tried a Quantum Autoencoder (QAE). If the Classical Autoencoder is a photocopier, the Quantum Autoencoder is like a magical, multi-dimensional prism.

Instead of just looking at a flat photo, the QAE uses the weird rules of quantum mechanics (like superposition and entanglement) to compress information in a way that classical computers can't.

How they tested it:
They used a real-world dataset called BETH, which is like a giant logbook of network traffic from a "honeypot" (a fake server designed to attract hackers).

The Challenge: The dataset was heavily skewed. There were thousands of normal events and very few hacker attacks.
The Setup: They trained the Quantum model only on the normal events, just like the classical model.

The Magic Tricks: Encoding and Compression

To make the quantum computer understand the data, the researchers had to translate the data into "quantum language." They tried four different translation methods (called Encodings):

Amplitude Encoding: Like trying to fit a whole library of books into the thickness of a single sheet of paper.
Angle Encoding: Like turning a dial for every feature of the data.
Dense-Angle Encoding: A clever trick where they squeeze two pieces of information onto one dial (qubit). This is like packing two suitcases into one small space.
Efficient SU2 Encoding: A complex method that twists and links the dials together to save space.

They also tried different "blueprints" for the quantum circuit, called Ansatz (think of these as different architectural designs for the machine).

The Results: The Quantum Advantage

Here is the punchline of the paper:

When the researchers tested the models on the hacker data, the Quantum Autoencoder won, but with a catch.

The Scenario: They gave the models a very small amount of training data (a "data-limited" setting).
The Outcome:
- The Classical Autoencoder (the photocopier) got a score of 0.77. It missed a lot of bad guys because it didn't have enough practice.
- The Quantum Autoencoder (the prism) got a score of 0.87. It was much better at spotting the anomalies, even though it was trained on far fewer examples.

Why? The paper suggests that the Quantum model, specifically the one using Dense-Angle Encoding and a RealAmplitude design, was able to "understand" the shape of "normal" data much more efficiently. It didn't need as many examples to learn the pattern.

The Bottom Line

The paper claims that in a world where we have tons of normal data but very little data on actual attacks, Quantum Autoencoders might be a better tool than classical ones.

They demonstrated that by using specific quantum tricks (like squeezing two data points into one qubit), they could build a security system that is more sensitive to "oddities" even when it hasn't seen many examples of them before.

In short: The quantum machine learned the "vibe" of normal traffic faster and better than the classical machine, making it a promising tool for catching cyber threats when data is scarce.

Technical Summary: Quantum Autoencoders for Anomaly Detection in Cybersecurity

Problem Statement

Cybersecurity anomaly detection faces significant challenges due to the extreme imbalance between normal and anomalous events, where normal activities vastly outnumber attacks. Furthermore, the rapid evolution of cyber threats necessitates large, high-quality, labeled datasets for supervised learning, which are often unavailable or impractical to maintain. While Classical Autoencoders (CAEs) have been employed to learn the distribution of "normal" data to identify outliers via reconstruction error, they often struggle in data-limited settings. This paper investigates whether Quantum Autoencoders (QAEs) can offer superior data efficiency and performance in detecting anomalies within cybersecurity contexts, specifically when trained on limited samples of normal data.

Methodology

Dataset and Preprocessing

The study utilizes the BPF-Extended Tracking Honeypot (BETH) dataset, a real-world collection of over 8 million network traffic events from cloud honeypots. The dataset is highly imbalanced, with anomalous events constituting a small fraction of the training and validation sets (0.2% and 0.4%, respectively) but dominating the test set (90.7%) to simulate a realistic detection scenario.

Preprocessing: Features were engineered according to established protocols (Highnam et al.), including mapping process and user IDs, handling return values, and applying target encoding to high-cardinality categorical features.
Feature Selection: The initial transformed dataset contained 24 features. Experiments focused on subsets of these features, specifically evaluating models with 8, 16, and 24 features depending on the encoding scheme.

Quantum Autoencoder (QAE) Architecture

The QAE implementation, built using Qiskit and simulated on local hardware and Google Colab L4 GPUs, follows a three-layer conceptual structure:

Input/Encoder: Encodes classical data into quantum states.
Bottleneck: Compresses the state by discarding "trash" qubits.
Output/Decoder: In this specific implementation, explicit reconstruction is bypassed. Instead, the discarded trash qubits are compared against reference qubits using a SWAP test.

Anomaly Scoring:
The training objective minimizes the cost function derived from the SWAP test, which measures the fidelity ( $S$ ) between the trash qubits and reference states. The anomaly score ( $A$ ) is defined as $A = 1 - S$ . A score of 0 indicates a normal instance (high fidelity), while 1 indicates a maximally anomalous instance.

Experimental Variables:

Encoding Techniques: The study evaluated four distinct quantum encoding strategies:
1. Amplitude Encoding: Maps data to state amplitudes.
2. Angle Encoding: Maps features to rotation angles ( $R_y$ ).
3. Dense-Angle Encoding: Encodes two features per qubit using both $R_y$ and $R_z$ rotations, reducing qubit requirements.
4. Efficient SU2 Encoding: An entanglement-based scheme from Qiskit.
Ansatz Types: Parameterized circuits including RealAmplitudes and PauliTwoDesign were tested.
Training Configuration: The models were trained solely on normal data using the COBYLA gradient-free optimizer for 60 iterations with a batch size of 64.

Classical Baseline

A Classical Autoencoder (CAE) was implemented for benchmarking, trained on the same normal data subsets to compare performance metrics including Precision, Recall, F1-score, and AUROC.

Key Results

The study demonstrates that QAEs can outperform CAEs in data-constrained environments, with performance heavily dependent on encoding and feature selection strategies.

Optimal Configuration: The best-performing QAE model utilized 8 features with Dense-Angle encoding and a RealAmplitudes ansatz.
Performance Metrics:
- In a data-limited setting, the optimal QAE achieved an F1 score of 0.87.
- The corresponding CAE achieved an F1 score of 0.77.
Data Efficiency: The QAE demonstrated the ability to outperform the classical counterpart even when trained on substantially fewer samples.
Encoding Impact: The results highlighted that the choice of quantum encoding significantly impacts model expressivity and performance. For instance, the Efficient SU2 encoding required larger feature sets (16 or 24 features) to avoid over-compression issues that occurred with smaller feature sets.

Significance and Claims

The authors claim that this work provides empirical evidence that Quantum Autoencoders offer practical advantages for anomaly detection in cybersecurity, particularly in scenarios characterized by data limitations.

Data Efficiency: The primary contribution is the demonstration that QAEs can achieve higher detection accuracy (F1 score) than CAEs while utilizing fewer training samples. This addresses the critical challenge of maintaining large, labeled datasets for evolving cyber threats.
Encoding Sensitivity: The paper underscores that the performance of quantum models is highly dependent on the selection of encoding techniques and feature strategies, suggesting that domain-specific optimizations are necessary for realizing the full potential of Quantum Machine Learning (QML).
Practical Applicability: By utilizing the BETH dataset, which reflects real-world network conditions and attack patterns, the study argues that QAEs are not merely theoretical constructs but viable candidates for operational cybersecurity challenges where anomalous data is scarce.

The work concludes that while specialized QAE variants are still in early stages of development, the observed improvements in data-limited settings suggest a promising trajectory for quantum-enhanced anomaly detection.