💻 computer science

Scalable Malware Family Classification Using Quantum Kernel Based Machine Learning

This paper proposes a scalable Quantum Kernel-based Machine Learning framework that combines structural feature extraction, Linear Discriminant Analysis, and Nyström approximation to achieve 80.88% accuracy in classifying 18,836 malware samples across 23 families, outperforming classical baselines while overcoming the quadratic computational costs of traditional kernel methods.

Original authors: Ratun Rahman, Hassan Jalil Hadi, Christopher Gabriel Pedraza Pohlenz, Ali Shoker

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ratun Rahman, Hassan Jalil Hadi, Christopher Gabriel Pedraza Pohlenz, Ali Shoker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Needle in a Haystack" That Looks Like a Haystack

Imagine you are a security guard at a massive airport (the internet). Every day, thousands of people (computer files) walk through. Most are innocent travelers, but some are smugglers carrying dangerous contraband (malware).

The trouble is, the smugglers are getting very good at disguising themselves. They wear the same clothes as the innocent people, carry similar bags, and even speak the same language. Furthermore, there are thousands of different "families" of smugglers, each with their own unique style, but they all look so similar that it's incredibly hard to tell them apart.

Traditional security guards (standard computer programs) try to sort these people by looking at simple things like bag size or shoe color. But because the smugglers are so clever and look so alike, the guards often make mistakes, especially when the airport gets too crowded to check everyone individually.

The New Solution: A "Quantum Detective" with a Magic Lens

The researchers in this paper built a new kind of security system called Scalable Quantum Kernel-based Machine Learning (QKML). Think of this as hiring a team of "Quantum Detectives" who have a special magic lens.

Here is how their system works, step-by-step:

1. The "ID Card" (Feature Extraction)

First, the system looks at every file and creates a detailed "ID card" for it. It doesn't just look at the file name; it checks the internal structure, the size, the "entropy" (how messy or scrambled the code looks), and other hidden traits. This is like checking a traveler's passport, DNA, and the texture of their suitcase.

2. The "Grouping Room" (Supervised Projection)

The ID cards are huge and complicated. If you tried to show them to a Quantum Detective, they would get overwhelmed. So, the researchers use a tool called LDA (Linear Discriminant Analysis).

The Analogy: Imagine you have a giant, messy room full of people. You want to sort them into 23 different groups (families). Instead of looking at every single detail of every person, you ask them to stand in a specific formation that makes the groups obvious. You shrink the room down so the groups are clearly separated. This makes the job much easier for the next step.

3. The "Magic Lens" (Quantum Kernel)

Now comes the special part. The researchers use a Quantum Feature Map.

The Analogy: Imagine the Quantum Detective has a special pair of glasses that can see "invisible connections." In the normal world, two files might look different. But through the Quantum Lens, the detective can see that they share a hidden, complex pattern because they were made by the same criminal family.
This lens uses Quantum Circuits (tiny, complex machines that follow the rules of quantum physics) to measure how similar two files really are. It's much better at spotting subtle, tricky similarities than a standard computer.

4. The "Speed Trick" (Nyström Approximation)

Here is the biggest hurdle: If you have 18,000 files, checking every single file against every other file to see if they are similar would take forever (like checking every person in the airport against every other person). This is too slow for real life.

The Analogy: Instead of interviewing every single traveler, the Quantum Detective picks a small, smart group of "Landmarks" (about 8,000 people). They interview these landmarks thoroughly. Then, for the remaining travelers, they just ask, "Who do you look most like among these landmarks?"
This trick, called Nyström Approximation, allows the system to learn from all the data without getting bogged down in math that takes years to calculate. It's like using a map of major cities to navigate a whole country, rather than trying to memorize every single street.

The Results: Did It Work?

The researchers tested this system on a massive dataset of 18,836 malware samples from 23 different families (plus some harmless software).

The Score: The Quantum Detective got it right 80.88% of the time.
The Comparison: They compared this to the best "standard" security guards (classical machine learning models like K-Nearest Neighbors or Support Vector Machines). The standard guards got scores ranging from about 52% to 79%.
The Winner: The Quantum system beat all of them. It was better at telling the tricky, look-alike families apart.

Why This Matters

The paper claims that this method is a "sweet spot." It uses the powerful pattern-recognition abilities of quantum computing (the magic lens) but uses a smart shortcut (the landmarks) so it doesn't crash the computer or take too long.

In short: They built a system that can look at a huge pile of confusing, look-alike computer viruses, use a quantum "magic lens" to spot the hidden family connections, and sort them out faster and more accurately than current methods, all without needing a supercomputer that costs a billion dollars.

Note: The paper explicitly states these results were achieved using a simulator (a computer program pretending to be a quantum computer), not a real physical quantum machine, but it proves the math works for when real machines become available.

Technical Summary: Scalable Malware Family Classification Using Quantum Kernel–Based Machine Learning

Problem Statement

Malware family classification is a critical task in cybersecurity, essential for threat attribution, lineage analysis, and the development of defense strategies. However, the task has become increasingly difficult due to the rapid proliferation of malware variants, widespread obfuscation, packing, and automated generation techniques. These factors result in high inter-family structural similarity and polymorphism, challenging traditional machine learning models.

Existing classical approaches, including linear classifiers, tree-based models, and deep learning, face specific limitations in large-scale, multiclass settings:

Expressivity: Handcrafted features often fail to capture subtle nonlinear relationships among malware families.
Scalability: Deep learning models require extensive labeled data and computational resources, often leading to reduced interpretability.
Class Imbalance and Similarity: Pronounced class imbalance and structural overlap limit the generalization capabilities of classical models.
Quantum Limitations: While Quantum Machine Learning (QML) offers potential through high-dimensional Hilbert space embeddings, current research is often restricted to binary classification, small-scale datasets, or synthetic data. Furthermore, the construction of full quantum kernel matrices scales quadratically ( $O(N^2)$ ) with the number of samples, making direct application to large malware repositories computationally impractical.

Methodology

The authors propose a Scalable Quantum Kernel-based Machine Learning (QKML) framework designed to address both accuracy and efficiency constraints in multiclass malware family classification. The framework integrates four primary components:

1. Dataset Construction and Feature Representation

The study utilizes a large-scale dataset (QLCD) comprising 18,836 samples across 23 malware families (including 22 malicious families and one benign class).

Feature Extraction: Static structural features are extracted from Portable Executable (PE) files, including file entropy, section statistics, and ratios of virtual to raw sizes. Categorical attributes (e.g., machine type, packer) are one-hot encoded.
Preprocessing: Features undergo Z-score normalization and median imputation for missing values. Families with fewer than 50 samples are excluded to ensure reliable multiclass learning.

2. Supervised Feature Projection (LDA)

To mitigate the exponential growth of quantum circuit complexity with input dimension and the limited number of available qubits, the framework employs Linear Discriminant Analysis (LDA) as a supervised dimensionality reduction step.

LDA projects the high-dimensional feature vectors into a low-dimensional, class-aware subspace of dimension $k \le C-1$ (where $C$ is the number of classes).
This step maximizes between-class separability while minimizing within-class variance, creating a compact representation suitable for quantum encoding.

3. Quantum Kernel Construction

The projected feature vectors are encoded into quantum states using a parameterized quantum feature map (a parameterized quantum circuit, PQC).

Encoding: Features are mapped to rotation angles (e.g., $R_Y$ gates) on $n$ qubits.
Kernel Definition: The quantum kernel is defined by the squared fidelity between two quantum states: $K(z_i, z_j) = |\langle \phi(z_i) | \phi(z_j) \rangle|^2$ . This fidelity-based kernel captures nonlinear similarities that classical kernels may miss.

4. Scalable Kernel Approximation (Nyström Method)

To overcome the $O(N^2)$ computational bottleneck of full kernel matrix construction, the authors employ the Nyström approximation.

Landmark Selection: A subset of $M$ landmark samples ( $M \ll N$ ) is selected from the training set.
Low-Rank Approximation: The full kernel matrix is approximated using submatrices derived from the landmarks. This allows the construction of an explicit low-dimensional feature representation $\psi(z_i)$ for all samples.
Classification: A classical ridge regression model is trained on these Nyström-approximated features to perform multiclass classification. This approach enables learning from the entire dataset without subsampling, reducing the dominant complexity to $O(NM)$.

Key Contributions

The paper outlines four primary contributions:

Scalable QKML Framework: A novel pipeline combining fidelity-based quantum kernels with supervised LDA projection to enable class-aware dimensionality reduction and enhance discriminative power.
Nyström Approximation for Quantum Kernels: A method to construct a low-rank representation of the quantum kernel using landmark samples. This allows training on the full dataset without subsampling bias while significantly reducing the number of required quantum fidelity evaluations.
Large-Scale Real-World Dataset: The curation and preprocessing of the QLCD dataset, containing 18,836 samples from 23 families, specifically designed to evaluate QML in realistic, multiclass cybersecurity scenarios.
Empirical Validation: Comprehensive experiments demonstrating that the proposed framework outperforms classical machine learning baselines under identical feature and data splits, achieving 80.88% accuracy.

Experimental Results

Experiments were conducted on a statevector-based quantum simulator using a hybrid classical-quantum pipeline.

Performance vs. Baselines: The proposed QKML framework achieved 80.88% accuracy, outperforming classical baselines including Softmax Regression (52.79%), Linear SVM (78.73%), KNN (79.56%), and Gaussian Naive Bayes (58.44%).
Ablation Studies:
- Projection: Replacing supervised LDA with unsupervised SVD reduced accuracy from 80.88% to 78.83%, confirming the value of label-aware projection.
- Landmarks: Accuracy improved from ~74% (1,000 landmarks) to ~81.8% (8,000 landmarks), showing diminishing returns beyond $M=8000$ .
- Circuit Depth: Increasing circuit repetitions ( $L$ ) and qubit count ( $n$ ) generally improved performance, with the best results at $n=8, L=4$ .
Per-Class Performance: The model demonstrated strong performance on major families (e.g., Mediyes F1=0.988, Winwebsec F1=0.986) and the benign class (F1=0.774). Performance was lower for minority classes with high structural similarity (e.g., LockBit, LokiBot), consistent with known challenges in malware classification.
Efficiency:
- Runtime: The quantum pipeline completed training in 426.58 seconds, significantly faster than classical baselines like KNN (830s) and GaussianNB (1161s).
- Computational Cost: The quantum model required orders of magnitude fewer operations (gate and statevector proxies) compared to classical models, suggesting potential energy efficiency advantages.
- Scalability: A "Quantum-Lite" configuration (fewer qubits and landmarks) achieved 74.82% accuracy with a 5.5x speedup, demonstrating resource-efficient scaling.

Significance and Claims

The authors claim that this work bridges the gap between the theoretical expressivity of quantum kernel methods and the practical scalability requirements of real-world malware analysis.

Measurable Advantage: The study demonstrates that scalable quantum kernel techniques can offer measurable performance advantages (higher accuracy and lower loss) over classical baselines in complex, multiclass cybersecurity tasks.
Practical Feasibility: By integrating Nyström approximation and supervised projection, the framework enables the use of all available training data without the prohibitive quadratic cost of full kernel construction, making it suitable for large-scale repositories.
Future Direction: The paper positions this work as a practical step toward incorporating quantum-enhanced learning into operational malware analysis pipelines, particularly for near-term quantum devices (NISQ) and hybrid workflows.

Limitations Noted by Authors:
The authors explicitly state that the evaluation relies on a classical quantum simulator rather than real hardware, meaning noise and decoherence effects are not captured. Additionally, while Nyström approximation improves scalability, the cost of state preparation still grows with qubit count and circuit depth, and the current implementation is not yet deployed in a real-time, streaming environment.