🤖 AI

Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Models

This paper proposes a zero-label malware family classification framework that leverages a weighted hierarchical ensemble of pretrained large language models to aggregate decision-level predictions, thereby enhancing robustness and scalability in open-world scenarios without relying on labeled datasets or model retraining.

Original authors: Samita Bai, Hamed Jelodar, Tochukwu Emmanuel Nwankwo, Parisa Hamedi, Mohammad Meymani, Roozbeh Razavi-Far, Ali A. Ghorbani

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samita Bai, Hamed Jelodar, Tochukwu Emmanuel Nwankwo, Parisa Hamedi, Mohammad Meymani, Roozbeh Razavi-Far, Ali A. Ghorbani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to identify a criminal. In the old days, you'd look for a specific fingerprint or a unique tattoo (these are like handcrafted features in traditional malware detection). But modern criminals are smart; they wear masks, change their clothes, and even alter their fingerprints (this is obfuscation and packing). If you rely only on those specific details, you'll miss them.

This paper proposes a new way to catch these digital criminals using a team of AI detectives (Large Language Models, or LLMs) who don't need to be taught the specific "look" of a criminal. Instead, they use their general understanding of language and code to figure out what the criminal is trying to do.

Here is the breakdown of their solution, explained simply:

1. The Problem: The "One Detective" Trap

Usually, when we use AI to find malware (bad software), we train it on thousands of labeled examples. But in the real world, new malware appears every day, and we often don't have labels for it yet.

The Issue: If you ask just one AI detective to identify a criminal, it might guess wrong. One AI might think a "Trojan" (a hidden spy) is a "Virus" (a self-replicating germ) because they look similar. Another might get confused by a "Backdoor" (a secret entry) and call it a "Spyware."
The Analogy: Imagine asking a single person to identify a bird in a foggy forest. They might guess "Eagle" when it's actually a "Hawk." They are smart, but they can be inconsistent.

2. The Solution: A "Council of Experts"

Instead of relying on one AI, the authors created a Council of Experts. They used four different powerful AI models (like Qwen, CodeLLaMA, GPT-4, and GPT-5).

No Training Needed: These AIs are already "pre-trained" (they've read the whole internet and millions of code snippets). The researchers didn't need to re-teach them. They just asked them a simple question: "What kind of bad software is this?"
The Zero-Label Magic: This is the cool part. They didn't need a list of "correct answers" to train the AI. They just let the AIs use their existing knowledge to guess.

3. The Secret Sauce: The Weighted Hierarchy

Just asking four AIs and taking a "majority vote" isn't enough, because some AIs are better than others, and some are confused by specific types of malware. The authors built a three-step voting system:

Step A: The "Weighted" Vote (Listening to the Best)

Not all AIs are created equal. Some are better at spotting "Ransomware" (data lockers), while others are better at spotting "Trojans."

The Analogy: Imagine a jury. If one juror is a famous forensic expert, their vote counts more than a juror who just guessed. The researchers gave "more votes" to the AIs that were historically more accurate.

Step B: The "Hierarchy" (Big Picture First)

Sometimes the AIs can't agree on the exact name (e.g., is it a "Spyware" or a "Bot"?). Instead of getting stuck, the system zooms out.

The Analogy: If the jury can't agree if the criminal is a "Pickpocket" or a "Shoplifter," they first agree on the bigger category: "Thief."
The system groups malware into broad behaviors first (e.g., "Self-Replicating," "Stealing Data," "Installing Payloads"). Once they agree on the broad behavior, they try to pick the specific family. This stops the AIs from arguing over tiny details when they are actually mostly right about the big picture.

Step C: The "Tie-Breaker"

If the AIs are still split, the system uses a rule of thumb: "Pick the most specific, informative answer."

The Analogy: If the jury is split between "Dog" and "Poodle," the rule says, "Go with Poodle" because it tells you more about the animal.

4. The Results: A Smarter Detective Team

The researchers tested this on a massive dataset of real-world Windows malware.

The Outcome: The "Council of Experts" (the weighted hierarchical ensemble) was much better at identifying malware families than any single AI could be on its own.
Why it matters: It works even when the malware is heavily disguised (packed/obfuscated) and even when the researchers didn't have a "cheat sheet" (labeled data) to teach the AI.

Summary Metaphor

Think of traditional malware detection as a security guard checking a list of known faces. If the criminal wears a mask, the guard fails.

This new method is like a team of seasoned detectives sitting in a room. They don't have a list of faces. Instead, they look at the criminal's behavior (how they move, what tools they carry).

They each make a guess.
The most experienced detectives get to speak louder (Weighting).
If they can't agree on the exact name, they agree on the type of crime first (Hierarchy).
They combine their wisdom to catch the criminal, even if the criminal is wearing a disguise.

This approach makes cybersecurity more robust, scalable, and ready for the "unknown" threats of the future.

1. Problem Statement

Malware family classification is a critical yet challenging task in cybersecurity, particularly in "open-world" scenarios characterized by:

Obfuscation and Packing: Techniques that hide malicious intent and alter static features.
Rapid Evolution: New variants and polymorphic code that render static signatures and handcrafted features obsolete.
Data Scarcity: A lack of large, labeled datasets required for supervised deep learning training.
Limitations of Existing Methods: Traditional machine learning (ML) and deep learning (DL) approaches rely heavily on feature engineering, supervised retraining, or costly dynamic analysis (sandboxing), which limits their scalability and robustness against concept drift.

While Large Language Models (LLMs) show promise in zero-shot reasoning and code understanding, individual LLMs suffer from prediction instability, inconsistency across prompt variations, and hallucinations when classifying semantically overlapping malware families.

2. Methodology

The authors propose a Zero-Label, Decision-Level Hierarchical Ensemble Framework that aggregates predictions from multiple pretrained LLMs without fine-tuning or labeled training data.

A. Core Components

Input & Prompting:
- Dataset: The framework utilizes the SBAN (Source-Binary-Assembly-Natural Language) dataset, specifically the malware subset containing Windows PE samples with source code and API imports.
- Zero-Shot Prompting: A single, fixed prompt instructs LLMs to classify a sample into one of 10 canonical malware families (e.g., Trojan, Worm, Ransomware, Botnet). The output is strictly restricted to the family name to minimize hallucination.
- Models: The ensemble consists of four diverse pretrained LLMs: Qwen3-4B, CodeLLaMA-7B, GPT-4.1, and GPT-5.1. (DeepSeekCoder was excluded due to poor performance).
Normalization & Validation:
- Raw LLM outputs are normalized to map synonymous labels (e.g., "Backdoor" vs. "RAT") to canonical names.
- Invalid responses (explanations, markdown, empty strings) are filtered out.
Weighted Hierarchical Ensemble Strategy:
Instead of simple majority voting, the framework employs a three-stage decision process:
- Stage 1: Weighted Family Voting: Models are assigned reliability weights ( $w_i$ ) based on their Macro-F1 scores on a small, human-labeled "gold set" (200 samples). If a family achieves a weighted score $> 50\%$ of the total weight, it is selected immediately.
- Stage 2: Behavior-Group Consolidation: If no single family wins, predictions are aggregated into 5 coarse-grained behavior groups (e.g., "Trojan-like," "Self-Replicating," "Installer"). If a behavior group wins the weighted majority, the highest-scoring specific family within that group is selected.
- Stage 3: Specificity-Based Tie Resolution: If ties persist, a predefined specificity ranking resolves the ambiguity (e.g., preferring "Backdoor" over generic "Trojan" if both are tied, as it is more behaviorally informative).

B. Workflow

The pipeline operates as follows:
Input Source Code $\rightarrow$ Parallel LLM Inference (Zero-Shot) $\rightarrow$ Normalization $\rightarrow$ Weighted Hierarchical Aggregation $\rightarrow$ Final Family Label.
Note: The gold set is used only for weight calibration and evaluation, never for training or inference.

3. Key Contributions

Decision-Level Aggregation: Formulates malware classification as a decision aggregation problem over pretrained LLMs, eliminating the need for feature engineering, supervised retraining, or dynamic execution.
Performance-Aware Weighting: Introduces a macro-F1-based weighting scheme that leverages the complementary strengths of different LLMs, reducing the impact of individual model instability.
Hierarchical Decision Structure: Designs a behavior-aware hierarchy (Coarse $\to$ Fine) that mimics analyst reasoning. This reduces semantic ambiguity between overlapping families (e.g., distinguishing between Trojans and Backdoors) by first resolving broad behavioral categories.
Robustness in Constrained Settings: Demonstrates that this approach outperforms individual models and flat ensembles in scenarios where labeled data and dynamic analysis are unavailable.

4. Experimental Results

The framework was evaluated on a 200-sample gold-standard subset of the SBAN dataset.

Baseline Performance: Individual LLMs showed high variance. GPT-4.1 achieved the highest accuracy (0.710), while GPT-5.1 had the best Macro-F1 (0.427). CodeLLaMA significantly underperformed (Macro-F1: 0.159).
Ensemble Performance:
- Uniform Voting: Improved accuracy to 0.740 (Macro-F1: 0.481).
- Weighted Voting (No Hierarchy): Slightly lower accuracy (0.735) than uniform voting, suggesting weighting alone is insufficient without structural guidance.
- Weighted Hierarchical Ensemble (Final): Achieved the best overall performance with 0.750 Accuracy and 0.481 Macro-F1.
Key Findings:
- The hierarchical structure significantly improved robustness by resolving "coarse-grained" ambiguities before attempting fine-grained classification.
- The ensemble consistently outperformed the strongest single model (GPT-4.1) across all metrics.
- The method proved effective even with a small calibration set (200 samples) and no fine-tuning.

5. Significance and Future Directions

Practicality for Open-World Analysis: The framework offers a scalable solution for real-world malware analysis where labeled data is scarce and malware evolves rapidly. It removes the dependency on expensive dynamic sandboxes.
Interpretability: The hierarchical structure aligns with human analyst reasoning (behavior first, then family), making the decision process more transparent than "black-box" deep learning models.
Limitations & Future Work:
- Current reliance on static code inputs limits detection of runtime-only behaviors.
- The hierarchy is manually defined; future work aims to induce data-driven hierarchies.
- Evaluation is currently limited to a small gold set; scaling to larger corpora is planned.

Conclusion: The paper establishes that weighted hierarchical ensembling of LLMs is a superior approach for zero-label malware family classification, offering a robust, interpretable, and highly accurate alternative to traditional supervised learning methods in dynamic threat environments.