🤖 machine learning

Constructing Multi-label Hierarchical Classification Models for MITRE ATT&CK Text Tagging

This paper introduces a multi-label hierarchical classification framework for automating MITRE ATT&CK text tagging that achieves high accuracy (94% at the tactic level and 82% at the technique level) using classical machine learning methods, significantly outperforming GPT-4o while eliminating the need for complex LLM-based architectures.

Original authors: Andrew Crossman, Jonah Dodd, Viralam Ramamurthy Chaithanya Kumar, Riyaz Mohammed, Andrew R. Plummer, Chandra Sekharudu, Deepak Warrier, Mohammad Yekrangian

Published 2026-01-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andrew Crossman, Jonah Dodd, Viralam Ramamurthy Chaithanya Kumar, Riyaz Mohammed, Andrew R. Plummer, Chandra Sekharudu, Deepak Warrier, Mohammad Yekrangian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to organize a massive, chaotic library of stories about cyber-attacks. Every time a new story (a "threat report") comes in, you need to file it under specific categories so other librarians can find it later.

In the world of cybersecurity, this filing system is called MITRE ATT&CK. It's like a giant, two-level filing cabinet:

The Top Drawer (Tactics): This answers the question, "Why did the bad guy do this?" (e.g., "They wanted to steal passwords" or "They wanted to hide their tracks").
The Bottom Drawer (Techniques): This answers the question, "How did they do it?" (e.g., "They used a specific type of phishing email" or "They used a specific software glitch").

For years, security experts have had to read these reports and manually file them into the correct drawers. It's slow, boring, and prone to human error. This paper is about building a robot assistant to do the filing for them.

The "Bottom-Up" Construction

Instead of trying to build a giant, complex robot from scratch (a "top-down" approach), the authors built their system step-by-step, like stacking blocks. They started small and added complexity only when they knew the previous block was solid.

Step 1: The Simple Classifier (The "One-Label" Robot)
First, they taught a simple machine learning model to look at a sentence and guess one "Why" (Tactic). They tested this against GPT-4o, the famous AI chatbot.
- The Result: The simple robot won. It got about 82% of the answers right, while GPT-4o only got 59% right. The authors found that for this specific, structured task, a simple, classical tool was actually better than the fancy, complex AI.
Step 2: The Multi-Label Classifier (The "Multiple-Label" Robot)
Real life isn't simple; a single sentence often has multiple reasons for an attack. So, they upgraded the robot to guess the top 3 "Whys" instead of just one.
- The Result: This boosted their accuracy to 94%.
Step 3: The Hierarchical Classifier (The "Full Filing" Robot)
Finally, they taught the robot to guess the "Why" and the "How" together. If the robot guesses the "Why" is "Stealing Passwords," it then looks for the specific "How" (e.g., "Keylogging").
- The Result: This system achieved 82% accuracy for the full "Why + How" combination.

The "Secret Sauce" (Why it works)

The authors didn't use the latest, most expensive "Generative AI" or complex neural networks. Instead, they used classical machine learning (specifically something called "Stochastic Gradient Descent" with "TF-IDF").

Think of it like this: Instead of building a super-complex, expensive Ferrari to drive to the grocery store, they built a very reliable, efficient bicycle. It gets the job done faster, costs less, and is easier to fix.

They also added a privacy lock (hashing). Imagine they wanted to share their robot with the world, but they were worried about sharing the secret recipes (the data) used to train it. They figured out a way to scramble the data so the robot could learn from it without anyone being able to see the original ingredients. This allowed them to release the robot to the public safely.

The "New Library" Test

To see if their robot was truly smart, they didn't just test it on the stories it learned from. They gave it a completely new set of stories about financial threats (specific to banking) that it had never seen before.

The Result: At first, the robot was confused (because the new stories were different). But, when they gave it just a tiny bit of training on these new stories, it quickly learned to file them correctly. This proves the robot is flexible and can adapt to new situations without needing a massive amount of new data.

The Big Takeaway

The paper concludes that you don't always need the most expensive, complex AI to solve a problem. By breaking the task down into small, manageable steps and using reliable, classical tools, they built a system that:

Is more accurate than a top-tier AI chatbot (GPT-4o) for this specific job.
Is faster and cheaper to run.
Is safe to share with the public.
Can adapt to new types of data with very little extra training.

They have made this "robot librarian" available for free on GitHub so other security teams can use it to organize their own cyber-threat reports.

Technical Summary: Constructing Multi-Label Hierarchical Classification Models for MITRE ATT&CK Text Tagging

Problem Statement

The MITRE ATT&CK framework serves as a critical knowledge base for cybersecurity, organizing threat actor behaviors into a hierarchy of tactics (the "why" or goals of an attack) and techniques (the "how" or methods used). A primary application of this framework is the manual annotation of cyber-threat intelligence reports, vulnerability descriptions, and threat scenarios to facilitate downstream analysis. Currently, this tagging process remains largely manual, creating a bottleneck for security analysts. While various computational methods have emerged to automate or semi-automate this task, there is a lack of a unified framework to categorize these efforts and a need for robust, accessible models that do not rely on complex Large Language Models (LLMs) or resource-intensive architectures.

Methodology

The authors propose a "bottom-up" stage-wise construction of a multi-label hierarchical classification system, diverging from canonical "top-down" AI/ML architectural selections. This approach is guided by a stratified "task space" formulation (Table 1) that categorizes tagging tasks by complexity, ranging from single-label multiclass classification to complex multi-label hierarchical mapping.

The methodology proceeds through three experimental stages using classical machine learning (specifically Stochastic Gradient Descent (SGD) classifiers with TF-IDF vectorization):

Experimental Stage 1 (Multiclass Tactic Classification):
- Task: Map a sentence ( $D$ ) to a single tactic ( $T$ ).
- Approach: A baseline multiclass SGD model was trained on 14,405 general cyber-intelligence sentences.
- Comparison: The model was benchmarked against GPT-4o using a Text-to-Text prompt engineering approach.
- Result: The classical SGD model significantly outperformed GPT-4o (81.95% vs. 59% accuracy).
Experimental Stage 2 (Multi-Label Hierarchical Construction):
- Task: Extend the system to predict multiple tactics and their corresponding techniques (Task ID 7 in the task space).
- Approach:
  - Top-N Prediction: The multiclass model was modified to output the top $n=3$ predicted tactics rather than a single label.
  - Hierarchical Conditioning: For each predicted tactic, a separate, tactic-specific multiclass SGD model was trained to predict the top $m=3$ associated techniques.
  - Security: A hashing mechanism (MurmurHash3) was integrated into the vectorization process to encrypt sensitive training data without compromising model performance, facilitating public release.
- Result: The system achieved approximately 94% accuracy at the tactic level and 82% accuracy at the technique level (when cast as multiclass transformations).
Experimental Stage 3 (Generalization to New Domains):
- Task: Evaluate the baseline model on a sparse dataset of 552 financial threat scenarios produced by JPMorgan Chase subject matter experts.
- Approach: The model trained on general cyber-intelligence data was tested on this new domain, and subsequently, a new model was trained from scratch on a small subset of the threat scenario data.
- Result: The baseline model showed limited generalization (41% accuracy), but the architecture proved adaptable, improving to 66% accuracy with minimal retraining on the new domain data.

Key Contributions

Stratified Task Space Formulation: The paper provides a structured taxonomy (Table 1) for the MITRE ATT&CK text tagging task, organizing existing AIML approaches and clarifying pathways for future development.
Bottom-Up Model Construction: The authors demonstrate a methodology for building hierarchical models that organically encapsulates CISA's "Best Practices for MITRE ATT&CK Mapping" through experimentation rather than pre-defined complex architectures.
Performance vs. Complexity: The study establishes that classical machine learning methods (SGD + TF-IDF) can meet or surpass state-of-the-art performance (including GPT-4o) for this specific task, removing dependence on LLMs, RAG, agents, or complex DAG-based methods.
Public Release and Security: The authors released a shareable, secure version of their tagging system (via GitHub) that includes data hashing to protect sensitive training data while maintaining high performance.
Domain Adaptation Example: The paper illustrates how the baseline model can be bootstrapped for new, sparse datasets (financial threat scenarios), highlighting the potential for low-resource model building.

Results

Tactic Level Accuracy: The multi-label hierarchical approach achieved roughly 94% accuracy at the tactic level.
Technique Level Accuracy: The approach achieved roughly 82% accuracy at the technique level.
GPT-4o Comparison: GPT-4o achieved only ~60% accuracy at the tactic level, significantly lower than the classical SGD approach.
Hierarchical Merit: The results confirmed that the hierarchical approach is effective; techniques were never predicted correctly when paired with an incorrect tactic, validating the conditional modeling strategy.
Data Security: The integration of hashing for data encryption did not significantly impact system performance, enabling secure public distribution.

Significance and Claims

The paper claims that the cybersecurity community faces a gap between specialized security activities and the rich mathematical/technical literature in AIML. By providing a rigorous task formulation and a high-performing, accessible, and secure baseline model, the authors aim to bridge this gap.

The authors emphasize that while many approaches exist, widespread adoption by security specialists is rare, often due to usability issues in low-resource or sparse-data settings. Consequently, the significance of this work lies not just in the performance metrics, but in the usability, shareability, and extensibility of the proposed system. The paper posits that a straightforward, classical machine learning approach that respects data privacy and requires minimal resources is a prerequisite for the actual adoption of automated tagging tools by security professionals. The authors explicitly state that their goal is to facilitate further developments and provide a foundation for future work, rather than claiming a final, perfect solution.