🤖 machine learning

Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports

This paper establishes a new empirical baseline for multi-label ATT&CK classification on complex unstructured Cyber Threat Intelligence reports by introducing a human-annotated dataset of 2,076 sentences and evaluating seven open-source LLMs, which collectively achieved a low micro-averaged F1 score of 0.22, indicating that current models remain insufficient for production-grade deployment despite a positive correlation between parameter size and performance.

Original authors: Ahmed Ryan, Saad Sakib Noor, Md Erfan, Shaswata Mitra, Sudip Mittal, Md Rayhanur Rahman

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Ryan, Saad Sakib Noor, Md Erfan, Shaswata Mitra, Sudip Mittal, Md Rayhanur Rahman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard trying to read a thousand-page, messy diary written by a chaotic criminal. This diary (called a CTI Report) doesn't just say "I stole the car." It says, "I picked the lock, hotwired the engine, drove to the bank, and then wiped the security cameras."

Your job is to read every sentence and tag it with the correct "crime code" from a massive rulebook called MITRE ATT&CK. This rulebook has 211 different codes for every possible way a criminal can attack. The problem? One sentence often contains multiple crimes at once, and the language is tricky.

For a long time, computers were terrible at this. They were like a robot that could only read simple, single-sentence notes. If the note was complex, the robot got confused.

Then came Large Language Models (LLMs). Think of these as super-smart, well-read interns who can understand context and nuance. People hoped these interns could finally read the messy diaries and tag the crimes perfectly.

This paper is the report card for seven of these open-source interns. The researchers wanted to see if these AI interns are actually ready to do the job, or if they are still too inexperienced.

Here is what they did and what they found, explained simply:

1. The Test: A "Real-World" Exam

Previous tests for these AI interns were like giving them a quiz with easy, single-answer questions (e.g., "The criminal used a key." -> Tag: Lock Picking).

The researchers in this paper said, "That's not how real life works." They created a harder, more realistic exam:

They took 83 real, messy security reports written by human experts.
They broke them down into 2,076 individual sentences.
They had human experts manually tag every sentence with the correct crime codes. Some sentences had no codes; some had one; some had up to nine codes at once!
This became the "Answer Key" (Ground Truth) to grade the AI.

2. The Contestants

They picked 7 different open-source AI models.

Think of these models as students of different sizes. Some are small (8 billion "brain cells" or parameters), and some are giants (236 billion).
They tested them in different "modes":
- Zero-Shot: Just giving the instructions.
- Few-Shot: Giving the AI a few examples to study first.
- Chain-of-Thought: Asking the AI to "think out loud" step-by-step before answering.
- Temperature: Turning the AI's "creativity" knob up or down (0 = strict robot, 0.5 = slightly more creative).

3. The Results: The Interns Are Still Learning

The results were a bit of a reality check.

The Score: Even the best AI model only got a 22% score (on a scale where 100% is perfect).
- Analogy: Imagine a student taking a difficult final exam and getting a D-. They are passing, but they are missing most of the answers.
Size Matters (Mostly): The biggest, most complex models (the "Giant" ones) generally did better than the small ones. There was a clear link: bigger brain = slightly better score.
The "Magic" Settings Didn't Work:
- Giving the AI examples (Few-Shot) didn't really help.
- Asking the AI to "think step-by-step" (Chain-of-Thought) didn't really help.
- Changing the "creativity" knob (Temperature) made almost no difference.
- Analogy: It's like giving a student a different colored pencil or telling them to "try harder," but if they don't know the material, the score doesn't change.

4. The One Thing That Helped: The "Cheat Sheet"

The researchers tried one more thing. They gave the AI a Retrieval-Augmented Generation (RAG) setup.

Analogy: Instead of just relying on what the AI memorized, they gave the AI an open book (the official rulebook) to look at while taking the test.
The Result: The score jumped from 22% to 32%.
This was the biggest improvement of all. It proved that the AI isn't failing because it's "dumb"; it's failing because it doesn't have the current rules in front of it.

5. What Went Wrong?

The researchers looked at the mistakes the AI made:

False Positives: The AI saw a word like "evade" and guessed "Hijacking," even though the sentence was about something else. It was guessing based on keywords rather than understanding the story.
False Negatives: The AI missed a crime because it didn't understand that "staging a ransomware binary" was actually a specific type of transfer. It lacked the deep context.

The Bottom Line

The paper concludes that current open-source AI models are not yet ready to be the sole security guards for classifying complex cyber threats. They are too inaccurate (scoring only 22-32%) to be trusted in a real-world security center without human help.

The only thing that significantly improved their performance was giving them access to the official rulebook (RAG) during the test. Simply changing how you ask the questions (prompts) or making the model bigger didn't solve the core problem.

In short: The AI interns are smart, but they are currently too error-prone to work alone. They need a human supervisor and a reference manual to do the job correctly.

Technical Summary: Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification

Problem Statement

Classifying Cyber Threat Intelligence (CTI) reports into MITRE ATT&CK techniques is critical for proactive defense but remains a labor-intensive process due to the complexity of unstructured text, multi-step attack patterns, and the 211-class taxonomy. While Large Language Models (LLMs) have shown promise in understanding context, existing evaluations suffer from significant limitations: they rely on simplified, single-technique sentences that do not reflect the multi-label nature of real-world reports, leading to inflated performance metrics. Consequently, the baseline performance of open-source LLMs on complex, unstructured CTI reports remains unevaluated, leaving security practitioners without empirical evidence to guide the integration of these models into threat ingestion pipelines.

Methodology

To address this gap, the authors constructed a rigorous experimental framework spanning dataset creation, model evaluation, and statistical analysis.

1. Dataset Construction (RQ1)
The authors curated a ground-truth dataset comprising 2,076 sentences extracted from the "Case Summary" sections of 83 complex unstructured CTI reports (April 2020–February 2025) sourced from The DFIR Report.

Annotation Process: A six-phase annotation strategy was employed involving independent labeling, cross-labeling, relabeling of low-agreement instances, and dispute resolution.
Scope: Sentences were mapped to 114 unique ATT&CK techniques (from the 211 available in Enterprise v17.1). The dataset includes both technique-positive (1,281 sentences) and negative (795 sentences) classes to evaluate false positive rates.
Quality Control: The process achieved a mean inter-annotator agreement of $\kappa = 0.68$ (substantial agreement), with 65 techniques exceeding the $\kappa \geq 0.7$ threshold.
Characteristics: The dataset exhibits severe class imbalance (Imbalance Ratio = 229.0) and a long-tail distribution, with 40.2% of labeled sentences containing multiple techniques (mean cardinality = 1.58).

2. Model Evaluation (RQ2)
Seven open-source LLMs ranging from 8B to 236B parameters (including Llama 3.1, Gemma 3, DeepSeek-V2.5, and GPT-OSS variants) were evaluated.

Experimental Design: A full factorial design tested 53 inference configurations combining:
- Prompt Strategy: Zero-Shot vs. Three-Shot (N=3).
- Reasoning: Chain-of-Thought (CoT) vs. No CoT.
- Temperature: 0.0 vs. 0.5.
RAG Baseline: A Retrieval-Augmented Generation (RAG) pipeline was implemented using a FAISS vector database of ATT&CK documentation to ground the LLM70 model in external knowledge, comparing it against standard inference.
Metrics: Micro-averaged Precision, Recall, and F1 scores were used as primary metrics to reflect operational coverage across all technique instances.

3. Statistical Analysis (RQ3)
Spearman's rank correlation ( $\rho$ ) was applied to assess the monotonic relationships between inference settings (parameter size, prompt strategy, reasoning, temperature) and classification performance.

Key Contributions

Dataset: A publicly available, human-annotated ground-truth dataset of 2,076 sentences from 83 unstructured reports, mapped to 114 ATT&CK techniques, representing the largest dataset of its kind to date.
Performance Benchmark: A systematic evaluation establishing an empirical baseline for multi-label ATT&CK classification, revealing that current open-source LLMs struggle significantly with this task.
Statistical Analysis: A correlation study quantifying the impact of model size and inference hyperparameters, identifying parameter size as the only statistically significant predictor of performance.

Results

Baseline Performance: The highest-performing non-RAG configuration (DS236, 236B parameters, T=0.0, 3-shot, CoT) achieved a micro-averaged F1 score of only 0.22.
Model Variance: Performance varied significantly by model size and architecture. The 236B model (DS236) and 70B model (LLM70) formed the top cluster (F1 0.20–0.22), while smaller models (e.g., GPT20, 20B) performed near random chance (F1 $\leq$ 0.03).
Hyperparameter Impact:
- Parameter Size: Showed a statistically significant positive correlation with F1 score ( $\rho = 0.85, p = 0.014$ ).
- Prompt Strategy & Reasoning: Neither Few-Shot prompting nor Chain-of-Thought reasoning produced statistically significant gains across the model configurations ( $p > 0.05$ ).
- Temperature: Variations between 0.0 and 0.5 had no statistically significant effect on performance.
RAG Impact: Implementing RAG with LLM70 increased the micro F1 score to 0.32 (a $\Delta$ of +0.10), primarily driven by a substantial increase in recall (from 0.23 to 0.41). This improvement was attributed to external knowledge grounding rather than prompt tuning.
Error Analysis: Failures were categorized into Type I (false positives due to keyword over-reliance or hallucination) and Type II (false negatives due to lack of domain knowledge or inability to synthesize multi-stage behaviors).

Significance and Claims

The paper claims to provide the first empirical baseline for open-source LLMs on complex, multi-label CTI classification. The authors assert that current open-source LLMs are insufficient for production-grade ATT&CK classification, as evidenced by the low F1 scores (max 0.22) compared to prior specialized tools that often report higher scores on simplified datasets.

The study concludes that prompt engineering strategies (N-shot, CoT, temperature) are not effective levers for closing the performance gap in this domain. Instead, the results indicate that parameter size is a critical factor, and retrieval grounding (RAG) is the most productive direction for improvement, offering a significant boost in recall. The authors position their dataset and findings as a reproducible foundation for future research, suggesting that future work should focus on hierarchical RAG, parameter-efficient fine-tuning, and hierarchy-aware metrics rather than further prompt tuning.