🤖 AI

From Theory to Practice: Code Generation Using LLMs for CAPEC and CWE Frameworks

This paper introduces a novel dataset of 615 vulnerable code snippets across Java, Python, and JavaScript, generated using GPT-4o, Llama, and Claude models to map Common Attack Pattern Enumerations (CAPEC) and Common Weakness Enumerations (CWE) descriptions, thereby addressing the lack of detailed, labeled vulnerability data for advancing security research and machine learning-based detection systems.

Original authors: Murtuza Shahzad, Joseph Wilson, Ibrahim Al Azher, Hamed Alhoori, Mona Rahimi

Published 2026-04-06

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Murtuza Shahzad, Joseph Wilson, Ibrahim Al Azher, Hamed Alhoori, Mona Rahimi

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to fix a leaky faucet, but the only instruction manual you have is a dry, technical list of parts and pressure ratings. It tells you what is broken, but it doesn't show you how the water actually sprays out or what the broken part looks like in real life. You're left guessing.

This is exactly the problem the researchers at Northern Illinois University faced with software security.

The Problem: A Manual Without Pictures

In the world of cybersecurity, there are two massive "encyclopedias" maintained by a group called MITRE:

CAPEC: A list of all the different ways hackers can attack software (like "SQL Injection" or "Cross-Site Scripting").
CWE: A list of the specific weak spots in code that allow those attacks to happen.

The problem? These encyclopedias are full of definitions, but they are missing the "how-to" videos. They describe the attack in theory, but they rarely provide actual lines of code showing what a vulnerable program looks like. It's like having a dictionary of words but no sentences to show how they are used.

Because of this, developers, students, and security experts struggle to truly understand how these attacks work in the real world.

The Solution: Hiring a Super-Intelligent Robot Chef

To fix this, the researchers decided to use Large Language Models (LLMs)—the same kind of AI that powers tools like ChatGPT. Think of these AIs as super-intelligent robot chefs who have read every cookbook (code repository) in the world.

Here is how they used the robots:

The Recipe (The Prompt): The researchers didn't just ask the AI, "Write bad code." That would be like asking a chef to "make a mess." Instead, they gave the AI a very specific recipe. They fed it the description of the attack (from CAPEC) and the list of weak spots (from CWE).
The Ingredients (Context): Sometimes, the attack description wasn't detailed enough. So, the researchers used a smart tool (SBERT) to find the 5 most closely related "weaknesses" to add to the recipe. This ensured the AI had enough context to cook up a perfect example.
The Cooking (Generation): The AI (specifically GPT-4o, along with Llama and Claude) then generated actual code snippets in Java, Python, and JavaScript.
- It wrote code that looked normal but contained a hidden trap (a vulnerability).
- It added comments explaining exactly where the trap was and why it was dangerous.
- It wrote a summary explaining the "story" of the attack.

The Result: A New "Cookbook" for Security

The team created a massive new dataset containing 615 different examples of vulnerable code.

To make sure the robots did a good job, the researchers acted as food critics:

Did it compile? (Could the code actually run, or was it gibberish?) Yes, about 90% of the time.
Was it relevant? (Did the code actually show the attack described?) Yes, experts agreed on this almost 100% of the time.
Was it readable? (Was it easy to understand?) Yes, the code was rated very highly for clarity.

They even checked if the robots were consistent. If they asked the same robot to cook the same dish five times, did they get the same result? Yes. The code was 98-99% identical every time, proving the AI wasn't just guessing randomly.

Why This Matters

This new dataset is like opening a security training gym.

For Students: Instead of just reading about a "lockpick," they can see a lockpick in action and learn how to spot it.
For Developers: They can see exactly what a "bad habit" looks like in their own code so they can avoid making the same mistake.
For AI Researchers: They can use this dataset to train other AI tools to automatically find and fix these security holes in the future.

The Catch (Limitations)

The researchers admit the robot chefs aren't perfect yet.

Sometimes they tried to write code for attacks that don't involve software at all (like "bypassing a physical lock"), which is a bit silly for a coding robot.
They only used three programming languages, though the world uses many more.

The Bottom Line

This paper is about bridging the gap between theory and practice. By using AI to generate real-world examples of security flaws, the researchers have created a powerful new tool. It turns dry, boring definitions into living, breathing examples that help us build safer software for everyone.

In short: They taught the AI to show us the "bugs" in the system so we can learn how to squash them before the hackers do.

1. Problem Statement

The paper addresses a critical gap in cybersecurity research and education: the lack of practical, code-level examples corresponding to theoretical vulnerability descriptions.

The Gap: While the MITRE frameworks CAPEC (Common Attack Pattern Enumeration and Classifications) and CWE (Common Weakness Enumeration) provide comprehensive catalogs of attack patterns and weaknesses, they often lack corresponding code snippets.
The Statistics: An analysis of CAPEC v3.9 and CWE v4.14 revealed that approximately 40% of CAPECs and 47% of CWEs lack code examples. Furthermore, existing code availability is heavily skewed toward Java, with extremely low coverage for JavaScript (≈0.5% of CWEs) and Python (≈2% of CWEs).
The Consequence: This scarcity hinders the development of advanced vulnerability detection tools, limits the training of machine learning (ML) models, and makes it difficult for educators and developers to understand how theoretical vulnerabilities manifest in real-world code.

2. Methodology

The authors propose a novel pipeline to automatically generate vulnerable code snippets for every CAPEC entry using Large Language Models (LLMs). The process involves three main stages:

A. Data Enrichment and Mapping (CAPEC $\to$ CWE)

Challenge: Some CAPEC entries have few or no associated CWEs in the official MITRE data, leading to insufficient context for LLMs.
Solution: The authors use SBERT (Sentence-BERT) to find semantically similar CWEs for CAPECs with sparse mappings.
Strategy:
- If a CAPEC has $\ge$ 5 related CWEs, all are used.
- If a CAPEC has $< 5$ related CWEs, the system adds the top semantically similar CWEs (via SBERT) until a total of 5 is reached.
- This ensures a consistent, context-rich input (CAPEC description + 5 CWE descriptions) for the LLM.

B. Prompt Engineering and Code Generation

Model Selection: The primary model used is GPT-4o. The study also utilized Llama 3 (70B) and Claude-3.5-Sonnet for consistency checks.
Framework: The pipeline uses LangChain to manage the interaction between the dataset and the LLM.
Prompt Design: A structured prompt instructs the model to act as an expert, generating a concise code snippet in Java, Python, or JavaScript that embodies the specific CAPEC attack pattern. The prompt requires:
1. A code snippet.
2. A functional description.
3. Inline comments explaining the vulnerability and mapping specific lines to CWE IDs.
4. Output in JSON format.

C. Dataset Construction

The final dataset comprises 615 CAPEC code snippets across three programming languages (Java, Python, JavaScript), creating one of the most extensive resources of its kind.

3. Key Contributions

Novel Dataset: Creation of a large-scale, diverse dataset linking CAPEC attack patterns to executable vulnerable code in three major languages.
Semantic Mapping Methodology: A robust approach using SBERT to augment CAPEC-CWE mappings, ensuring high-quality context for LLMs even when official data is sparse.
Cross-Model Validation: Demonstration that multiple state-of-the-art LLMs (GPT-4o, Llama, Claude) can consistently generate high-quality, semantically aligned vulnerability code.
Resource for ML and Education: Providing a structured resource specifically designed to train ML models for automatic vulnerability detection and to serve as educational material for secure coding.

4. Results and Evaluation

The authors conducted a rigorous evaluation focusing on compilability, relevance, readability, and consistency.

Compilability:
- Approximately 90% of the generated code snippets compiled successfully.
- Python showed the highest success rate, followed by Java and JavaScript.
Relevance to CAPEC:
- Four independent reviewers (Ph.D. students) evaluated 30 random samples per language.
- 100% of the samples were deemed relevant to their corresponding CAPEC descriptions by at least three evaluators.
- Inter-Rater Agreement: Average Pairwise Agreement (APA) scores were high: 95.83% (Java), 93.33% (Python), and 96.67% (JavaScript).
Readability:
- Code was rated on a 1–5 scale. Python code received the highest scores (mostly 5/5), indicating high clarity and structure.
Consistency (Cosine Similarity):
- Using CodeBERT, the authors measured the semantic similarity between code generated by the same model (across 5 datasets) and different models.
- Intra-model consistency: GPT-4o showed cosine similarities $>0.99$ for Java/Python and $>0.98$ for JavaScript.
- Inter-model consistency: High alignment was observed between GPT-4o, Llama, and Claude, with cosine similarities generally exceeding 0.98, confirming that the generated vulnerability patterns are robust and model-agnostic.

5. Significance and Future Work

Significance: This work bridges the gap between theoretical vulnerability catalogs and practical implementation. It enables the training of more accurate ML-based vulnerability detectors and provides a standardized benchmark for evaluating code security tools.
Limitations:
- Some generated code corresponds to non-software CAPECs (e.g., physical locks), which are irrelevant for code analysis.
- The fixed threshold of 5 CWEs may not be optimal for all cases.
Future Directions:
- Implementing a pre-filtering mechanism to exclude non-software CAPECs.
- Expanding the dataset to include more languages (e.g., C++, Go).
- Integrating the dataset into real-world vulnerability scanning and remediation tools.
- Exploring more advanced LLMs (e.g., Claude-3 Opus) for potentially higher fidelity.

In conclusion, the paper demonstrates that LLMs, when guided by structured prompts and enriched semantic context, can reliably generate high-quality, educational, and research-grade vulnerable code, significantly advancing the field of software security.