💻 computer science

Towards Automated Pentesting with Large Language Models

This paper introduces RedShell, a privacy-preserving and hardware-efficient framework that leverages fine-tuned Large Language Models to automate the generation of valid and semantically aligned offensive PowerShell code for ethical pentesting on Microsoft Windows systems.

Original authors: Ricardo Bessa, Rui Claro, João Trindade, João Lourenço

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ricardo Bessa, Rui Claro, João Trindade, João Lourenço

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard for a massive, high-tech castle (a computer network). Your job is to find the weak spots in the walls before the bad guys do. This job is called Penetration Testing (or "Pentesting").

Usually, to find these weak spots, you have to write complex, dangerous "keys" (code) that can pick specific locks. Writing these keys by hand is slow, difficult, and requires years of training.

Now, imagine someone invented a super-smart robot assistant (an LLM, or Large Language Model) that can write these keys for you. But there's a catch: most of these robots are "good citizens." They have strict rules that say, "I cannot write a key that opens a door to a bank vault, even if you ask me nicely."

This paper introduces RedShell, a special, custom-built robot designed to help the security guards (ethical hackers) do their job faster and better, without breaking the rules.

Here is the story of RedShell, broken down simply:

1. The Problem: The "Good" Robots Won't Help

The authors noticed that while "good" robots (like ChatGPT) are great at writing stories or emails, they refuse to write the specific, dangerous code needed to test security. Meanwhile, bad guys are using "dark" robots (like WolfGPT) to write malicious code to attack people.

Security experts needed a robot that could write these "bad" codes on purpose, but only inside a safe, controlled lab to test defenses.

2. The Solution: Building RedShell

The team built RedShell. Think of RedShell not as a new robot from scratch, but as a specialized training camp for an existing smart robot.

The Student: They picked a very smart, open-source robot (called Qwen2.5-Coder) that already knew how to speak many programming languages.
The Textbook: They created a special textbook. It contained thousands of examples of "bad" PowerShell code (a language Windows computers use) paired with simple instructions like, "Write a code to check who is logged in."
The Training: Instead of teaching the robot from the beginning (which takes supercomputers and huge electricity bills), they used a technique called Fine-Tuning.
- Analogy: Imagine a master chef who already knows how to cook everything. Instead of teaching them how to hold a knife, you just give them a specific recipe book for "Spicy Dishes" and say, "Now, practice making these specific spicy dishes." The chef learns the specific style very quickly without needing a new kitchen.

3. Why RedShell is Special

The paper highlights three superpowers of RedShell:

Privacy (The Secret Lab): Most big robots live in the cloud. If you ask them to test your company's security, you have to send your secret data to their servers. RedShell runs locally on your own computer. It's like doing your training in your own garage; no one else sees your notes or your test results.
Efficiency (The Lightweight Box): Other robots need a warehouse full of expensive graphics cards to learn. RedShell was trained on a single, standard graphics card (like the ones in high-end gaming PCs). It's like training a marathon runner in a small park instead of a massive stadium.
The "Bad" Guy Persona: The robot was taught to act like a hacker. When you ask it, "How do I bypass this firewall?" it doesn't say, "I can't do that." It says, "Here is the code to test that firewall."

4. Did It Work? (The Exam Results)

The team put RedShell through a series of tests to see if it was any good.

Grammar Check (Syntax): They checked if the code it wrote actually made sense. Result: 90% of the time, the code was perfect. It didn't have typos that would crash the computer.
Meaning Check (Semantics): They compared RedShell's code to code written by human experts. Result: RedShell's code was very similar to the human experts' code (over 50% similarity), beating other famous robots that were trained on much larger, more expensive datasets.
The Real Test (Functional): They actually ran the code in a fake, safe environment to see if it did what it was supposed to do. Result: RedShell successfully performed the "attacks" (like checking for open doors or stealing fake passwords) just like a human expert would.

5. The Ethical Twist

The authors are very careful here. They admit that this technology is dangerous. If a bad guy gets RedShell, they could use it to hack real people.

However, they argue that you have to know how the thief thinks to catch the thief. By giving security experts a tool to simulate attacks, they can find the holes in their own walls before the bad guys find them. They also made sure their training data didn't include instructions on how to destroy real systems, keeping the focus on "testing" rather than "destroying."

The Big Picture

This paper is essentially saying: "Artificial Intelligence is a powerful tool. If we hide it or ban it, bad guys will still find it. But if we build safe, specialized versions of it for the good guys, we can make our digital castles much stronger."

RedShell is that specialized tool: a privacy-focused, efficient, and highly skilled assistant that helps security experts stay one step ahead of the hackers.

1. Problem Statement

The paper addresses the growing challenge in offensive cybersecurity where attackers utilize "Dark LLMs" (e.g., WolfGPT, XXXGPT) to generate malicious code with minimal human intervention. Conversely, ethical hackers (pentesters) face significant bottlenecks in traditional penetration testing:

Manual Effort: Writing offensive code (specifically PowerShell for Windows) requires high expertise, time, and effort.
Data Scarcity: There is a lack of high-quality, publicly available datasets for training models on offensive code, particularly compared to general-purpose languages like Python or Java.
Privacy & Resource Constraints: Existing solutions often rely on proprietary, closed-source LLMs (like ChatGPT), which pose data privacy risks (sending sensitive target info to external servers) and have ethical guardrails that block malicious code generation. Furthermore, pre-training large models is computationally expensive.

2. Methodology

The authors propose RedShell, a privacy-preserving, hardware-efficient framework designed to automate the generation of offensive PowerShell code for Microsoft Windows environments.

A. Dataset Construction (Ground Truth)

Reference Dataset: The authors started with an existing corpus of 1,127 malicious PowerShell snippets paired with natural language descriptions (from Liguori et al.).
Extended Dataset: To overcome size and diversity limitations, the authors manually curated and added 1,135 new samples, doubling the dataset size.
- Sources: Added samples from frameworks like Nishang, PowerUpSQL, MicroBurst, and CTF walkthroughs.
- Annotation: Samples were classified according to the MITRE ATT&CK framework (covering 14 tactics), significantly improving coverage in areas like Discovery, Defense Evasion, and Credential Access.
Split: The final dataset was split 90/10 for training and testing.

B. Model Selection & Fine-Tuning

Models: Three open-weight, state-of-the-art models were selected based on coding proficiency and size (<8B parameters):
1. Qwen2.5-7B (General purpose)
2. Qwen2.5-Coder-7B-Instruct (Code-specific)
3. Llama3.1-8B
Training Strategy:
- Hardware: Single NVIDIA GeForce RTX 4090 (24GB VRAM).
- Technique: Used LoRA (Low-Rank Adaptation) via the Unsloth framework to enable efficient fine-tuning without full model retraining.
- Configuration: Batch size of 8, learning rate $2 \times 10^{-4}$ , and a specific system prompt instructing the model to act as a "malicious PowerShell generator" without explanations.
- Privacy: All training and inference occur locally, ensuring no sensitive pentest data leaves the environment.

C. Evaluation Framework

The framework was evaluated across three dimensions:

Syntactic Validity: Using PSScriptAnalyzer to check for parse errors, warnings, and security rule violations.
Semantic Similarity: Comparing generated code against reference snippets using metrics: ROUGE-L, METEOR, BLEU, Edit Distance (ED), and Exact Match (EM).
Functional Reliability: A controlled "Red Team" simulation involving two VMs (attacker and target). The model was tasked with generating payloads for 10 specific attack stages (e.g., port scanning, credential dumping, privilege escalation) to verify if the code actually executed the intended malicious behavior.

3. Key Contributions

RedShell Framework: A modular, lightweight framework for fine-tuning LLMs specifically for offensive PowerShell generation, adaptable to other languages (e.g., Assembly).
Enhanced Dataset: The creation of an extended, manually curated dataset of 2,262 malicious PowerShell samples annotated with MITRE ATT&CK tactics, addressing a critical data gap in the field.
Hardware-Efficient Training: Demonstrated that high-quality offensive code generation can be achieved using a single consumer-grade GPU and LoRA fine-tuning, avoiding the massive resource costs of pre-training or full fine-tuning.
Privacy-Centric Approach: A solution that eliminates the need to send sensitive audit data to third-party API providers.

4. Results

Syntactic Performance:
- RedShell models achieved >90% syntactic validity (successfully parsed without errors).
- Qwen2.5-Coder performed best with >95% parsing accuracy.
- The models successfully generated code that adhered to offensive requirements, often triggering safety warnings (e.g., Invoke-Expression usage) which is expected for malicious payloads.
Semantic Performance:
- RedShell outperformed state-of-the-art reference models (CodeGPT, CodeGen, CodeT5+) and proprietary models (ChatGPT-3.5, DeepSeek) across all similarity metrics.
- Edit Distance: Achieved >50% average code similarity with reference snippets.
- Qwen2.5-Coder achieved the highest scores, demonstrating that specialized fine-tuning on a smaller, targeted dataset outperforms larger, generic pre-trained models in this specific domain.
Functional Performance:
- In the simulated pentest scenario, RedShell's Qwen2.5-Coder achieved 100% correctness (adherence to offensive methodologies) and high effectiveness.
- It successfully executed multi-step attacks (e.g., downloading scripts to RAM, brute-forcing credentials, using Mimikatz) where proprietary models often failed due to ethical guardrails or lack of specific domain knowledge.

5. Significance

Automation of Red Teaming: RedShell proves that LLMs can significantly automate the tedious aspects of penetration testing, allowing ethical hackers to focus on strategy rather than syntax.
Democratization of Offensive AI: By showing that high-performance offensive models can be trained on a single GPU with open weights, the paper lowers the barrier to entry for creating specialized security tools.
Defensive Insight: The paper highlights that "Dark LLMs" are a tangible threat. By developing and testing RedShell in controlled environments, defenders can better understand how attackers might leverage AI to automate exploits, thereby improving defensive strategies.
Ethical Balance: The authors emphasize that while the tool generates malicious code, it is designed for non-destructive testing. They argue that ethical hackers must possess similar tools to simulate and defend against the inevitable rise of AI-driven attacks.

In conclusion, RedShell represents a significant step forward in applying Generative AI to cybersecurity, offering a robust, private, and efficient alternative to manual code generation and proprietary AI tools for offensive security operations.