RedSage: A Cybersecurity Generalist LLM

The paper introduces RedSage, an open-source, locally deployable cybersecurity LLM trained on a massive curated dataset and agentic augmentation pipeline, which achieves state-of-the-art performance on specialized cybersecurity benchmarks while also improving general reasoning capabilities.

The Gist

Imagine you are trying to teach a brilliant but very general student how to become a master cybersecurity expert. This student (the AI) is already smart—they know how to write essays, solve math problems, and chat about history. But if you ask them, "How do I stop a hacker from stealing my passwords?" or "What does this specific error code mean?", they might guess or give a vague answer because they haven't studied the specific "textbooks" of the cyber world.

This paper introduces RedSage, a new AI assistant designed specifically to be that cybersecurity expert. Here is how they built it, explained through simple analogies:

1. The Problem: The "Generalist" vs. The "Specialist"

Currently, most AI assistants are like general practitioners. They know a little bit about everything. If you ask them about cybersecurity, they might give a generic answer.

• The Risk: Some companies use "black box" AI (like a secret recipe) that requires sending your private data to the cloud. This is risky for sensitive security info.
• The Gap: Open-source models (which you can run on your own computer) are usually too "dumb" about security because they haven't read enough security books.

2. The Solution: Building RedSage

The researchers built RedSage using a three-step "training camp" process.

Step A: The Massive Library (Continual Pre-training)

Imagine the AI is a student who has read the entire internet. The researchers took that student and said, "Now, we are going to make you read only cybersecurity books."

• The Analogy: They filtered the entire internet to find 11.8 billion "words" (tokens) specifically about hacking, defense, and security tools. It's like taking a library of 100 million books and pulling out every single one that has the word "Cybersecurity" on the spine.
• The Result: The AI now has a deep, foundational knowledge of the field, not just surface-level facts.

Step B: The "Role-Play" Simulator (Agentic Augmentation)

Reading books is good, but practicing is better. The researchers didn't just give the AI more books; they created a simulator.

• The Analogy: Imagine a drama teacher who takes a single line from a script (like "The server is down") and forces the student to act out 10 different scenes based on it. The student might play the "panic-stricken admin," the "calm hacker," or the "helpful consultant."
• What they did: They used a smart computer agent to take 28,000 high-quality security documents and turn them into 266,000 conversations. The AI practiced answering questions, explaining complex tools, and simulating real-world security scenarios. This turned dry facts into practical skills.

Step C: The Final Exam (RedSage-Bench)

How do you know the student is ready? You need a test.

• The Analogy: Most security tests are like multiple-choice quizzes where you just pick "A, B, C, or D." RedSage created a final exam that includes:
  1. Knowledge: "What is a firewall?"
  2. Skills: "How would you fix this specific vulnerability?"
  3. Tools: "Write the exact command to scan this network."
• They also added a "human-like" grader (another AI) to check if the answers were not just correct, but helpful and detailed.

3. The Results: The "Small" Giant

The most impressive part is the size. RedSage is an 8-billion parameter model.

• The Analogy: Think of other big AI models as Olympic weightlifters (huge, powerful, but expensive and slow). RedSage is a sprinter. It's smaller and lighter, but because it trained specifically for this sport, it runs faster and smarter than the heavyweights in this specific race.
• The Outcome: RedSage beat the other models by a significant margin (up to 5-6 points higher) on security tests. Even better, it didn't lose its ability to do general tasks (like writing emails or solving math); it actually got better at those too because the security training sharpened its logic.

4. Why This Matters to You

• Privacy: Because RedSage is "open-source" and small enough, you can run it on your own computer (or a local server). You don't have to send your secret company data to a big tech cloud. It stays in your house.
• Accessibility: It acts like a 24/7 senior security expert sitting next to you, ready to help analyze threats or explain tools, without needing a million-dollar budget.

In a nutshell: The researchers took a smart general AI, fed it a massive diet of security books, trained it with thousands of role-playing scenarios, and tested it with a rigorous exam. The result is a small, fast, private, and incredibly smart cybersecurity assistant that is ready to help protect our digital world.

Technical Summary

Here is a detailed technical summary of the paper "RedSage: A Cybersecurity Generalist LLM".

1. Problem Statement

The cybersecurity domain faces a critical shortage of skilled analysts and requires proactive defense strategies against increasingly sophisticated threats. While Large Language Models (LLMs) show promise as assistants, existing solutions suffer from three major limitations:

• Privacy Risks: Many rely on proprietary APIs, preventing local deployment and exposing sensitive organizational data.
• Lack of Domain Adaptation: Open-source models often lack deep cybersecurity knowledge because they are trained on general data or undergo insufficient domain-specific fine-tuning.
• Incomplete Evaluation: Existing benchmarks often focus solely on knowledge recall (Multiple Choice Questions) and fail to assess practical skills, tool proficiency (e.g., CLI, Kali Linux), or the quality of open-ended reasoning.

2. Methodology

The authors propose RedSage, an open-source, locally deployable cybersecurity assistant built on the Qwen3-8B base model. The development follows a rigorous, data-centric pipeline comprising three main stages:

A. Data Curation and Continual Pretraining

To build a robust domain foundation, the authors curated 11.8 billion tokens of cybersecurity-specific data:

• CyberFineWeb: A large-scale corpus derived by filtering the general web corpus (FineWeb) using a binary classification model (ModernBERT-base). This yielded 125M documents (89.8B tokens), from which a deduplicated subset of ~11.7B tokens was selected for continual pretraining.
• RedSage-Seed: A high-quality, manually curated collection of 28,637 documents (~0.15B tokens) organized into three categories:
  • Knowledge: Frameworks (MITRE ATT&CK, CWE, CAPEC, OWASP).
  • Skills: Offensive techniques, CTF write-ups, and hacking tutorials.
  • Tools: CLI cheat sheets and Kali Linux documentation.
• RedSage-Dump: An additional 459K documents (0.7B tokens) from trusted sources (NIST, RFCs, NVD) to supplement pretraining.
• Strategy: The model underwent Continual Pretraining (CPT) on CyberFineWeb followed by RedSage-Seed/Dump, with a 30% replay ratio of general educational data (FineWeb-Edu) to mitigate catastrophic forgetting.

B. Agentic Data Augmentation for SFT

To bridge the gap between static documents and interactive assistant workflows, the authors designed an agentic augmentation pipeline:

• Planner Agent: Analyzes seed data chunks to identify skill sets (e.g., vulnerability analysis, command generation) and devises augmentation strategies.
• Augmenter Agent: Transforms these plans into 266,000 multi-turn conversations (RedSage-Conv), simulating realistic expert-assistant dialogues.
• General Instruction Integration: The cybersecurity data was combined with general instruction data (SmolLM3/SmolTalk2) to ensure the model retains broad capabilities (summarization, math, reasoning).
• Preference Alignment: The model was further refined using Direct Preference Optimization (DPO) on the Tulu 3 Preference Mixture dataset to align outputs with human preferences for helpfulness and safety.

C. Benchmark Construction (RedSage-Bench)

The authors introduced RedSage-Bench, a comprehensive evaluation suite addressing gaps in existing benchmarks:

• Scale: 30,000 Multiple Choice Questions (MCQs) and 240 Open-Ended Q&A items.
• Dimensions: Covers Knowledge (frameworks), Skills (offensive/defensive operations), and Tool Proficiency (CLI, Kali).
• Quality Control: A multi-stage verification pipeline using LLM-as-a-Judge (Llama-3.3-70B and Qwen2.5-72B) with Chain-of-Thought prompting to ensure factual correctness, self-containment, and high-quality distractors. Open-ended answers were scored on a 0–10 scale for depth and relevance.

3. Key Contributions

1. RedSage Model: An open-source 8B parameter model achieving state-of-the-art (SOTA) performance in cybersecurity tasks while remaining locally deployable on consumer GPUs.
2. Data Pipeline: The creation of CyberFineWeb (11.7B tokens) and RedSage-Seed (28K curated docs), combined with an agentic augmentation strategy that generated 266K high-quality training samples.
3. RedSage-Bench: A new benchmark that uniquely evaluates knowledge, practical skills, and tool usage simultaneously, including rigorous open-ended quality assessment.
4. Openness: Full release of models, datasets, code, and evaluation pipelines to ensure reproducibility.

4. Experimental Results

RedSage was evaluated against general-purpose LLMs (Llama-3.1-8B, Qwen3-8B) and specialized cybersecurity models (Foundation-Sec, DeepHat, Lily-Cybersecurity).

• RedSage-Bench Performance:
  • MCQs: RedSage-8B-Ins achieved 85.73% accuracy, outperforming the strong Qwen3-8B baseline (81.85%) by +3.88 points.
  • Open-Ended QA: RedSage-8B-DPO achieved the highest mean correctness (+7% over the second-best) and quality scores, demonstrating superior reasoning and helpfulness.
• External Cybersecurity Benchmarks:
  • RedSage surpassed baselines by up to +5.59 points on established benchmarks like CTI-Bench, CyberMetric, SECURE, and SecEval.
  • It significantly outperformed specialized models like Foundation-Sec and DeepHat, particularly in tool proficiency and offensive skills.
• General Capabilities:
  • Despite heavy domain tuning, RedSage maintained or improved performance on general LLM benchmarks (Open LLM Leaderboard), including ARC-Challenge, GSM8K (math), and IFEval (instruction following).
  • RedSage-8B-DPO achieved a mean score of 74.33, surpassing the 32B parameter Qwen3-32B (73.17) on general tasks, highlighting the efficiency of the training methodology.
• Scaling: A scaling experiment on Qwen3-32B using partial RedSage data showed consistent gains, proving the pipeline's transferability to larger models.

5. Significance

• Privacy-Preserving Deployment: By achieving SOTA results at the 8B scale, RedSage enables organizations to run powerful cybersecurity assistants locally on consumer-grade hardware, eliminating the need to send sensitive data to cloud APIs.
• Holistic Expertise: Unlike previous models that focus only on theory, RedSage demonstrates proficiency in operational tasks (using tools, writing commands) and reasoning, bridging the gap between theoretical knowledge and practical application.
• Reproducibility and Community Impact: The full release of the data curation and agentic augmentation pipeline sets a new standard for open research in domain-specialized AI, allowing the community to build upon a verified, high-quality foundation.
• Ethical Considerations: The authors acknowledge the dual-use nature of offensive security knowledge but emphasize responsible release, strict licensing for research, and the inclusion of safety alignment (DPO) to promote ethical use.

In conclusion, RedSage demonstrates that domain-aware continual pretraining combined with agentic data augmentation and rigorous benchmarking can produce a highly capable, efficient, and open cybersecurity assistant that outperforms larger proprietary models while preserving data privacy.