🤖 AI

GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks

This paper introduces GenTI, a novel LLM-driven benchmark and framework that leverages a large-scale dataset of annotated detection rules to automate the generation of high-quality, context-aware IDPS rules, significantly improving the detection of unseen attacks while reducing false positives.

Original authors: Hassan Jalil Hadi, Rehana Yasmin, Ali Shoker

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hassan Jalil Hadi, Rehana Yasmin, Ali Shoker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your computer network is a busy city, and Intrusion Detection and Prevention Systems (IDPS) are the police officers patrolling the streets. Their job is to spot bad guys (hackers) and stop them.

Traditionally, these police officers work with a physical "Wanted" poster book. If a criminal matches a photo in the book (a known attack), the officer stops them. But if a criminal shows up wearing a new mask or using a new disguise (a "zero-day" or unseen attack), the officer doesn't recognize them and lets them pass.

The problem is that writing new "Wanted" posters is slow. It requires human experts to sit down, study the new criminal, and manually write a description. By the time they finish, the criminal might have already caused damage.

Enter GenTI: The AI Police Dispatcher.

This paper introduces a new system called GenTI (Generative Threat Intelligence). Think of it as a super-smart AI dispatcher that doesn't just read the old "Wanted" book; it can instantly write new, accurate "Wanted" posters for criminals it has never seen before, just by looking at a description of their behavior.

Here is how it works, broken down into simple steps:

1. The Massive Library (The Dataset)

Before the AI can learn, it needs to study. The researchers built a massive library called GTI.

The Collection: They gathered over 200,000 real "Wanted" posters (rules) from existing police databases (Snort, Suricata, YARA).
The Secret Sauce: Unlike old libraries that just had a photo, this library includes detailed context. Every poster is linked to a "criminal profile" (Cyber Threat Intelligence) that explains who the bad guy is, what their tricks are, and how they usually operate.
The Analogy: Imagine if every "Wanted" poster didn't just say "Man in a red hat," but also said, "This guy is part of the 'Red Hat Gang,' uses a specific type of lockpick, and usually strikes on Tuesdays." This extra context helps the AI understand the logic behind the rule, not just the text.

2. The Training School (The Methodology)

The AI (a Large Language Model) doesn't just memorize these posters; it goes through a rigorous training school with four levels:

Level 1 (Grammar): Learning how to write a perfect "Wanted" poster so the police officers (the IDPS software) can actually read it without getting confused.
Level 2 (Context): Learning to connect the poster to the criminal's profile (e.g., "This rule stops the 'Phishing' technique").
Level 3 (Self-Correction): The AI is taught to think out loud (Chain-of-Thought). Before finalizing a poster, it asks itself, "Does this make sense? Did I miss a detail?"
Level 4 (The Double-Check): A special "Verification Loop" (Chain-of-Verification) acts like a strict supervisor. It checks the new poster three times:
1. Is the grammar correct?
2. Does the logic match the criminal's profile?
3. Crucially: Will this poster accidentally arrest innocent people (False Positives)?

3. The Test Drive (The Results)

The researchers tested this AI in a real-world simulation. They gave it descriptions of new, unknown attacks and asked it to write the rules.

The Old Way: Without this new system, the AI could only catch about 45% of these new, unseen attacks.
The GenTI Way: With their special training and library, the AI caught 87.4% of the new attacks.
Fewer Mistakes: The old way often flagged innocent people as criminals (8.5% error rate). GenTI reduced this to just 2.3%, meaning the police are much more precise.
Overall Score: The system achieved a near-perfect score of 89.4% on a complex quality test, beating even the most famous AI models (like GPT-4) at this specific task.

The Bottom Line

The paper claims that GenTI is the first system to successfully teach an AI how to write its own security rules for brand new, unseen threats by combining a massive library of real-world data with a strict "think, verify, and correct" process.

Instead of waiting for a human expert to manually write a new rule for every new virus, this system acts like a self-evolving police force that can instantly generate the tools needed to stop new criminals the moment they appear.

Technical Summary: GenTI – Benchmarking LLMs for Autonomous IDPS Rule Generation

1. Problem Statement

Rule-based Intrusion Detection and Prevention Systems (IDPS), such as Snort and Suricata, remain the dominant choice for production environments due to their precision and interpretability. However, their reliance on manually crafted, signature-driven rules limits their adaptability to emerging and zero-day threats. Security experts must manually analyze vulnerabilities and document attack patterns to create rules, a process that cannot keep pace with the sophistication of modern attacks, which often employ obfuscation and undisclosed vulnerabilities.

Furthermore, existing public datasets (e.g., CICIDS2017, UNSW-NB15) are primarily designed for traffic classification at the flow or header level. They lack the structured, payload-level information, rule metadata, and Cyber Threat Intelligence (CTI) mappings required to train Large Language Models (LLMs) for automatic rule synthesis. Consequently, there is a critical gap in:

Data: The absence of a large-scale, publicly available benchmark dataset tailored for IDPS rule generation that includes payload-level semantics and CTI context.
Methodology: Existing automatic rule generation methods often fail to capture payload-level behaviors or lack systematic mechanisms to verify the syntactic and semantic correctness of generated rules.
Evaluation: A lack of systematic verification for whether automatically generated rules can operate with acceptable false-positive rates in realistic production environments.

2. Methodology

The authors propose GenTI (Generative Threat Intelligence), a framework comprising a new benchmark dataset (GTI) and an LLM-driven pipeline for generating, refining, and verifying IDPS rules.

A. The GTI Dataset

The GTI dataset is a CTI-enriched corpus aggregating over 200,000 rules:

150,000+ IDPS Rules: Aggregated from Snort, Suricata, and Emerging Threats.
50,000+ YARA Rules: Focused on payload-level signatures.

Key Features of GTI:

Structured Schema: Rules are normalized into a unified format containing header features (IP, ports, protocols), flow features, and payload features.
CTI Enrichment: Every rule is mapped to external intelligence sources, including MITRE ATT&CK (tactics/techniques), D3FEND (defensive techniques), CVE/CWE, AlienVault OTX, and MISP.
Rule-Context and Relationships Annotation (RCRA): Annotations linking rules to specific attack contexts, malware families, and actionable response types (alert, drop, reject).
Prompt Engineering: The dataset includes structured input prompts (analyst-style descriptions) and target outputs (canonical rules), along with Chain-of-Thought (CoT) rationales and Chain-of-Verification (CoVe) annotations.

B. The GenTI Framework Pipeline

The framework operates as an end-to-end pipeline transforming analyst prompts and attack payloads into verified rules:

Pre-processing & Prompt Construction: Analyst prompts and representative malicious payloads are parsed to extract features (e.g., HTTP headers, C2 markers). These are converted into structured prompts ( $P_{struct}$ ) and natural language prompts ( $P_{user}$ ).
LLM-Based Rule Synthesis: An LLM (specifically fine-tuned DeepSeek-7B) generates an initial candidate rule ( $r_0$ ) and a CoT trace ( $C_0$ ) based on the prompts and CTI context.
Threat Intelligence Mapping: The system queries the CTI store to map the generated rule to specific ATT&CK techniques, IoCs, and malware families.
Iterative Refinement Loop: The candidate rule is evaluated against malicious and benign traffic. A feedback mechanism ( $F_{k+1}$ ) is generated based on metrics (detection rate, false positives, syntax validity) to refine the rule in subsequent iterations.
Chain-of-Verification (CoVe): A three-stage verification process is applied to the best candidate:
- Syntactic Check: Ensures compatibility with Snort/Suricata engines.
- Logical Check: Validates consistency between the rule conditions and the mapped ATT&CK/CTI context.
- False-Positive Analysis: Replays benign traffic to ensure the rule does not trigger excessive alerts.
Curriculum Learning: The training process utilizes a multi-stage curriculum, starting with simple header-only rules and progressively introducing complex payload patterns and dense CTI mappings to stabilize training and improve generalization.

3. Key Contributions

GTI Benchmark Dataset: The first large-scale, CTI-enriched corpus tailored for LLM-based NIDPS rule generation, consolidating 200k+ rules with rich metadata, CTI mappings, and RCRA.
Payload-Level Rule Generation: The ability to translate YARA-style payload signatures into IDPS-compatible logic, enabling the automatic generation of both detection and prevention rules (drop/reject) at the payload level.
Four-Stage Training Pipeline: A novel methodology comprising:
- Syntactic grounding via rule structure learning.
- Semantic enrichment via CTI-augmented CoT reasoning.
- CoVe-driven verification for false-positive reduction.
- Full-complexity integration with transfer learning.
Real-Time Evaluation: A comprehensive evaluation framework testing generated rules on open-source engines (Snort/Suricata) using realistic traffic, measuring syntax accuracy, semantic similarity, CTI coverage, and security effectiveness.

4. Experimental Results

The GenTI model (fine-tuned DeepSeek-7B) was evaluated against strong baselines, including GPT-4o, LLaMA-3-8B, and Mistral-7B, on a held-out test set.

Performance Metrics:

Composite Rule-Quality Score: GenTI achieved 89.4%, significantly outperforming baselines (e.g., GPT-4o at 80.4%, LLaMA-3-8B at 71.6%).
CTI Coverage: Achieved 94.8%, indicating superior alignment with threat intelligence frameworks compared to baselines (~75–83%).
Syntax Accuracy: 89.5% of generated rules were syntactically valid without manual correction.
Semantic Similarity: 87.2% match with expert-analyst reference rules.

Unseen Attacks Detection:

Detection Rate: The ability to detect previously unseen attacks improved from 45% (baseline) to 87.4% (GenTI).
False Positive Rate: Reduced from 8.5% to 2.3%, demonstrating that the CoVe loop effectively refines decision boundaries without sacrificing precision.

Ablation Study:
The study confirmed that QLoRA fine-tuning provided the largest single performance boost. Curriculum learning further improved semantic similarity and security effectiveness, while CoT/CoVe augmentations were critical for reducing residual errors and improving robustness.

5. Significance and Claims

The paper claims that GenTI establishes the first large-scale benchmark that tightly couples rule-level Cyber Threat Intelligence (CTI) with LLM-based automation. Its primary significance lies in enabling adaptive, self-evolving IDPS systems.

Unlike previous approaches that merely reproduce existing signatures or focus on traffic classification, GenTI demonstrates that LLMs can act as CTI-aware agents to:

Generate new, threat-intelligence-grounded rules for unseen attacks.
Operate with high syntactic correctness and low false-positive rates in production-like environments.
Bridge the gap between raw network traffic and actionable, verified prevention logic.

The authors conclude that their work moves the field beyond simple signature copying, providing a viable pathway for automating the maintenance and expansion of IDPS rulebases against evolving real-world threats.