RuleForge: Automated Generation and Validation for Web Vulnerability Detection at Scale

1. Problem Statement

Cybersecurity teams face a critical scalability challenge: the volume of newly disclosed Common Vulnerabilities and Exposures (CVEs) far exceeds the capacity for manual detection rule creation.

• The Gap: In 2025, the National Vulnerability Database (NVD) published over 48,000 new vulnerabilities, while internal teams could only manually create rules for a fraction of high-severity cases (roughly 1/15th of the volume).
• The Bottleneck: Converting structured vulnerability descriptions (e.g., Nuclei templates) into production-ready, JSON-based detection rules is time-intensive.
• The Risk: Manual processes cannot keep pace with the threat landscape, leaving organizations exposed. Furthermore, automated generation risks high false positives (flagging benign traffic as malicious), which can disrupt operations, or false negatives (missing attacks).

2. Methodology: The RuleForge System

RuleForge is an AWS internal system designed to automate the generation and validation of detection rules using Generative AI (via Amazon Bedrock). It employs a 5×5 generation strategy and a multi-stage validation pipeline.

A. System Architecture

1. CVE Repository & Selection:
   • Ingests CVE data from Nuclei templates weekly.
   • Uses a weighted scoring algorithm to prioritize CVEs based on content analysis (e.g., remote code execution, SQL injection), threat intelligence (CISA KEV), and organizational relevance.
2. Rule Generation Engine:
   • 5×5 Strategy: For each prioritized CVE, the system generates 5 parallel candidate rules simultaneously using different LLM temperature settings (0.7–0.9).
   • Iterative Refinement: Each candidate undergoes up to 5 refinement attempts. If a rule fails validation, specific feedback is fed back to the LLM to improve the next iteration.
   • Framework: Uses the DSPy framework for structured LLM interactions.
3. Validation Pipeline (5 Stages):
   • Stage 1: Synthetic Testing: The LLM generates 10 test cases (7 malicious, 3 benign). A rule passes if it correctly identifies at least 8/10 cases (allowing for a 2-error tolerance to account for synthetic test noise).
   • Stage 2: LLM-as-a-Judge Confidence Scoring: A novel mechanism where an LLM acts as a critic to evaluate the rule's Sensitivity (avoiding false negatives) and Specificity (avoiding false positives).
   • Stage 3: IP Validation: Rules are tested against 5 billion records of production web traffic. Rules matching 10–500 IPs are cross-referenced with threat intelligence (e.g., MadPot); >70% of matched IPs must be malicious.
   • Stage 4: IP Enrichment: Further analysis of matched IPs.
   • Stage 5: Human Review: Security engineers perform a final code review before deployment.

B. Key Innovation: LLM-as-a-Judge

Instead of asking the LLM "Is this rule correct?", the system prompts the model to act as a critic:

• Prompt Design: Asks, "What is the probability this rule fails to flag malicious requests?" (Sensitivity) and "What is the probability this rule targets a feature correlating with the vulnerability rather than the vulnerability itself?" (Specificity).
• Feedback Loop: If scores fall below thresholds (0.5 sensitivity, 0.7 specificity), the LLM's reasoning is appended to the prompt for the next generation attempt, providing specific guidance on why the rule failed.

3. Key Contributions

1. Automated Rule Generation at Scale: A system capable of generating JSON-based detection rules directly from structured Nuclei templates, bridging the gap between vulnerability disclosure and mitigation.
2. LLM-as-a-Judge Validation: A novel confidence scoring system that evaluates rules on sensitivity and specificity dimensions. It provides interpretable reasoning, not just a pass/fail signal.
3. Systematic Feedback Integration: A closed-loop mechanism where validation failures (both automated and human) are converted into specific prompts to iteratively refine rule candidates.
4. Proof-of-Concept Extensions:
   • Unstructured Data: Demonstrated the ability to generate rules from unstructured sources (blogs, NVD descriptions), though with a lower success rate (40%).
   • Agentic Workflow: A ReAct-based agent framework capable of routing vulnerabilities to appropriate rule generators (HTTP, Process, DNS, CloudTrail) or escalating to humans.

4. Results

The system was evaluated over the final four months of 2025 in a production environment:

• Productivity: Achieved a 336% productivity advantage over human-generated rules. AI output increased by 419% relative to baseline, while human output increased by only 38%.
• Quality Improvements (vs. Synthetic-Only Validation):
  • False Positives: Reduced by 67%.
  • Irrelevant Rules: Reduced rules matching no IPs by 71%.
  • Detection Capability: Maintained the same number of true positive rules as the baseline.
• Model Performance:
  • The LLM-as-a-judge system achieved an AUROC of 0.75 and an Expected Calibration Error (ECE) of 0.17, indicating good discriminative power and calibration.
  • Reasoning Alignment: In a small sample (9 rules), LLM reasoning matched human expert reasoning in 6/9 cases.
• Unstructured Data: A pilot on 10 unstructured sources yielded a 40% overall success rate (50% of generated rules were usable).

5. Significance and Lessons Learned

• Overcoming LLM Overconfidence: The paper highlights that LLMs are often overconfident and sycophantic. Asking the model to "find flaws" (negative phrasing) yields better calibration and discriminative power than asking "is this correct?" (positive phrasing).
• Domain Expertise in Prompting: Consulting with security experts to design evaluation prompts (focusing on false positives/negatives) significantly improved the LLM-as-a-judge's performance.
• Human-in-the-Loop: While automation is powerful, human review remains essential for final acceptance to ensure high precision and handle edge cases.
• Scalability: RuleForge demonstrates that generative AI can augment human security expertise to handle the exponential growth of CVEs, provided there is a robust, multi-stage validation framework.

Conclusion: RuleForge successfully automates the creation of web vulnerability detection rules, significantly increasing throughput while reducing false positives through a sophisticated feedback loop and LLM-as-a-judge validation. It represents a viable path toward fully automated CVE response systems.