🤖 AI

Evaluating LLM Generated Detection Rules in Cybersecurity

This paper introduces an open-source evaluation framework and benchmark metrics designed to assess the effectiveness of LLM-generated cybersecurity detection rules by comparing them against human-generated rules using a holdout set methodology.

Original authors: Anna Bertiger, Bobby Filar, Aryan Luthra, Stefano Meschiari, Aiden Mitchell, Sam Scholten, Vivek Sharath

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anna Bertiger, Bobby Filar, Aryan Luthra, Stefano Meschiari, Aiden Mitchell, Sam Scholten, Vivek Sharath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling city where every email is a letter traveling through the mail system. In this city, there are bad actors trying to sneak in fake letters, scams, and viruses disguised as friendly notes. To stop them, we employ "detectives" who write specific rules to spot these troublemakers. For example, a rule might say, "If a letter comes from a suspicious address and has a weird attachment, flag it!" These rules are the security guards of our digital city.

For a long time, only human detectives could write these rules. They studied thousands of letters, learned the tricks of the trade, and wrote precise instructions to catch the bad guys. But recently, a new kind of detective has arrived: Artificial Intelligence (AI). These AI agents are like super-fast apprentices who can read instructions and write their own rules in seconds. The big question on everyone's mind is: Can these AI apprentices actually do the job? Or are they just writing rules that sound smart but fail to catch the real criminals, or worse, start yelling "Fire!" at innocent people? This is the puzzle a team of researchers at Sublime Security set out to solve. They wanted to see if an AI could write security rules as well as a human expert, and if so, how much it would cost to keep that AI working.

The paper introduces a new way to test these AI detectives, focusing on a specific AI agent named ADÉ (pronounced "Ah-Day"). ADÉ is designed to look at a single malicious email and write a rule to catch similar bad emails in the future. The researchers didn't just ask, "Did it work?" They built a detailed scoring system to measure three things: how well the rule catches bad emails (accuracy), how likely the rule is to break if a hacker changes a tiny detail (robustness), and how much money it costs to get the AI to write a working rule (economic cost).

To test ADÉ, the researchers played a tricky game of "hide and seek" with human-written rules. They took a list of excellent, human-made rules from Sublime Security's collection and secretly hid one from ADÉ's memory. Then, they showed ADÉ a single bad email that the hidden rule was supposed to catch and asked, "Can you write a rule to catch this?" After ADÉ wrote its rule, the researchers compared it to the original human rule using a massive database of 45,000 emails.

The results were a mix of impressive skills and understandable limitations. When it came to writing rules that were hard to trick (robustness), ADÉ performed surprisingly well, scoring almost as high as the human experts. For instance, in tests involving emails with hidden JavaScript code, ADÉ's rules were just as tough to bypass as the human ones. However, ADÉ was much more cautious. While human experts wrote broad rules designed to catch thousands of bad emails based on years of experience, ADÉ wrote very narrow rules based only on the single email it was shown. As a result, ADÉ caught fewer total bad emails (True Positives) but also made far fewer mistakes by flagging good emails (False Positives). In fact, for some tests, ADÉ's rules had zero false alarms, whereas human rules sometimes flagged a few innocent messages.

The study also looked at the "cost" of using AI. Writing a rule isn't free; every time the AI tries to write code and fails, it costs money to run the computer. The researchers found that ADÉ usually needed only 1 to 3 attempts to write a rule that passed the technical checks. This translated to a very low cost, ranging from about $1.51 to $5.13 per successful rule, depending on the complexity of the task. Interestingly, the AI tended to write much more detailed comments explaining its logic than human authors did, making the rules easier to read.

Ultimately, the paper suggests that AI agents like ADÉ are becoming capable of writing high-quality security rules, especially when it comes to avoiding false alarms and staying within budget. However, they aren't quite ready to replace human experts entirely. Because the AI only sees one email at a time, it struggles to write the broad, sweeping rules that humans create after studying thousands of cases. The researchers conclude that while AI is a powerful tool that can handle specific tasks efficiently, the best security likely comes from a partnership where AI handles the heavy lifting of rule-writing, and human experts guide the strategy to ensure the rules catch the widest net of threats.

Technical Summary: Evaluating LLM Generated Detection Rules in Cybersecurity

Problem Statement

The integration of Large Language Models (LLMs) and AI agents into cybersecurity operations faces a significant trust deficit. While vendors promote agents as revolutionary for detection, experienced engineers remain skeptical, fearing the generation of "plausible sounding nonsense." A critical gap exists in the lack of standardized, open-source metrics to evaluate the effectiveness of LLM-generated security rules. Without rigorous measurement, organizations cannot verify if an agent can perform the job of a human detection engineer, nor can they build the necessary trust to deploy these tools without wasting time and energy. Furthermore, existing evaluation methods for code generation (e.g., functional correctness via unit tests) and intrusion detection (e.g., precision/recall) do not fully address the unique challenges of security rule generation, such as adversarial robustness, operational noise, and the economic cost of iterative generation.

Methodology

The authors present an open-source evaluation framework and benchmark designed to measure the quality of LLM-generated cybersecurity rules against a human-generated baseline. The methodology employs a "holdout set" approach, comparing an autonomous agent against human experts.

The Agent: ADÉ

The framework is illustrated using ADÉ (Automated Detection Engineer), an agentic system developed by Sublime Security. ADÉ operates in a model-agnostic architecture, utilizing a curated knowledge base of detection engineering best practices, TTPs, and MQL documentation, alongside tools for dynamic file analysis, computer vision, and sender reputation.

Workflow: ADÉ receives a single malicious email sample, analyzes it, searches existing rules for coverage gaps, and generates candidate MQL detection rules.
Iterative Refinement: Unlike traditional agents that struggle with long-context drift, ADÉ employs a closed feedback loop. It validates syntax, runs rules against the sample, and initiates threat hunts on historical data. Subagents analyze these results (True Positives, False Positives, and robustness) to provide feedback, allowing ADÉ to iteratively refine the logic until convergence.

Evaluation Metrics

The framework evaluates rules across three distinct dimensions:

Detection Accuracy:
- Utilizes a labeled dataset of ~45,000 emails (malicious and benign).
- Calculates a composite Score (Equation 1) that averages standard precision with the precision of Unique True Positives (TPs not caught by any other rule in the core feed).
- Rationale: In security, unique detections are highly valuable as they increase overall system recall. The framework acknowledges the difficulty of measuring False Negatives (the "unknown unknowns") and focuses on precision-based metrics.
Economic Cost of Syntactic Correctness:
- Adapts the "cost-to-pass" framework (Erol et al., 2025) to measure the economic impact of generating valid rules.
- Calculates Total Cost = (Cost per attempt) × (Number of attempts, $k$ ).
- Uses a pass@k metric to determine how many attempts are required for the agent to generate a syntactically correct MQL rule that passes validation.
Robustness (Brittleness):
- Develops a heuristic Brittleness Score ( $B \in [0, 100]$ ) based on the ratio of rewards (robust patterns like behavioral indicators, fuzzy matching) to penalties (brittle patterns like direct string matches on IPs, domains, or hashes).
- Rationale: Direct matches are easily bypassed by trivial adversarial evasions, whereas behavioral indicators offer generalizable solutions. The final Robustness Score is $1 - (B/100)$ .

Key Contributions

The paper introduces a novel benchmark framework specifically addressing autonomous detection rule generation:

Operational Viability Pipeline: An end-to-end evaluation combining functional correctness with real-world retro-hunting to surface noise thresholds, maintainability, and performance guardrails.
Built-in Adversarial Robustness: A specific metric to measure and provide feedback on the brittleness of rules, aiming to reduce trivial adversarial bypasses.
Open-Source Framework: The authors plan to release the modular pipeline to enable reproducible research and standardized evaluation for other security rule generation agents (e.g., YARA, custom DSLs).

Results

The study compared ADÉ-generated rules against held-out human-written rules from the Sublime Core Feed across 11 distinct attack scenarios (e.g., BEC, HTML smuggling, phishing).

Scope vs. Precision: ADÉ's rules were significantly narrower in scope, detecting fewer total messages and fewer known True Positives compared to human rules. This is attributed to ADÉ's constraint of working from a single email sample, whereas humans design rules based on broad behavioral patterns across large datasets.
False Positives: ADÉ generated rules with significantly fewer False Positives (often zero in the test set), reflecting its narrower view of the threat universe.
Performance Scores:
- Human rules achieved scores ranging from 0.711 to 0.996.
- ADÉ rules achieved scores ranging from 0.557 to 1.000.
- In several cases (e.g., "EML w/ Javascript in SVG File"), ADÉ achieved a higher score (0.998) than the human baseline (0.996), driven by high precision and unique TPs.
Robustness: ADÉ and human rules were found to be "quite comparable" in terms of brittleness. The base LLM successfully combined available tools to produce queries with similar resilience to adversarial evasion as human experts.
Cost Efficiency: The pass@k metric correlated closely with the actual dollar cost of running the LLM. ADÉ typically required 1–3 attempts to generate syntactically valid rules, with costs ranging from ~$1.50 to $5.13 per rule.
Qualitative Differences: ADÉ tended to generate rules with more extensive, helpful comments than human authors, though the human rules were broader in their coverage logic.

Significance and Claims

The paper claims that this methodology provides a realistic, multifaceted evaluation of LLM-based security rule generators. By moving beyond simple functional correctness to include economic cost and adversarial robustness, the framework offers a more holistic view of an agent's operational viability.

The authors modestly conclude that while ADÉ currently produces rules that are narrower in recall than human experts (due to its single-sample input constraint), it demonstrates the ability to generate high-quality, robust, and economically efficient rules. The framework serves as a foundation for iterative improvement, suggesting that future work could use these metrics to broaden ADÉ's scope to rival human recall while maintaining its precision and robustness. The paper does not claim that ADÉ has fully replaced human engineers but rather that it is a viable, measurable component of the detection workflow that can be systematically improved.