Verification-Guided Specification Synthesis with Large Language Models for Intrusion Detection Rules
This paper proposes a verification-guided framework that leverages large language models and counterexample-guided inductive synthesis to automatically generate highly accurate Suricata intrusion detection rules from HTTP request traces, achieving an 81.5% detection rate with zero false positives on real-world IoT attack data.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The internet is filled with smart devices, from thermostats to security cameras, that connect our homes and offices to the digital world. While these gadgets offer convenience, they often contain hidden weaknesses that attackers can exploit. When a new weakness is discovered, security experts must write specific instructions, known as detection rules, to tell network monitoring systems how to spot and stop the bad traffic before it causes harm. Traditionally, writing these rules is a slow, manual job that requires deep expertise, creating a dangerous gap between the moment an attack is seen and the moment a defense is ready. In recent years, powerful computer programs called large language models have shown promise in helping with this task, as they can read and understand complex text. However, simply asking these programs to write security rules has proven risky; they often create instructions that are too vague, catching harmless traffic along with the bad, or they fail to work correctly when tested against real-world conditions.
A team of researchers at Doshisha University in Japan has developed a new approach to bridge this gap, turning the process of creating security rules into a careful, step-by-step verification game. Instead of asking a computer program to write the final security rule in one go, the researchers first have the program analyze a single example of a bad attack and describe, in plain logical terms, exactly what makes that request dangerous. This description acts as a draft specification. The system then tests this draft against a large collection of harmless, normal traffic from real devices. If the draft mistakenly flags a normal request as an attack, the system treats that normal request as a "counterexample" and sends it back to the computer program with a specific question: "Why did you get this wrong?" The program then diagnoses its own error, adjusts its description to be more precise, and tries again. This cycle of proposing, testing, finding mistakes, and refining continues until the description perfectly separates the bad attacks from the good traffic. Only once this description has passed every test does a separate, automated tool translate it into the final, usable security rule.
The researchers tested this method on 281 real-world attacks that had been documented in public security records, ranging from attempts to steal data to commands that force devices to execute unauthorized actions. They paired these attacks with 281 examples of normal, harmless traffic collected from actual Internet-connected devices. The results showed that their method successfully generated working security rules for 81.5 percent of the attacks. Crucially, in all the tests, the system did not produce a single false alarm, meaning it never flagged a normal, safe communication as an attack. This stands in sharp contrast to other methods that rely on large language models to write rules directly, which often catch fewer attacks or, more dangerously, trigger alarms on innocent traffic. The study found that the key to this success was not just the ability of the computer program to understand the attack, but the rigorous process of checking its work against normal traffic and forcing it to fix its own mistakes before a rule was ever created.
The researchers also explored what happens when parts of their system are removed to understand which steps mattered most. They found that if they skipped the step where the system checks its work against normal traffic and tries to fix errors, the success rate dropped dramatically, catching less than half as many attacks. This confirmed that the ability to learn from mistakes was the most important part of their design. Conversely, when they removed the final step that translates the logical description into a formal rule, the system still caught many attacks, but the process became less consistent. The study suggests that the most effective strategy is to keep the thinking and the writing separate: let the computer program focus on understanding the logic of the attack and correcting its own errors, while a simple, reliable tool handles the final translation into a format that network security systems can use.
While the new method performed exceptionally well, the researchers noted that it is not a perfect solution for every type of threat. The system works best when the attack leaves a clear, unique signature in the data it sends, such as strange characters used to break into a system or specific commands that look nothing like normal requests. It struggled more with attacks that rely on the internal state of a device or the content of a response, which are harder to identify just by looking at the initial request. The authors acknowledge that future work will need to address these harder cases, perhaps by testing the rules against real devices or more complex simulations. For now, however, this verification-guided approach offers a significant step forward, proving that by combining the understanding power of large language models with a strict, iterative process of checking and correcting, it is possible to build security defenses that are both highly effective and remarkably precise.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.