Evolution of Log-Based Detection Rules in Public Repositories
This paper presents the first longitudinal analysis of log-based detection rule evolution in public repositories, revealing that rule changes are predominantly non-monotonic and reflect ongoing operational trade-offs between coverage and false positives rather than steady convergence toward stable forms.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to catch a specific type of criminal. You write down a set of instructions, or a "rule," to spot them. Maybe the rule says: "If you see a person wearing a red hat AND carrying a blue bag, raise the alarm."
In the world of cybersecurity, these "rules" are called log-based detection rules. They are the instructions security teams use to scan millions of computer records (logs) to find bad guys.
This paper is like a long-term study of how these detective instructions change over time. The researchers looked at two giant public libraries where security experts share their rules: the Sigma project (a community-driven group) and Splunk Security Content (a curated, professional collection). They tracked thousands of rules over nearly a decade to see how they evolved.
Here is what they found, explained through simple analogies:
1. The Rules Are Never "Finished"
You might think that once a detective writes a rule, it stays the same forever. But the researchers found that 56% of the rules get rewritten at least once.
Think of these rules like a recipe for a soup.
- At first, you write a recipe: "Add salt and pepper."
- Later, you realize it's too salty, so you change it to: "Add less salt."
- Then, you realize you forgot the garlic, so you add it back.
- Then, you realize the garlic makes it too spicy, so you remove it.
The study found that rules are constantly being tweaked. They don't just get "better" in a straight line; they go up and down, adding and removing ingredients constantly.
2. The "See-Saw" Effect
The most surprising discovery is that these rules rarely settle down. Instead, they act like a see-saw.
- Side A (Coverage): Sometimes, the detective wants to catch more criminals. They make the rule broader: "If you see a red hat OR a blue bag OR a green scarf, raise the alarm!" This catches more bad guys, but it also accidentally catches innocent people (false alarms).
- Side B (False Positives): Then, the detective gets annoyed by all the innocent people getting caught. They make the rule stricter: "Only raise the alarm if you see a red hat AND a blue bag." This stops the false alarms, but now you might miss some real criminals.
The study found that about one-third of the rules swing back and forth between these two sides for years. They don't find a "perfect" middle ground and stay there. Instead, they keep oscillating, trying to balance catching bad guys without annoying the security team with too many false alarms.
3. How They Studied the Changes
To understand these changes, the researchers couldn't just look at the text of the rules, because security experts often rewrite the same logic in different ways (like rewriting a sentence to sound more formal).
They invented a special tool called a "Predicate Graph."
- The Analogy: Imagine taking a complex sentence and turning it into a family tree diagram.
- The "AND" and "OR" words become the branches.
- The specific details (like "red hat" or "blue bag") become the leaves.
- By comparing these trees, they could see the structure of the rule, ignoring the messy text. This allowed them to see exactly when a rule added a new branch, removed a leaf, or rearranged the whole tree.
4. The "Why" Behind the Changes
The researchers used an AI (a Large Language Model) to read the changes and guess why the experts made them. They found three main reasons why rules keep changing:
- The "Tug-of-War" (Oscillation): Sometimes, the team can't decide between catching more people or having fewer false alarms. They try one version, hate the results, switch to the other version, hate that result, and switch back. It's like trying to find the perfect temperature on a shower that keeps fluctuating.
- The "Package Deal" (Coupled Changes): Sometimes, adding a new thing to catch forces you to add a new thing to exclude.
- Example: If you decide to catch a new type of virus, you might accidentally start catching a harmless program that looks similar. So, you have to add a rule to exclude that harmless program at the exact same time. You can't have one without the other.
- The "Lost in Translation" (Insufficient Evidence): Sometimes, the rule changes because the computer system it's watching changed its language (like a field name changing from "user_name" to "username"). The rule works, but you can't tell from the text alone if it's catching more or fewer people. It's like changing the language of a book; the story is the same, but the words are different.
5. The Big Takeaway
The main conclusion of the paper is that security rules are not static blueprints; they are living, breathing things.
They don't evolve toward a perfect, stable form. Instead, they reflect the ongoing struggle of security teams. They are constantly juggling the trade-off between:
- Not missing a bad guy.
- Not annoying everyone with false alarms.
The paper argues that because these rules are so messy and constantly changing, we need better tools to help security teams manage them. We need systems that understand the logic of the rules, not just the text, and tools that can help teams make these tough trade-offs without having to rewrite the whole rule from scratch every time.
In short: Writing a security rule is like tuning a radio. You keep turning the dial back and forth, trying to find the clearest signal, but the station keeps drifting, so you have to keep adjusting.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.