💻 computer science

SynAT: Enhancing Security Knowledge Bases via Automatic Synthesizing Attack Tree from Crowd Discussions

This paper proposes SynAT, an automatic approach that leverages Large Language Models and event-relation extraction to synthesize attack trees from crowd security discussions, effectively enhancing security knowledge bases like CVE, CAPEC, and HUAWEI's internal systems with timely threat intelligence.

Original authors: Ziyou Jiang, Lin Shi, Guowei Yang, Xuyan Ma, Fenglong Li, Qing Wang

Published 2026-02-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziyou Jiang, Lin Shi, Guowei Yang, Xuyan Ma, Fenglong Li, Qing Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "News" is Too Slow

Imagine you own a fortress (your software). You have a list of known ways to break in (a Security Knowledge Base). But there's a catch: the official list is updated very slowly. By the time the fortress guards officially write down a new way hackers are breaking in, the hackers have already used it to steal your treasure.

Meanwhile, regular people (developers) are chatting online on sites like Stack Overflow. They are saying things like, "Hey, I think someone could steal my login token if they intercept my internet traffic!" These conversations happen before the official list is updated.

The Challenge: These online chats are messy. They are full of questions, code snippets, and casual talk. It's hard to turn a messy conversation into a clear, structured plan of how an attack works.

The Solution: SynAT (The "Smart Translator")

The authors built a tool called SynAT. Think of SynAT as a super-smart translator that listens to messy online security chats and instantly turns them into a clear, structured Attack Tree.

An Attack Tree is like a family tree, but for bad guys.

The Root (Top): The bad guy's goal (e.g., "Steal the user's session").
The Branches: The different ways they can do it (e.g., "Hack the computer" OR "Intercept the network").
The Leaves: The specific tools or steps used.

How SynAT Works (The 3-Step Recipe)

1. The "Noise Filter" (Restricting the Scope)

Imagine a security chat is a huge room full of people talking about everything: pizza, coding, and security. You only care about the security talk.

What SynAT does: It uses a powerful AI (a Large Language Model) to act like a bouncer. It scans the whole chat and says, "Okay, ignore the pizza talk. Only keep the sentences that actually talk about how to break in."
The Result: It isolates the "golden sentences" that contain the attack info, throwing away the rest.

2. The "Detective" (Extracting Events and Relations)

Now that we have the golden sentences, we need to find the specific clues.

The Clues (Events): The detective looks for the "Trigger" (the action, like "steal"), the "Target" (what they want, like "session"), and the "Instrument" (what they use, like "malware").
The Connections (Relations): The detective figures out how these clues fit together. Does the hacker need to do both A and B? (An AND gate). Or can they do either A or B? (An OR gate).
The Magic: SynAT uses a special "transition-based" method. Imagine a conveyor belt moving through the text. As the belt moves, the detective grabs the clues and snaps them together in real-time, rather than reading the whole text, stopping, and then trying to remember everything. This makes it fast and accurate.

3. The "Architect" (Synthesizing the Tree)

Finally, the Architect takes the clues and the connections and builds the tree.

If the detective found that "Stealing the session" is the goal, and it can be done by "Accessing the computer" OR "Intercepting traffic," the Architect draws the tree with "Steal Session" at the top and an OR sign connecting the two methods below it.

Why It's a Big Deal (The Results)

The researchers tested SynAT on over 5,000 real security posts from Stack Overflow. They compared it to other top-tier AI tools and found:

Better Accuracy: SynAT was much better at finding the right attack steps and connecting them correctly than the other tools.
Faster Updates: In a real-world test with Huawei, SynAT found attack methods that were discussed on Stack Overflow months or even years before they were officially recorded in public security databases.
Practical Use: They actually used SynAT to update Huawei's private security database and public databases like CVE. The experts at Huawei said, "Yes, this is a real attack we hadn't documented yet," and added it to their list.

The Bottom Line

SynAT is like a security radar that listens to the "rumors" and "gossip" of the developer community. Instead of waiting for the official news to arrive, it instantly turns those early warnings into a clear map of how to break into a system, helping companies patch their holes before the bad guys can exploit them.

Technical Summary: SynAT

Problem Statement

Cyber attacks pose a significant threat to software systems, necessitating robust security knowledge bases (e.g., CVE, CAPEC, and corporate databases like Huawei's) to identify vulnerabilities and model threats. However, these official repositories suffer from time lags in information release, leaving systems vulnerable to emergent risks like zero-day attacks. Conversely, online knowledge-sharing platforms such as Stack Overflow contain rich, real-time crowd security discussions that often precede official disclosures. The challenge lies in the difficulty of directly synthesizing structured attack trees from unstructured textual discussions. Attack trees are highly abstract conceptual diagrams requiring the inference of complex relationships between multiple attack events, a task that is non-trivial to automate due to the noise and diversity of crowd-sourced text.

Methodology: SynAT

The paper proposes SynAT, an automatic approach to synthesize attack trees from crowd security posts. The methodology follows a three-stage pipeline:

1. Restricting the Scope of Sentences

To mitigate the noise inherent in full security posts, SynAT first narrows the input to sentences likely containing attack information.

Technique: It employs a Large Language Model (LLM) combined with Prompt Learning.
Implementation: A "Cloze-Testing" prompt template is designed where the LLM predicts specific sentences ( $K$ ) from the post's question and accepted answer that may contain attack trees.
Model Selection: Experiments compared Albert, T5, GPT-3, and ChatGPT. GPT-3 (specifically text-davinci-003) with Fix-prompt Tuning achieved the highest performance (97.24% Precision, 99.36% Recall) in identifying relevant sentences.

2. Joint Event and Relation Extraction

Once the scope is restricted, SynAT extracts attack events and their logical relations simultaneously using a transition-based framework.

Definitions:
- Attack Event: Composed of a Trigger (main verb), Target (object), and Instrument (tool).
- Relations: Defined as And (both required), Or (either sufficient), and AchievedBy (parent-child dependency).
Architecture: The model utilizes a state machine with 12 transition actions (e.g., GEN-TRG, RIGHT-SHIFT, O-DELETE) to traverse the restricted sentence once.
- State ( $S$ ): Includes stacks for triggers ( $\xi$ ), arguments ( $\tau$ ), and a buffer ( $\beta$ ), along with sets for events ( $E$ ) and relations ( $R$ ).
- Mechanism: A StackLSTM embeds the current state, and a multi-label classifier predicts the next action. This allows for $O(n)$ time complexity.
Refinement: Post-extraction, the system applies Transitive and Asymmetric rules to identify implicit relations and prevent circular dependencies in the tree.
Training: The model is fine-tuned using a combined loss function balancing event/relation extraction loss (SSVM) and action prediction loss (Cross-Entropy).

3. Synthesizing Attack Trees

The extracted events and relations are converted into a formal attack tree structure using three heuristic rules designed with security practitioners:

Identifying Attack Goal: The event representing the final target (e.g., "Steal session") becomes the root node.
Identifying Attack Methods: Intermediate events become method nodes, formatted as "attacker uses [instrument] to attack [target]."
Constructing Gates & Layers: AchievedBy relations define tree layers, while And and Or relations define the logic gates between nodes at the same layer.

Key Contributions

Technique: SynAT is presented as the first work to automatically synthesize attack trees from crowd security posts using a combination of LLM-based scope restriction and transition-based joint extraction.
Evaluation: A comprehensive evaluation on 5,070 Stack Overflow security posts and 2,350 GitHub issue reports.
Application: Practical deployment to enhance Huawei's internal security knowledge base and public databases (CVE, CAPEC).
Data: Public release of a dataset containing 1,354 labeled attack trees and source code.

Experimental Results

The study evaluated SynAT against state-of-the-art baselines (Structured-Joint+, Albert, T5, GPT-3, ED3C, MLBiNet, etc.) across four research questions:

Attack Tree Synthesis: SynAT achieved the highest similarity to ground truth, with an Average Hamming Distance (AHD) of 10.24% and Tree-edit Distance Similarity (TEDS) of 7.93%, outperforming the best baseline by 8.20% and 7.13% respectively.
Event Extraction: SynAT achieved an average F1 score of 80.93%, outperforming the best baseline by 8.50%.
Relation Extraction: SynAT achieved an average F1 score of 87.81%, outperforming the best baseline by 13.59%.
Knowledge Base Enhancement:
- Public Databases: Of 121 synthesized trees, 40.50% were proposed in Stack Overflow earlier than their CVE disclosure, and 16.53% were identified as potentially new attacks not yet in CVE.
- Private Database: In Huawei's database, 84.30% of synthesized trees were accepted by practitioners, resulting in a 42.32% increment to their existing attack tree repository.
- Human Evaluation: Practitioners rated SynAT's trees significantly higher (average score 3.8/5) than baselines (2.4–3.5) in terms of accurateness and adequateness ( $p < 0.01$ ).
- Patch Design: Integrating SynAT's trees improved the vulnerability fixing rate ($Fix@10$) by 35.14% for manual patching and 25.14% for automatic patching (CodeT5).

Significance and Claims

The paper claims that SynAT effectively bridges the gap between unstructured crowd discussions and structured security knowledge. By automating the extraction of attack trees, SynAT addresses the critical time lag in official security databases, enabling organizations to identify and mitigate emergent threats (such as zero-days) more rapidly. The successful application in both public (CVE/CAPEC) and private (Huawei) contexts demonstrates the approach's practicality in enhancing software security posture. The authors emphasize that while the method relies on heuristic rules and LLMs, the transition-based extraction ensures efficiency and high accuracy, making it a viable tool for continuous security knowledge base maintenance.