🤖 machine learning

LLMs for Zero-Shot Threat Detection via Structured Risk Indicators

This paper proposes a two-stage LLM framework that leverages retrieval-augmented generation to create structured, interpretable risk indicators from user activity timelines, achieving state-of-the-art zero-shot detection of insider threats and APTs by demonstrating that the quality of these indicators is the primary driver of performance.

Original authors: Abdullah Alghamdi, Siamak Layeghy, Marius Portmann

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abdullah Alghamdi, Siamak Layeghy, Marius Portmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital fortresses of modern organizations, security systems constantly watch for intruders. Some of these systems scan the traffic flowing between computers, looking for suspicious patterns in the data stream. Others sit directly on the computers themselves, monitoring what happens inside: who logs in, what files are opened, which devices are plugged in, and what emails are sent. This internal monitoring is crucial because the most dangerous threats often come from within. When an attacker steals a legitimate password or convinces an employee to act against the company, they can move through the system looking exactly like a normal user. To catch them, security teams must distinguish between a person working late on a project and someone quietly stealing sensitive data, a task that requires understanding the full story of a person's behavior over time, not just a single moment.

A team of researchers at the University of Queensland has developed a new way to help computers make this distinction without needing years of training on specific examples of bad behavior. Instead of asking a computer to look at a messy pile of raw security logs and immediately shout "threat" or "safe," they built a two-step system that acts more like a careful analyst. First, the system organizes a user's activity into a timeline and asks a powerful language model to translate that timeline into a clear, structured list of risk factors. It asks the model to identify specific things that might be wrong, such as accessing files at unusual hours or connecting to strange servers. Then, a second model looks at the sequence of these risk lists over time to spot patterns that suggest an attack is unfolding. This approach allows the system to detect both insider threats, where a trusted employee turns malicious, and advanced persistent threats, where a sophisticated external attacker hides in the network for a long time.

The researchers tested this method on two different sets of security data. One dataset simulated an insider threat scenario involving nearly one hundred employees over eighteen months, capturing everything from email exchanges to file transfers. The other dataset simulated a stealthy external attack on a network, tracking thousands of connections between computers. In both cases, the new system significantly outperformed the previous best method that relied on language models. On the insider threat data, the new approach improved the ability to correctly identify attacks while reducing false alarms by more than eleven percentage points. On the network attack data, the improvement was even more dramatic, jumping by over thirty-one percentage points. The key to this success was not just using a larger computer brain, but changing how the information was processed. By forcing the first model to break down complex logs into simple, interpretable risk indicators before making a final judgment, the system became much better at spotting the subtle differences between normal work and malicious activity.

One of the most interesting discoveries was how the system handled the need for context. The researchers found that for the smaller, less powerful language models, providing a summary of the user's past behavior helped them generate much better risk indicators. It was as if the model needed to see the user's history to understand what was unusual about the present moment. However, for the larger, more capable models, this extra historical context made little difference; they were already able to generate high-quality risk indicators on their own. This suggests that the quality of the initial risk assessment is the most important factor in catching these threats. If the first step of the process produces a clear and accurate picture of the danger, the second step can easily recognize the attack pattern. If the first step is vague or confused, the system struggles, regardless of how powerful the final classifier is.

The study also revealed that the way data is grouped matters immensely. The system works best when it looks at all the activities of a single person as one continuous story, rather than looking at isolated events or connections between machines. When the researchers tried grouping data by the connection between two computers instead of by the person using them, the system's performance dropped significantly. This is because an attacker's behavior is defined by their actions across many different tools and logs, and only by viewing the whole picture can the system see the full scope of the threat. The researchers noted that while the system was excellent at spotting the insider threats in the first dataset, the second dataset, which involved very stealthy network attacks, presented a harder challenge where the signals were much fainter. Even so, the structured approach still provided a solid foundation for detection, proving that breaking down complex security data into manageable, interpretable pieces is a powerful strategy.

Ultimately, this work demonstrates that the future of automated threat detection may not lie in training computers to memorize every possible attack, but in teaching them to reason about behavior in a structured way. By using language models to translate raw data into clear risk indicators and then analyzing those indicators over time, security systems can become more accurate and reliable. The findings suggest that the most effective setup depends on the specific type of threat being hunted and the capabilities of the models used, but the core principle remains the same: clarity in the intermediate steps leads to better decisions in the final outcome. This approach offers a promising path forward for protecting organizations against both the insider who turns against them and the outsider who tries to hide in plain sight.

Technical Summary: LLMs for Zero-Shot Threat Detection via Structured Risk Indicators

Problem Statement

Host-based Intrusion Detection Systems (HIDS) are critical for identifying insider threats and the host-level stages of Advanced Persistent Threats (APTs), such as lateral movement and data exfiltration. However, detecting these threats remains challenging because malicious actions often appear benign in isolation and require contextual reasoning across heterogeneous log sources (e.g., logon, file access, network traffic) and over extended timeframes.

Traditional machine learning approaches (e.g., Random Forest, GNNs) rely heavily on labeled training data and struggle to generalize to unseen attacks. While Large Language Models (LLMs) offer a promising zero-shot alternative for analyzing semi-structured logs, current LLM-based frameworks (such as GABM) typically attempt end-to-end classification directly from raw logs. This approach forces the model to simultaneously interpret heterogeneous events, reason about user behavior, and make a threat decision in a single inference step. The authors observe that this leads to unstable predictions and low precision (high false-positive rates), limiting practical deployment.

Methodology

The authors propose a two-stage LLM framework that replaces direct classification with a structured, interpretable intermediate representation. The framework operates under a zero-shot setting (no task-specific training) and consists of three core components:

1. Data Preparation and Temporal Windowing

Per-User Grouping: Heterogeneous logs are merged into a single chronological timeline for each user. This provides a unified behavioral view, as malicious patterns often span multiple log sources (e.g., file access followed by email exfiltration).
Sliding Windows: Timelines are segmented into overlapping fixed-size windows (stride 1). A window is labeled malicious if it contains any malicious log entry.
Temporal-Context Sampling: Due to computational constraints, a deterministic sampling strategy selects windows for inference. It retains all malicious bursts and a symmetric context of benign windows (±10 for APTs, ±30 for insider threats) to preserve temporal transitions without processing the entire corpus.

2. Behavior-Aware Retrieval (RAG)

To ground risk assessment in personalized history, the framework employs Retrieval-Augmented Generation (RAG):

Embedding: Each window is encoded into a dense vector using a lightweight sentence transformer.
Retrieval: For a given window, the system retrieves the top- $k$ (3) most similar past windows from the same user's history (within a 500-window lookback).
Similarity Features: Five deterministic similarity metrics (e.g., max similarity, top- $k$ average) are computed to quantify how anomalous the current window is relative to the user's baseline.
No-RAG Baseline: A configuration exists where LLM1 receives only the current window, omitting retrieved context and similarity features, to isolate the contribution of retrieval.

3. Dual-LLM Inference

The framework utilizes two distinct LLM stages, instantiated with open-weight models (DeepSeek-R1-Distill-Qwen 32B and Llama 3.1 8B):

Stage 1: Risk Indicator Generation (LLM1)
- Input: The current window's log content (structured as JSON event objects) plus, in the RAG setting, retrieved past windows and similarity features.
- Task: The model acts as a cybersecurity analyst to generate a structured vector of normalized risk indicators (scores in [0,1]).
- Output: A fixed-length vector of threat-specific features (17 for CERT r5.2 insider threats; 20 for PicoDomain APTs). These features cover categories like behavioral deviation, data exfiltration, lateral movement, and authentication anomalies.
- Constraint: The output is strictly a JSON object of scores, removing free-form reasoning noise.
Stage 2: Temporal Classification (LLM2)
- Input: The ordered sequence of risk indicator vectors produced by LLM1 for a user.
- Task: The model classifies each window as benign (0) or malicious (1) by reasoning over the temporal evolution of the risk scores.
- Reasoning: The prompt instructs LLM2 to identify "bursts" of elevated risk and coordinated attack patterns (e.g., lateral movement followed by exfiltration) rather than evaluating windows in isolation. It handles chunked processing (20 windows with 5-window overlap) to manage context limits.

Key Contributions

Structured Zero-Shot Framework: The paper proposes a novel architecture that decouples feature generation from classification. By replacing raw log classification with the generation of structured, interpretable risk indicators, the framework enables more robust zero-shot detection of insider threats and APTs.
Personalized Behavioral Grounding: Unlike prior work that retrieves external threat intelligence (e.g., MITRE ATT&CK), this framework retrieves personalized historical behavior. This allows risk indicators to be generated relative to an individual's established activity patterns, distinguishing normal variation from genuine malice.
Empirical Analysis of Model Roles and Retrieval: The study provides a comprehensive analysis of how model capability and retrieval interact:
- Retrieval as a Substitute: Retrieval significantly benefits weaker LLMs (Llama 3.1 8B) by improving the quality of generated risk indicators, acting as a substitute for internal reasoning capacity. Stronger models (DeepSeek 32B) achieve comparable performance without retrieval.
- Optimal Assignment: The most effective assignment of models to the two stages depends on the dataset. For stealthy APT data (PicoDomain), the stronger model is required at the feature generation stage (LLM1). For insider threat data (CERT r5.2), the stronger model is more effective at the classification stage (LLM2).

Results

The framework was evaluated on two benchmark datasets: CERT r5.2 (insider threats) and PicoDomain (APTs). It was compared against the state-of-the-art GABM framework.

Performance Gains: All configurations outperformed GABM.
- CERT r5.2: The best configuration (Llama 8B as LLM1, DeepSeek as LLM2, with RAG) achieved an F1-score of 64.14%, an improvement of 11.40 percentage points over GABM (52.74%). It achieved high recall (94.47%) while significantly improving precision (48.55% vs. 35.82%).
- PicoDomain: The best configuration (DeepSeek as LLM1, Llama 8B as LLM2, No-RAG) achieved an F1-score of 50.87%, an improvement of 31.50 percentage points over GABM (19.37%). Precision improved by 25.85 percentage points while maintaining high recall (83.50%).
Ablation Studies:
- Window Representation: Structured JSON event objects significantly outperformed verbose text or simple token sequences, highlighting the importance of rich, structured input for LLM1.
- Grouping Strategy: Per-user grouping substantially outperformed connection-pair grouping, confirming that merging all log sources into a single user timeline is critical for capturing behavioral context.
Feature Analysis:
- On CERT r5.2, LLM1 features showed clear separation between benign and malicious classes (mean absolute difference ~0.057).
- On PicoDomain, per-window features showed very weak separation (difference ~0.008), with some features even inverted (benign scoring higher). The authors attribute this to the stealthy nature of APT beaconing, noting that detection on PicoDomain relies on the temporal aggregation of these weak signals by LLM2 rather than strong per-window signals.

Significance and Claims

The paper claims that the primary driver of zero-shot cyber threat detection performance is the quality of the generated risk indicators, not merely the raw classification capability of the LLM.

Architectural Superiority: The dual-stage architecture (Feature Generation $\to$ Temporal Classification) is shown to be superior to end-to-end classification, which struggles with the complexity of heterogeneous logs.
Precision-Recall Balance: The framework successfully addresses the high false-positive rate (low precision) typical of previous LLM-based approaches (like GABM) by grounding risk assessment in personalized historical baselines and using structured indicators.
Contextual Limits: The authors modestly note that the reported precision reflects a sampling strategy enriched for malicious activity. In a real-world, predominantly benign stream, the false-positive rate would likely be higher, though the relative improvements over baselines are expected to hold.
Boundary Cases: The paper acknowledges that for highly stealthy APT traffic (PicoDomain), per-window detection is a "boundary case" where the signal is near-chance level, and the framework's success relies on the temporal reasoning of LLM2 to aggregate these weak signals.

The work demonstrates that combining personalized behavioral retrieval with structured, LLM-generated risk indicators provides a unified and effective approach for zero-shot detection of both insider threats and APTs.