Technical Summary: Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations

Problem Statement

Security teams face an overwhelming volume of false alarms and a shortage of specialized expertise required to analyze system audit logs for attack investigations. While Large Language Models (LLMs) have shown promise in security forensics, there is a significant gap in published work evaluating their specific capabilities for analyzing security audit logs and conducting attack investigations. This gap stems from a lack of diverse, well-labeled datasets and the complex, amorphous nature of real-world investigations, which require bridging the semantic gap between low-level system calls and high-level attacker behaviors.

Methodology

The AuditBench Benchmark

To address these limitations, the authors introduce AuditBench, a new evaluation benchmark designed to assess LLM capabilities in security log analysis. The benchmark encompasses:

• Scope: Four common security investigation tasks:
  1. Classification (Alert Triaging): Determining if an alert corresponds to a real attack or a false alarm.
  2. Persistence Investigation: Identifying mechanisms allowing attackers to maintain system access.
  3. Lateral Movement Investigation: Identifying outbound connections enabling remote access and control.
  4. Data Exfiltration Investigation: Identifying events where sensitive data is moved externally.
• Dataset Composition: The benchmark contains 51 distinct scenarios (25 from newly generated "Lab" data and 26 curated from the public DARPA OpTC dataset).
  • Lab Data: Generated on Linux and Windows VMs using the MITRE ATT&CK framework and Atomic Red Team (ART) to simulate specific attack techniques and benign activities.
  • OpTC Data: Curated from the DARPA OpTC dataset, the largest public system audit log dataset, with manual verification of ground truth labels.
• Data Representations: The benchmark evaluates two distinct log representations:
  1. Raw: Direct output from native OS log collectors (e.g., auditd, procmon).
  2. Edge: A pre-processed, graph-based representation (provenance graph) where log lines are converted into edges representing subject-action-object relationships. This representation reduces redundancy and volume, particularly for Windows logs.
• Evaluation Framework: An automated pipeline that computes True Positive Rate (TPR), False Positive Rate (FPR), and F1 scores. It also evaluates the quality of LLM-generated explanations using a Natural Language Inference (NLI) approach, where an LLM acts as a judge to verify if the explanation is semantically entailed by the ground truth.

Experimental Setup

The study evaluated five frontier LLMs: GPT-5, GPT-5 mini, Gemini 2.5 Pro, Gemini 2.5 Flash, and Llama 4 Maverick. The experiments tested various design choices, including:

• Model Size: Comparing large frontier models against smaller, high-performing models.
• Prompt Engineering: Comparing two prompt versions (a general template vs. a refined version with specific domain constraints and persona instructions).
• Data Representation: Evaluating performance on raw vs. edge representations.
• Chunking: Analyzing the impact of different log chunk sizes on performance.

Key Results

1. Mixed Performance and False Positives

No single model achieved strictly superior performance across all tasks. A dominant trend observed was an overly-suspicious interpretation of audit logs, leading to high false positive rates (FPR) across many tasks.

• Exfiltration: Models generally performed best on data exfiltration tasks, with some achieving high TPR and low FPR.
• Lateral Movement: This task yielded the lowest median F1 scores, with models frequently misclassifying benign activity as attacks.
• Classification: While some models achieved perfect TPR, they often suffered from high FPRs, flagging benign system activities as malicious.

2. Model Size vs. Performance

Contrary to the expectation that larger models always perform better, the study found that smaller models often matched or exceeded the performance of larger models.

• On several tasks (e.g., persistence and lateral movement on edge data), smaller models like GPT-5 mini and Llama 4 Maverick achieved higher or equivalent F1 scores compared to GPT-5 and Gemini 2.5 Pro.
• This suggests that for security investigations, practitioners may not require the most expensive, largest models to achieve optimal results.

3. Impact of Data Representation

The choice of data representation significantly impacted performance, but the effect was model-dependent.

• For some models (e.g., Llama 4), the compact edge representation led to strictly better performance across all tasks compared to raw logs.
• For others (e.g., Gemini 2.5 Pro), raw logs sometimes yielded better results.
• This indicates that data pre-processing and transformation are critical levers for optimizing both cost and performance.

4. Prompt Engineering Effects

Prompt variations had model-dependent effects. While a refined prompt (Prompt v2) generally improved performance for many models by reducing false positives through specific domain constraints, it degraded performance for others (e.g., Gemini 2.5 Flash on classification). This cautions against "one-size-fits-all" prompt strategies when comparing models.

5. Explanation Quality and Error Analysis

• Explanation Quality: When models correctly identified attacks (True Positives), their generated explanations were of high quality (average entailment scores > 4.0/5.0), accurately capturing key attack behaviors without spurious details.
• Error Characteristics: False positives were rarely shared across models (low error overlap). The primary drivers of false positives were:
  1. System Object Names: Models flagged activities based on "non-standard" or suspicious-sounding file/process names (e.g., clear_logs, invis.vbs).
  2. Event Sequences and Volume: Models misinterpreted high-volume activities or specific execution chains (e.g., svchost spawning cmd.exe) as malicious without sufficient context.

Significance and Contributions

The paper claims to provide a foundational resource for the security and AI communities:

1. Novel Benchmark: It contributes AuditBench, an open-source dataset and evaluation framework covering diverse attack scenarios, benign activities, and multiple log representations. This enables systematic study of LLMs in security log analysis.
2. Empirical Insights: The study offers practical guidance for practitioners, suggesting that smaller models and specific data transformations (like edge representations) can be more cost-effective and performant than simply scaling up model size.
3. Design Guidance: It highlights that design choices (prompts, data representation) are highly model-specific, urging developers and security teams to avoid direct head-to-head model comparisons without exploring the design space for their specific context.
4. Future Directions: The work identifies the need to mitigate high false positive rates caused by sensitivity to object names and activity volume, and suggests exploring model ensembles to leverage the low overlap in errors for improved robustness.

The authors emphasize that their work is an initial step, providing the necessary data and tools for future research to improve LLM robustness and utility in security operations.