CAM-LDS: Cyber Attack Manifestations for Automatic Interpretation of System Logs and Security Alerts

This paper introduces CAM-LDS, a comprehensive and reproducible dataset of cyber attack manifestations across 81 techniques designed to overcome the scarcity of labeled data, thereby enabling and demonstrating the potential of Large Language Models for automated, semantically rich interpretation of system logs and security alerts without relying on manual domain-specific configurations.

The Gist

Imagine you are a detective trying to solve a crime inside a massive, bustling city. The city is your computer network, and the crime is a cyberattack. To solve the crime, you have access to millions of pages of police reports, security camera footage, and phone logs. These are your system logs.

The problem? There are too many pages, they are written in confusing shorthand, and most of them are just boring reports about people buying coffee or turning on lights. Finding the actual criminal activity is like trying to find a single needle in a haystack made of other needles.

This paper introduces a new tool to help detectives: CAM-LDS.

1. The "Training Gym" (The Dataset)

Usually, when researchers want to teach computers to spot hackers, they have to use real crime data. But real crime data is messy, private, and often missing the "who, what, and why."

The authors built a gym for hackers (a controlled, open-source test environment). Inside this gym, they hired "good guys" to act like "bad guys" and perform 7 different types of heists.

• The Heists: They didn't just break in; they did everything a real hacker does: scanning the perimeter, picking locks, stealing keys, hiding in the shadows, and running away with the loot.
• The Recordings: They recorded everything that happened on the "police reports" (logs) during these heists.
• The Result: They created a massive, labeled library called CAM-LDS. It contains 81 different "moves" a hacker can make, all clearly marked so researchers know exactly what happened. It's like having a video game replay where every cheat code is highlighted.

2. The "Super Detective" (The AI)

For years, security experts have tried to use computers to read these logs. But old computers were like strict librarians: they could only find things if you gave them a specific list of keywords (e.g., "If you see the word 'hack', raise an alarm"). If the hacker used a new word or a different method, the computer missed it.

Enter Large Language Models (LLMs), like the AI behind ChatGPT.

• The Analogy: Think of an LLM as a super-intelligent detective who has read every book in the library. Instead of just looking for keywords, this detective understands the story. It can read a log entry and say, "Hmm, this looks like someone trying to pick a lock, even though they didn't use the word 'pick'."

3. The Experiment

The authors took their "Training Gym" (CAM-LDS) and handed the logs to the "Super Detective" (the LLM) to see if it could figure out what the hackers were doing.

• The Challenge: They didn't teach the AI anything beforehand. They just said, "Here are some logs. Tell me what happened." This is called a "zero-shot" test.
• The Results:
  • The Star Performer: For about one-third of the attacks, the AI got it 100% right immediately. It spotted the exact technique the hacker used.
  • The Good Student: For another one-third, the AI was very close. It might not have picked the exact move, but it was in the top 10 most likely guesses.
  • The Struggler: For the rest, the AI was confused. This usually happened when the hacker was very sneaky, didn't leave many footprints, or when the logs were too quiet.

4. Why This Matters

The paper found three big things:

1. Hiding is Hard (but possible): Even when hackers try to be quiet, they usually leave some trace in the logs, like a change in how fast the computer is working or a weird pattern in the data.
2. Old Alarms Miss Stuff: The standard security alarms (Intrusion Detection Systems) only caught a small fraction of these attacks. They are like smoke detectors that only go off if the fire is huge; they miss the small sparks.
3. AI is a Game Changer: The AI detective was able to understand the context. It could tell the difference between a system administrator doing their job and a hacker doing the same thing, just by looking at the timing and the surrounding clues.

The Bottom Line

This paper is a major step forward because it gives researchers a standardized, open-source playground to train and test AI detectives.

Before this, researchers were trying to learn how to spot hackers by looking at blurry, private photos. Now, they have a crystal-clear, labeled video game replay. The results show that AI is ready to help humans make sense of the chaos in computer logs, turning a mountain of confusing data into a clear story of what the bad guys are doing.

In short: We built a perfect crime scene simulator, gave it to a super-smart AI, and found out that the AI is getting really good at solving the case. Now, we just need to teach it how to handle the really tricky, sneaky criminals.

Technical Summary

Here is a detailed technical summary of the paper "CAM-LDS: Cyber Attack Manifestations for Automatic Interpretation of System Logs and Security Alerts."

1. Problem Statement

Log data is fundamental for intrusion detection and forensic investigations, yet manual analysis is hindered by high data volumes, heterogeneous formats, and unstructured messages. While automated methods exist, they typically rely on:

• Domain-specific configurations: Expert-defined rules and handcrafted parsers.
• Limited semantic understanding: Inability to explain the underlying causes of anomalies or understand the context of an attack chain.
• Data scarcity: A lack of publicly available, labeled datasets that cover a broad range of attack techniques across diverse tactics, specifically designed for evaluating Large Language Model (LLM) capabilities in security.

Existing datasets often focus on network traffic, are restricted to Windows environments, rely on noisy red-teaming exercises with poor reproducibility, or lack granular labeling for specific MITRE ATT&CK techniques.

2. Methodology

The authors introduced CAM-LDS (Cyber Attack Manifestation Log Data Set) through a rigorous, reproducible methodology designed to isolate attack traces from background noise.

• Scenario Design & Selection:

  • Technique Selection: From the MITRE ATT&CK Enterprise matrix (v18.1), 122 techniques were selected based on observability (must generate log artifacts), feasibility (must run in a Linux environment), and automation (must be fully scriptable).
  • Kill Chain Construction: Techniques were grouped into 7 coherent attack scenarios representing realistic kill chains. Variants were introduced to test different execution paths for the same objective.
  • Total Scope: The dataset covers 81 distinct techniques across 13 tactics.

• Infrastructure & Execution:

  • Environment: A fully open-source, virtualized Linux environment (AttackBed) using Infrastructure-as-Code (OpenTofu, Terragrunt, Ansible).
  • Network Topology: A segmented enterprise network (Internet, DMZ, LAN, User, Admin zones) with 18 distinct data sources.
  • Emulation Tool: AttackMate was used to execute attacks. Unlike simple scripts, AttackMate supports interactive sessions and keystroke simulation to ensure realism while maintaining repeatability.
  • Noise Reduction: A 15-minute stabilization period post-boot and a 15-second interval between attack steps were enforced. A filtering process removed log events identical to a baseline "idle" period.

• Data Collection:

  • Sources: System logs (audit, syslog, auth, cron), application logs (ZoneMinder, Docker, Nextcloud, etc.), network traffic (Netflows, PCAPs), and IDS alerts (Wazuh for host-based, Suricata for network-based).
  • Output: Raw logs, network captures, system configurations, and filtered datasets are released publicly.

3. Key Contributions

1. CAM-LDS Dataset: The first publicly available dataset specifically designed for LLM-based interpretation of system logs and security alerts. It includes 34 simulations, 243 attack steps, and covers 81 techniques across 13 tactics in a Linux environment.
2. Systematic Analysis of Manifestations: A detailed categorization of how attacks manifest in logs, including:
   • Command Observability: Whether the executed command is visible in logs.
   • Event Frequencies: Analysis of log volume spikes.
   • Performance Metrics: Correlation of attacks with CPU, memory, and load changes.
   • Alert Correlation: Evaluation of detection by signature-based IDS.
3. Zero-Shot LLM Evaluation: An illustrative case study using ChatGPT to interpret logs without training data, demonstrating the potential and limitations of LLMs in security operations.

4. Key Results & Findings

A. Dataset Characteristics

• Coverage: The dataset covers 37.5% of all MITRE ATT&CK techniques (81 out of 216) and 16 of the 19 most prevalent techniques found in threat intelligence reports.
• Observability:
  • 47.3% of attack steps left explicit command indicators in audit logs.
  • 17.3% left indicators in other logs (syslog, access logs).
  • 32.1% generated events without direct command references.
  • 18.5% generated no log events after filtering.
• Manifestation Types: Attacks manifest as:
  1. Direct command traces.
  2. Indirect consequences (e.g., file creation, config changes).
  3. High-frequency anomalies (e.g., scanning, brute-forcing).
  4. System performance metric shifts (e.g., CPU spikes during scans).
• IDS Limitations: Only 43 out of 198 attack steps triggered Low-to-High severity alerts in either Wazuh or Suricata. 155 steps remained undetected by signature-based rules, highlighting the gap in current detection capabilities.

B. LLM Interpretation Performance

In a zero-shot setting (no training data), an LLM (ChatGPT) was tasked with classifying log excerpts into the top 10 MITRE ATT&CK techniques.

• Accuracy:
  • ~33% of attack steps were classified perfectly (correct technique at rank #1 or #2).
  • ~33% were classified adequately (correct technique within the top 10).
  • ~33% were poorly classified (no correct technique in top 10).
• Correlation with Manifestations:
  • Highest Accuracy: Occurred when attack commands were visible in both audit logs and other sources.
  • Frequency Impact: Steps generating high volumes of logs (>1,000 events) were classified more accurately, likely due to stronger indicators and better sampling.
  • Alert Impact: Steps triggering IDS alerts were significantly more likely to be correctly classified, suggesting alerts provide valuable semantic context.

C. Qualitative Insights

The LLM demonstrated the ability to:

• Correlate events across multiple log sources (e.g., linking a web request in access logs to a shell execution in audit logs).
• Identify context (e.g., recognizing a "sudo" command as benign in isolation but malicious when followed by credential dumping).
• Detect tools via user-agent strings (e.g., identifying "Nikto" in access logs).

5. Significance and Future Implications

• Bridging the Gap: CAM-LDS addresses the critical lack of reproducible, labeled data for evaluating AI/LLM in cybersecurity.
• Complementing IDS: The results confirm that signature-based IDS miss a majority of attack steps. LLMs offer a promising complementary approach for semantic interpretation and root cause analysis.
• Operational Utility: LLMs can assist analysts by explaining why an event is suspicious, reducing the time spent on manual log review.
• Future Directions:
  • Integrating asset context (software versions, configurations) to improve accuracy (e.g., knowing a specific version is vulnerable).
  • Chain-of-thought reasoning across multiple attack steps rather than isolated analysis.
  • Exploring smaller, locally deployable models to address privacy and cost concerns.
  • Expanding to Windows environments and other OSs.

The paper concludes that while LLMs are not yet a silver bullet, they possess significant potential for automating log interpretation, provided they are supported by high-quality, context-rich datasets like CAM-LDS.