Multi-Source Cybersecurity Logs: An ATT&CK-Labeled Dataset and SLM Evaluation

Technical Summary: Multi-Source Cybersecurity Logs: An ATT&CK-Labeled Dataset and SLM Evaluation

Problem Statement
Modern multi-stage cyberattacks traverse system, network, and browser logs simultaneously. Detecting these attacks requires correlating events across all three sources to reconstruct the full attack chain. While machine learning (ML) methods offer a path to learning these cross-source patterns, they require labeled multi-source data, which is currently unavailable in the public domain. Existing datasets suffer from significant limitations: network-only datasets (e.g., CICIDS, UNSW-NB15) lack host and browser visibility; host-focused datasets (e.g., LMDG, CICAPT-IIoT) omit browser telemetry; and the closest prior work, ATLAS, includes all three sources but lacks granular MITRE ATT&CK technique labels, categorizing events only as malicious or benign. Consequently, researchers cannot train or evaluate models that reason across all three sources at the technique level.

Methodology
To address this gap, the authors constructed a new dataset and evaluated the learnability of Small Language Models (SLMs) on it through the following methodology:

1. Dataset Construction:

   • Scope: The dataset comprises 870 sessions (70 attack simulations and 800 benign user sessions) recorded over 13 months (January 2025 – February 2026), totaling approximately 2.3 million events.
   • Sources: System, network, and browser logs were captured simultaneously on Windows endpoints (physical and virtual) for 20-minute intervals per session.
   • Attack Simulation: Attacks were generated using genuine tools (e.g., Revenge-RAT, BD2 .NET Injector, Playit.gg for C2) rather than synthetic traffic generators. Scenarios covered 12 ATT&CK tactics and 53 techniques, including Initial Access, Execution, Persistence, Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, Collection, Exfiltration, Command and Control, and Impact.
   • Benign Data: 800 sessions captured routine user activities (web browsing, productivity, software development, system administration, entertainment) to ensure model robustness against false positives.
   • Labeling: Malicious events were tagged with specific ATT&CK technique IDs. Data was structured into "chunks" (sequences of seven timestamped events) serving as the input unit for SLMs.

2. Model Evaluation:

   • Models: Three instruction-tuned SLMs were selected: Qwen2.5-1.5B, Llama-3.2-3B, and Phi-4-Mini (3.8B parameters).
   • Fine-Tuning: All models were fine-tuned using Low-Rank Adaptation (LoRA) with a rank of 16 and scaling factor of 32.
   • Tasks: The models were evaluated on two tasks:
     1. Chunk Classification: Determining if a chunk is "normal" or "suspicious."
     2. ATT&CK Technique Identification: Generating the specific technique IDs and reasoning for suspicious events.
   • Metrics: Ten metrics were used, including accuracy, precision, recall, F1-scores (macro and weighted), exact-match accuracy, average partial match, and word-level F1.

Key Contributions

1. A Novel Multi-Source Dataset: The paper releases a dataset combining simultaneous system, network, and browser logs with per-entry ATT&CK technique labels derived from real attack tooling. It covers 12 tactics and 53 techniques.
2. Benchmarking SLMs: The study provides a controlled evaluation of three SLMs in both base and LoRA-fine-tuned forms on multi-source security data.
3. Empirical Evidence of Learnability: The authors demonstrate that the dataset contains a learnable signal, as fine-tuning significantly improved model performance across all metrics.

Results

• Performance Gains: Fine-tuning resulted in consistent improvements across all three models and all ten metrics.
  • Chunk Classification: Base model accuracy hovered around 7–8%. After fine-tuning, accuracy rose to between 89.9% and 97.0%.
  • Over-Alerting Correction: Base models exhibited severe over-alerting, labeling nearly all normal traffic as suspicious (e.g., correctly identifying only 0.1% to 0.7% of normal sessions). Fine-tuning resolved this, with recall for normal sessions improving to 89.6%–98.0%.
  • Model Variation: Phi-4-Mini (3.8B) performed best, achieving 97.0% accuracy in chunk classification and identifying 97.0% of suspicious sessions. Llama-3.2-3B struggled with rare attacks, identifying only 39.5% of suspicious sessions.
• Technique Identification: This task remained challenging. Base models achieved 0% exact-match accuracy. Fine-tuned models improved, with Phi-4-Mini reaching 41.6% exact-match accuracy. However, high partial-match scores (0.857–0.931) and word-level F1 scores (0.819–0.930) indicate that the models captured most underlying reasoning even when failing to reproduce exact output strings.

Significance and Claims
The paper claims that controlled simulations can produce high-quality, multi-source log datasets suitable for training SLMs. The primary significance lies in demonstrating that:

1. Data Scarcity is Addressed: The new dataset fills the void of public, multi-source data with granular ATT&CK labeling.
2. SLMs are Viable for Triage: While base SLMs are operationally unusable due to false positives, fine-tuned SLMs achieve sufficient reliability to support Security Operations Center (SOC) analysts in initial triage.
3. Reasoning Capability: The high partial-match scores suggest that fine-tuned SLMs can assist human analysts by suggesting relevant techniques and reasoning, even if exact technique identification is not yet perfect.

The authors maintain modesty regarding their conclusions, noting that the results indicate trends rather than proven rankings due to the limited sample size of models tested. They acknowledge that while the performance jump is genuine, the specific superiority of one model over another requires further validation on production logs and repeated trials.