Technical Summary: From Telemetry to Techniques

Problem Statement

Traditional signature-based and rule-based threat detection methods struggle to identify modern cyber attacks because they rely on predefined indicators of compromise (IOCs) and fail to capture complex behavioral patterns. While behavior-centric approaches using system telemetry have gained traction, significant research gaps remain. Existing studies often rely on static features or handcrafted rules, limiting generalization across heterogeneous attack behaviors. Furthermore, many approaches treat security events as independent observations, ignoring the sequential and semantic relationships embedded within adversarial behaviors. Additionally, the use of general-purpose language models often lacks domain-specific understanding of cybersecurity terminology, and severe class imbalance in security datasets hinders the reliable detection of low-frequency adversarial techniques.

Methodology

This paper proposes a behavior-centric framework to classify MITRE ATT&CK techniques using Sysmon telemetry generated through Atomic Red Team attack emulations. The methodology focuses on reconstructing discrete Sysmon events into behaviorally meaningful sequences and evaluating different learning paradigms for classification.

1. Data Preprocessing and Feature Selection

The study utilized the Splunk Attack Data repository, specifically datasets generated by Atomic Red Team simulations within the Attack Range environment. Raw Sysmon XML logs underwent a rigorous preprocessing pipeline:

• Feature Selection: From the heterogeneous Sysmon schema, features were selected based on their ability to capture process execution context, relationships, resource interactions, and temporal patterns. Key fields retained included EventID, UtcTime, process GUIDs (ProcessGUID, ParentProcessGUID, etc.), and event-specific attributes like CommandLine, Image, and TargetFilename.
• Normalization and Abstraction: Environment-specific artifacts (e.g., specific file paths, user SIDs, machine-specific GUIDs) were anonymized or replaced with generic tokens (e.g., [USER], [GUID], [BASE64]) to prevent models from learning host-specific noise.
• Deduplication: Redundant events were removed, reducing the dataset from 427,774 to 148,500 events.

2. Event Correlation Strategies

Two distinct strategies were employed to reconstruct behavioral sequences from discrete events:

• GUID-based Correlation: This approach leverages persistent process identifiers (ProcessGUID) generated by Sysmon to reconstruct process lineage and causal relationships. It groups events by process hierarchy, inter-process interactions (e.g., injection, access), and lifecycle, preserving the structural context of the attack chain regardless of PID reuse.
• Temporal-based Correlation: This approach groups events occurring within a one-second time window sharing the same MITRE ATT&CK label. It relies on chronological proximity to capture behavioral patterns without explicit process relationship reconstruction.

3. Learning Paradigms and Imbalance Handling

The study evaluated two categories of models under various class imbalance mitigation strategies:

• Traditional Machine Learning (ML): Three ensemble classifiers (LightGBM, Extra Trees, Random Forest) were trained on numerical embeddings generated by SecBERT. To address class imbalance, SMOTE (Synthetic Minority Over-sampling Technique) was applied to the embedding space.
• Transformer-Based Models: Two cybersecurity-specific language models, SecBERT and SecureBERT, were fine-tuned directly on the textual behavioral representations. Instead of synthetic oversampling, algorithm-level strategies were used: Adaptive Focal Loss (dynamically adjusting the focusing parameter) and Cost-Sensitive Focal Loss (incorporating class-weighted penalties). A baseline configuration without explicit imbalance mitigation was also tested.

Key Contributions

1. Behavior-Centric Framework: The paper proposes a systematic framework that reconstructs Sysmon telemetry into contextual behavioral sequences using both GUID-based and temporal event correlation.
2. Comparative Evaluation: It performs a comprehensive comparison between traditional machine learning classifiers and cybersecurity-specific Transformer models for MITRE ATT&CK technique classification.
3. Imbalance Strategy Analysis: The study investigates both data-level (SMOTE) and algorithm-level (Focal Loss variants) class imbalance handling strategies to identify effective combinations for this specific domain.
4. Correlation Strategy Validation: It empirically identifies the most effective combination of correlation strategy, learning paradigm, and imbalance handling for ATT&CK technique recognition.

Experimental Results

The experiments involved 24 configurations across two datasets (GUID-correlated and Temporal-correlated) and multiple models.

• Correlation Strategy: GUID-based correlation consistently outperformed temporal-based correlation across all evaluated classifiers. This indicates that preserving process-level behavioral context (lineage and interaction) contributes more substantially to technique recognition than temporal grouping alone.
• Model Performance:
  • Best Overall: The SecureBERT model trained on GUID-correlated sequences without explicit class imbalance mitigation achieved the highest performance (Accuracy: 0.586, Macro F1: 0.357).
  • Traditional ML: Among traditional models, LightGBM with SMOTE on GUID-correlated data performed best (Accuracy: 0.501).
  • Imbalance Handling: For Transformer models, the baseline (no cost-sensitive learning) generally outperformed the Focal Loss variants. In contrast, SMOTE provided consistent improvements for traditional ML models, particularly on GUID-correlated data.
• Per-Class Analysis: Techniques with distinctive behavioral patterns (e.g., T1588, T1059) achieved high F1-scores (>0.9), while techniques with overlapping behaviors (e.g., persistence mechanisms like T1547, T1543) or low sample counts (e.g., T1553) struggled, often being confused with "Indicator Removal" (T1070) or other persistence techniques.

Significance and Claims

The paper claims that the quality of behavioral representation has a greater influence on classification performance than the choice of classifier alone. The primary finding is that preserving structural process relationships (via GUID-based correlation) before semantic representation learning significantly enhances technique classification performance.

The study concludes that the combination of GUID-based event correlation and the baseline SecureBERT model provides the most effective framework for automated MITRE ATT&CK technique identification from Sysmon telemetry. The authors note that their findings offer practical guidance for selecting effective combinations of event correlation, learning models, and imbalance handling strategies for behavior-centric cyber threat classification.

The authors acknowledge limitations, specifically that the data was generated in a controlled laboratory environment (Atomic Red Team) rather than real-world enterprise settings, and that the complex structure of correlated sequences limits the applicability of conventional data augmentation techniques. They suggest future work should evaluate the framework on large-scale real-world telemetry and investigate context-aware augmentation methods.