Structural Leakage in Host Intrusion Alert Corpora: An Evaluation Framework for ATT&CK Technique and Tactic Mapping

Technical Summary: Structural Leakage in Host Intrusion Alert Corpora

Problem Statement
Host-based intrusion detection systems (HIDS) generate abundant telemetry, yet interpreting these alerts as structured evidence of adversary behavior remains challenging. While frameworks like MITRE ATT&CK provide a behavioral vocabulary for mapping security events, prior research has largely focused on incident narratives rather than raw, rule-generated host alerts. A critical gap exists in understanding whether ATT&CK-aligned classification remains valid when applied to alert corpora that are inherently repetitive, rule-driven, and prone to "structural data leakage." Specifically, standard train-test splits often fail to account for the fact that Wazuh rule identifiers and alert descriptions map almost deterministically to specific ATT&CK labels. Consequently, naïve evaluation methods risk inflating performance estimates by rewarding the memorization of alert templates rather than genuine semantic generalization.

Methodology
This study employs a leakage-aware experimental design using data collected from a controlled Cyber Range environment. The infrastructure consisted of 38 virtual machines (Windows and Linux) monitored by Wazuh v4.10.2, subjected to a multi-stage Advanced Persistent Threat (APT) simulation covering reconnaissance, lateral movement, and impact.

The methodology proceeds through the following stages:

1. Data Collection & Pre-processing: A corpus of 2,408 alerts was parsed. The dataset was filtered to include only alerts with ATT&CK Enterprise tactic and technique labels (48.38% of the total). A critical pre-processing step involved exact-text deduplication, removing 345 duplicate rows to ensure that repeated alert formulations did not bias the evaluation.
2. Leakage Diagnostics: The study analyzed structural properties of the corpus, specifically examining "label purity" across rule identifiers, alert descriptions, and decoders. Results indicated that both rule_id and description exhibited 100% label purity in the labelled subset, confirming that these fields act as strong shortcuts to the target class.
3. Evaluation Strategy: To mitigate leakage, the study rejected conventional random train-test splits in favor of grouped cross-validation based on Wazuh rule_id. This ensures that no rule family appears in both training and test folds, forcing the model to generalize across unseen rule templates rather than memorizing specific alert patterns.
4. Baseline Model: A nearest-neighbour semantic similarity approach was used. Alerts were represented as the concatenation of normalized description text and decoder names, encoded using the all-MiniLM-L6-v2 sentence-transformer. Classification was performed via 1-nearest-neighbour (1-NN) using cosine similarity.
5. Metrics: Performance was evaluated at both the Tactic and Technique levels using Accuracy, Macro F1, and Weighted F1 to account for class imbalance.

Key Results
The study demonstrates that while ATT&CK-aligned signal exists within Wazuh alerts, its recoverability is highly dependent on the evaluation constraints:

• Impact of Leakage Control: Under grouped cross-validation (by rule_id), performance metrics were significantly more modest than what would be expected from naïve splits, confirming that previous high-performance claims in similar contexts may have been artifacts of structural leakage.
• Tactic vs. Technique Stability: Tactic-level mapping proved more robust than technique-level mapping.
  • Tactic Level: Achieved 0.5444 Accuracy and 0.4205 Macro F1 across 4 classes.
  • Technique Level: Achieved 0.4428 Accuracy and 0.2634 Macro F1 across 8 classes.
• Class-Specific Performance: Techniques associated with distinctive authentication patterns (e.g., Password Guessing, Brute Force) remained recoverable even under strict grouping. Conversely, techniques with sparse representation or overlapping structural properties (e.g., Account Access Removal, Scheduled Task) collapsed under leakage-aware evaluation.
• Confusion Patterns: Confusion matrices revealed that while Credential Access was reliably recovered, significant confusion existed between Impact and Defense Evasion, and Valid Accounts frequently acted as a dominant predicted class for other techniques.

Key Contributions

1. Leakage-Aware Evaluation Framework: The paper establishes a methodological standard for evaluating ATT&CK mapping on rule-generated alert corpora. It argues that grouped cross-validation by rule identifier is a necessary condition to distinguish genuine behavioral signal from template memorization.
2. Characterization of Structural Leakage: The study empirically demonstrates that Wazuh alert corpora contain strong structural leakage channels (specifically rule_id and description), which can artificially inflate predictive performance if not controlled.
3. Operational Validity of Alert Mapping: The research confirms that host intrusion alerts can support ATT&CK-aligned behavioral interpretation, but only when framed as a leakage-aware problem. It suggests that tactic-level abstraction offers a more stable and operationally useful starting point for cyber risk interpretation than fine-grained technique-level classification in this context.

Significance and Claims
The paper claims that its primary significance lies not in achieving high predictive accuracy, but in demonstrating a rigorous, operationally defensible approach to evaluating security telemetry. The author argues that without leakage-aware evaluation, studies using structured alert corpora risk producing results that reflect dataset artifacts rather than transferable behavioral evidence.

The findings suggest that:

• Naïve evaluation is insufficient: Standard random splits are methodologically unreliable for rule-driven alert datasets because they test template memorization.
• Signal is selective: ATT&CK signal in host alerts is real but uneven; it is strong for behaviors with distinctive semantic patterns (like brute-force attacks) but weak for behaviors that rely on broader context or sparse examples.
• Methodological Prerequisite: Leakage assessment must be treated as a first-order requirement in cybersecurity NLP research, particularly when dealing with datasets generated by detection engines where structural fields may encode target labels.

The study concludes that while Wazuh alerts can contribute to ATT&CK-aligned cyber risk analysis, their utility is contingent upon explicitly addressing duplication, structural leakage, and class imbalance through rigorous evaluation protocols.