Beyond Single Reports: Evaluating Automated ATT&CK Technique Extraction in Multi-Report Campaign Settings

1. Problem Statement

Cyber Threat Intelligence (CTI) reports are essential for understanding attack campaigns, yet large-scale attacks (campaigns) are documented across multiple reports from diverse sources (e.g., government agencies, vendors, researchers). These reports often provide complementary information: some offer high-level strategic overviews, while others provide granular forensic details.

• The Gap: Existing automated methods for extracting MITRE ATT&CK techniques from CTI reports are primarily evaluated on single-report datasets. This approach leads to a fragmented view of campaign behavior, missing techniques that are only mentioned in specific reports.
• The Consequence: Missing a technique means missing the associated mitigation controls, leaving organizations exposed to undetected threats.
• Research Goal: The study aims to evaluate whether aggregating multiple reports for a single campaign improves extraction performance compared to single-report analysis and to quantify the impact of these errors on control coverage.

2. Methodology

The authors conducted a conceptual replication and extension of a prior systematic study by Büchel et al. [10], moving the evaluation context from single reports to a multi-report campaign setting.

Dataset Construction

• Training Data: The AnnoCTR dataset (118 techniques) was used for training/fine-tuning. To ensure coverage of techniques missing in AnnoCTR but present in the test set, 51 additional samples were manually curated from the official MITRE ATT&CK knowledge base.
• Testing Data: The CTIfecta dataset, comprising 90 CTI reports across three major attack campaigns:
  • SolarWinds (21 reports)
  • XZ Utils (28 reports)
  • Log4j (41 reports)
• Filtering: Reports were filtered to ensure they contained at least one ATT&CK technique but excluded those explicitly listing ATT&CK IDs to prevent trivial mapping.

Evaluated Approaches

The study evaluated 29 state-of-the-art methods spanning three categories:

1. Named Entity Recognition (NER): 6 rule-based/pattern-driven methods (e.g., TTPDrill, AttacKG) using syntactic pipelines (tokenization, POS tagging, lemmatization).
2. Encoder-based Classification: 15 methods using pre-trained models (BERT, RoBERTa variants like CySecBERT, SciBERT, DarkBERT) for multi-label text classification.
3. Decoder-based LLMs: 8 methods using Large Language Models (Llama-3.1-8B) with various configurations:
   • Zero-shot/Few-shot prompting (RAW, FSP).
   • Retrieval-Augmented Generation (RAG).
   • Supervised Fine-Tuning (SFT) and combinations (FSP+RAG).

Evaluation Metrics & Analysis

• Aggregation Strategy: Predictions from individual reports were aggregated at the campaign level (union of predicted techniques) and compared against the ground truth of the entire campaign.
• Performance Metrics: Precision, Recall, and F1-score (macro-averaged).
• Misclassification Analysis: Used text embeddings and t-SNE to analyze if semantic similarity between technique descriptions caused errors.
• Control Coverage: Mapped extracted techniques to P-SSCRM (Proactive Software Supply Chain Risk Management) controls to quantify the "mitigation gap."
• Saturation Analysis: Used a greedy incremental approach to determine the minimum number of reports required to reach performance saturation (defined as <0.05 improvement in metrics).
• Report Characteristics: Analyzed word count, sentence count, readability scores (Flesch), publication date, and vendor source.

3. Key Contributions

1. Campaign-Level Replication: The first systematic evaluation of 29 extraction methods across multiple reports for the same campaign, moving beyond the single-report paradigm.
2. Control-Level Metrics: Introduced metrics to quantify how extraction errors propagate into missed mitigation controls (P-SSCRM), highlighting the real-world security impact.
3. Performance Saturation Thresholds: Empirically determined that most methods saturate after 5–15 reports, providing a cost-benefit guideline for data collection.
4. Report Characterization: Identified that technical density and length (not readability) are the primary drivers of extraction performance.
5. Semantic Error Analysis: Characterized the specific nature of misclassifications, revealing that semantic overlap and shared tactics are major sources of error.

4. Key Results

RQ1: Multi-Report vs. Single-Report Performance

• Overall Improvement: Aggregating multiple reports improved the F1-score by approximately 26% compared to single-report analysis.
• Method Divergence:
  • Decoder-based LLMs: Showed the most significant improvement (F1 increased from 47.7 to 60.1, +26.1%).
  • NER: Showed consistent improvement (F1 increased from 56.8 to 65.2, +14.8%).
  • Encoder-based Classification: Surprisingly, performance degraded significantly (F1 dropped from 59.3 to 41.5, -30.0%) when moving to multi-report settings, likely due to noise accumulation in classification tasks.

RQ2: Misclassification & Semantic Similarity

• Semantic Overlap: 33.3% of misclassifications occurred between semantically similar techniques (cosine similarity ≥ 0.7).
• Shared Tactics: Up to 79.2% of false positives and 73.3% of false negatives involved techniques sharing the same MITRE tactic (e.g., Defense Evasion, Discovery).
• Specific Errors: Techniques like T1587 (Develop Capabilities) and T1588 (Obtain Capabilities) were frequently confused.

RQ3: Impact on Control Coverage

• The Mitigation Gap: Even the best-performing method (Decoder-based LLM with RAG) achieved only 77.1% control coverage on the SolarWinds campaign.
• Disproportionate Impact: A 10% error rate in technique extraction resulted in a **23% gap in control coverage**.
• Systemic Gaps: All methods consistently missed controls in the Governance (G), Deployment (D), and Environment (E) domains of the P-SSCRM framework.

RQ4: Performance Saturation

• Threshold: Most approaches reached performance saturation after 5 to 15 reports.
• Exceptions: The XZ Utils campaign required up to 22–30 reports for LLMs to saturate, suggesting some campaigns are inherently more complex or fragmented.
• Precision vs. Recall: NER achieved perfect precision after just 1 report but lower recall. LLMs required more reports to maximize recall.

RQ5: Report Characteristics

• Length & Technicality: Longer reports with high technical detail (hashes, IPs, specific payloads) consistently performed better, regardless of low readability scores.
• Readability: Readability scores (Flesch) showed no correlation with performance; in fact, "harder to read" reports often contained the critical technical data needed for extraction.
• Source: Reports from government agencies (e.g., CISA) and established vendors (e.g., Google, Apache) contributed the most to performance saturation.

5. Significance and Implications

• Paradigm Shift: The study advocates for a shift from single-report evaluation to multi-report campaign analysis in CTI research. Relying on a single report is insufficient for comprehensive threat modeling.
• Practical Guidelines: Security teams should aim to aggregate 10–15 high-quality reports per campaign to achieve diminishing returns. They should prioritize reports from authoritative sources that are technically dense, even if they are difficult to read.
• Method Selection:
  • SOC/Incident Response: May prefer NER for high precision (fewer false alarms).
  • Threat Intelligence: Should prefer Decoder-based LLMs for high recall and comprehensive coverage, accepting a higher false-positive rate.
• Control Awareness: Extracting techniques is not enough; researchers must evaluate the downstream impact on mitigation controls. A small gap in technique detection can lead to significant gaps in defensive posture.

In conclusion, the paper demonstrates that while automated extraction is improving, it remains limited by semantic ambiguity and the fragmented nature of CTI reporting. Aggregating multiple reports is a necessary step to bridge the gap between theoretical technique detection and practical control implementation.