Operationalising Cyber Risk Management Using AI: Connecting Cyber Incidents to MITRE ATT&CK Techniques, Security Controls, and Metrics

This paper presents a novel framework and a fine-tuned sentence-transformer model that leverages Natural Language Processing to automatically map cyber incidents to MITRE ATT&CK techniques, CIS Critical Security Controls, and SMART metrics via the "Cyber Catalog," thereby enabling resource-constrained organizations to operationalize evidence-based cyber risk management with high accuracy.

The Gist

Imagine you are the security guard of a small shop. One day, a thief breaks in, smashes a window, and steals a cash register. You write down what happened: "Someone jumped the fence, broke the glass, and took the register."

Now, imagine you have to explain this to a team of experts who speak a very specific, complex language about how thieves operate. They don't just want to know "a thief broke in"; they want to know the specific technique used (e.g., "Fence Jumping" or "Glass Smashing"). Once they know the technique, they can tell you exactly which lock or alarm to buy to stop it next time.

The Problem:
For most small businesses, this process is a nightmare. It takes a long time, requires expensive experts, and is often done by hand. If you don't have a team of experts, you might buy the wrong lock, or worse, not know you need one at all.

The Solution (This Paper):
This research presents a "smart assistant" (an AI) that acts as a translator and a guide. It takes your messy, everyday description of a break-in and instantly translates it into the expert language, then tells you exactly what to do about it.

Here is how they built it, using simple analogies:

1. The "Cyber Catalog" (The Master Recipe Book)

The researchers built a massive, organized library called the Cyber Catalog. Think of it as a giant cookbook that connects three things:

• The Crime: How the bad guys attack (using a standard list called MITRE ATT&CK).
• The Defense: The locks, alarms, and rules you can use (called CIS Controls).
• The Scorecard: How to measure if your defense actually works (called SMART Metrics).

Before this, these three things were in different rooms. You had to run back and forth to connect them. This Catalog puts them all on one shelf, so if you see a "Glass Smashing" technique, the book instantly tells you: "Buy a reinforced window (Control) and check if it's installed 100% of the time (Metric)."

2. The "Smart Translator" (The AI Model)

To make the connection automatic, they trained a computer brain (an AI) to read security incident reports.

• The Challenge: The AI needed to learn the difference between "someone broke a window" and "someone smashed a window." To a human, they are the same. To a computer, they are different words.
• The Training: They started with a smart but generic AI (like a student who knows general English). They taught it specifically about cyber crimes.
• The "Fake" Homework: They didn't have enough real examples of break-ins to teach the AI well. So, they used another AI (a creative writer) to invent 75,000 new fake break-in stories that sounded real but used different words. This was like giving the student 75,000 practice exams instead of just a few.
• The "Tricky" Questions: To make the AI really sharp, they gave it "hard negative" examples. Imagine showing the AI a picture of a cat and asking, "Is this a dog?" Then showing a picture of a very fluffy cat and asking, "Is this a dog?" The AI had to learn the tiny differences. This stopped it from making lazy guesses.

3. The Results (The "Aha!" Moment)

When they tested their new AI against the old, generic ones, the results were like comparing a novice guard to a master detective:

• The Old AI: Got about 58% of the connections right. It was okay, but it often missed the nuance.
• The New AI: Got about 79% right. That might not sound like a huge jump, but in the world of AI, it's a massive leap. It means the AI is now reliable enough to trust with real security decisions.
• The Error Rate: The old AI made big mistakes often. The new AI was incredibly consistent, rarely making a huge error.

Why This Matters for Everyone

This isn't just for giant tech companies with armies of security experts.

• For Small Businesses: It levels the playing field. A small shop owner can now use this tool to understand complex threats and buy the right protection without hiring a $200/hour consultant.
• Speed: Instead of a human spending hours reading a report and cross-referencing manuals, the AI does it in seconds.
• Proof: It doesn't just say "we are safe." It gives you a scorecard to prove to your boss or insurance company that your security measures are actually working.

The Bottom Line

The researchers built a universal translator for cyber security. They took the messy, confusing world of "hacker stories," translated it into a clear list of "bad guy moves," and then immediately handed you the "fix-it kit" and a "report card" to prove it worked. They made the complex simple, the slow fast, and the expensive accessible.

Technical Summary

1. Problem Statement

The paper addresses critical bottlenecks in modern cyber risk management, particularly for Small and Medium-sized Enterprises (SMEs) with limited resources:

• Manual Inefficiency: Traditional incident analysis requires analysts to manually map unstructured incident descriptions to structured adversary techniques (MITRE ATT&CK) and then to defensive measures (CIS Critical Security Controls). This process is time-consuming, prone to human error, and does not scale with incident volume.
• Lack of Objective Metrics: Security control assessment often relies on subjective judgment rather than quantifiable data. The absence of SMART (Specific, Measurable, Achievable, Relevant, Time-bound) metrics makes it difficult to prove the effectiveness of security investments or justify expenditures.
• Fragmented Knowledge: Threat intelligence (ATT&CK), defensive controls (CIS), and performance metrics exist in silos, lacking a unified framework to connect them automatically.

2. Methodology

The authors propose a comprehensive framework involving a new knowledge base and a fine-tuned AI model.

A. The "Cyber Catalog" (Knowledge Base)

A novel, unified knowledge base integrating three layers:

1. Foundation: 18 CIS Critical Security Controls (v8) and their 153 Safeguards.
2. Threat Layer: MITRE ATT&CK techniques mapped bidirectionally to CIS Safeguards.
3. Metrics Layer: Quantifiable, SMART metrics linked to each Safeguard (Inputs, Operations, Measures, Formulas) to enable objective monitoring.

• Output: A structured CSV dataset enabling automated analysis and integration into security orchestration platforms.

B. Dataset Preparation

• Base Data: 762 high-quality incident-technique pairs from the European Repository of Cyber Incidents (EuRepoC), manually annotated by experts.
• Synthetic Augmentation: To overcome data scarcity, the authors used GPT-5 to generate 100 semantically similar but lexically distinct variations for each original incident, creating ~76,200 synthetic examples.
• Quality Assurance:
  • Filtering: Manual review removed 1,214 low-quality/duplicate entries.
  • BERTScore Validation: Used to ensure semantic fidelity between original and augmented text (F1 threshold > 0.75).
  • Final Dataset: 74,986 incident-technique pairs (80% train, 10% val, 10% test).

C. Model Architecture & Training

• Base Model: all-mpnet-base-v2 (a sentence transformer combining BERT and XLNet advantages).
• Training Strategy:
  • Loss Function: Modified Multiple Negatives Ranking Loss (MNRL).
  • Duplicate-Aware Handling: Implemented NoDuplicatesDataLoader to prevent "false negatives" where multiple incidents map to the same technique within a single batch.
  • Hard Negative Mining: Used GPT-5 to identify semantically similar but incorrect techniques (hard negatives) to force the model to learn fine-grained distinctions.
• Hyperparameters: Learning rate $2 \times 10^{-5}$, batch size 16, 10 epochs, trained on an NVIDIA Quadro RTX 5000.

3. Key Contributions

1. The Cyber Catalog: A publicly available, structured knowledge base linking CIS Controls, MITRE ATT&CK, and SMART metrics, bridging the gap between threat intelligence and operational defense.
2. Advanced Data Augmentation Pipeline: A rigorous methodology using LLMs for synthetic data generation combined with BERTScore filtering and hard negative mining to create a high-quality training corpus for a specialized domain.
3. Domain-Specific Fine-Tuning: Demonstrated that fine-tuning general-purpose sentence transformers on cyber-specific data yields significant performance gains over baseline models.
4. Operational Framework: An end-to-end pipeline for automated incident triage, control mapping, and evidence-based risk assessment.

4. Results

The fine-tuned model (ft_mpnet_v6) significantly outperformed top baseline models (all-mpnet-base-v2, all-distilroberta-v1, all-MiniLM-L12-v2).

• Correlation Metrics:
  • Spearman Rank Correlation ($\rho$): 0.7894 (vs. ~0.58 for baselines), representing a ~37% relative improvement. This indicates "very strong" monotonic alignment.
  • Pearson Correlation ($r$): 0.8756, indicating a strong linear relationship.
• Error Metrics:
  • Mean Absolute Error (MAE): 0.1352 (vs. ~0.50 for baselines), a ~67% reduction.
  • Mean Squared Error (MSE): 0.0272 (vs. ~0.28 for baselines), indicating significantly fewer large outliers.
• Distribution Analysis: The fine-tuned model exhibited a narrow error distribution concentrated near zero, demonstrating high consistency and stability compared to the volatile performance of baseline models.

5. Significance and Impact

• Operational Efficiency: Drastically reduces the time required for incident triage and response planning, making it feasible for resource-constrained SMEs to handle high-volume incidents.
• Evidence-Based Decision Making: By linking incidents to controls and quantifiable metrics, organizations can objectively measure the effectiveness of their security posture, identify gaps, and prioritize investments based on data rather than intuition.
• Scalability: The automated approach scales to handle the growing volume of cyber incidents, addressing the shortage of skilled security analysts.
• Future Integration: The authors propose integrating this model into open-source Host Intrusion Detection Systems (HIDS) via plugins for real-time, threat-informed defense, and expanding the catalog to include NIST SP 800-53 controls for broader regulatory compliance.

Conclusion: The paper successfully demonstrates that combining a structured knowledge base (Cyber Catalog) with domain-specific fine-tuning of sentence transformers creates a robust, automated system for connecting cyber threats to defensive actions and measurable outcomes, effectively bridging the gap between threat intelligence and operational security management.