Technical Summary: AegisShield – Democratizing Cyber Threat Modeling with Generative AI

Problem Statement

Traditional threat modeling methodologies (e.g., STRIDE, PASTA, OCTAVE) face significant barriers to adoption, particularly for small and under-resourced organizations. These barriers include high complexity, substantial time requirements, reliance on specialized expertise, and the inability to scale effectively with modern, dynamic systems such as cloud environments, IoT, and Cyber-Physical Systems (CPS). While frameworks like MITRE ATT&CK and STRIDE provide structured approaches, their manual implementation often results in incomplete assessments, inconsistent outputs, and a "skills gap" where only a fraction of organizations feel confident in their threat coverage. Furthermore, existing AI-focused frameworks often rely on unrealistic assumptions about attacker capabilities or fail to integrate real-time threat intelligence, limiting their practical utility in diverse, real-world scenarios.

Methodology

This research employed a Design Science Research Methodology (DSRM) to develop and evaluate AegisShield, a Python-based, generative AI-enhanced threat modeling tool deployed via the Streamlit framework.

Tool Development

• Architecture: AegisShield integrates OpenAI's GPT-4o model with established cybersecurity frameworks (STRIDE, MITRE ATT&CK, DREAD) and threat intelligence sources (National Vulnerability Database [NVD], AlienVault Open Threat Exchange [OTX]). To optimize performance and minimize API dependency, the tool utilizes a locally stored STIX JSON file for MITRE ATT&CK data.
• Prompt Engineering: The system utilizes a structured prompt architecture designed to simulate expert decision-making. Prompts are engineered to enforce JSON-structured outputs, ensuring consistency and reducing hallucinations. The prompts cover five key components:
  1. MITRE ATT&CK Mapping: Selecting relevant Tactics, Techniques, and Procedures (TTPs) from the locally stored STIX dataset.
  2. Threat Identification: Generating threats across STRIDE categories (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) based on system descriptions and external context (including known vulnerabilities from NVD and OTX).
  3. Risk Assessment: Applying the DREAD model (Damage, Reproducibility, Exploitability, Affected Users, Discoverability) for quantitative risk scoring.
  4. Mitigation Strategies: Generating actionable countermeasures.
  5. Test Cases: Creating Gherkin-syntax test cases for Behavior-Driven Development (BDD).
• Data Collection: The evaluation utilized 15 diverse case studies from academic literature spanning domains such as healthcare, finance, IoT, and 5G. These studies provided expert-developed STRIDE threat models.
• Experimental Design:
  • Dataset: The tool generated 8,100 threats (30 batches × 15 case studies × 18 threats per batch) compared against 243 expert-extracted threats.
  • Metrics:
    • Complexity (H1): Measured via Flesch-Kincaid readability scores.
    • Semantic Similarity (H2): Measured using Cosine Similarity with the stsb-roberta-large Sentence-BERT model.
    • Framework Alignment (H3): Measured by the percentage of threats successfully mapped to MITRE ATT&CK TTPs.
  • Statistical Analysis: Non-parametric tests (Mann-Whitney U) were used for readability due to non-normal data distributions. One-proportion tests and confidence intervals were used for mapping success rates.

Key Contributions

1. Democratization of Threat Modeling: The research demonstrates that generative AI can lower the barrier to entry for threat modeling, making it accessible to organizations lacking dedicated cybersecurity architects.
2. Standardization and Automation: AegisShield automates the generation of structured, standardized threat artifacts (STRIDE threats, DREAD scores, mitigations, and test cases) in minutes, a process that traditionally takes days or weeks.
3. Integration of Threat Intelligence: The tool incorporates data from NVD and OTX to provide contextual threat intelligence within the prompts, ensuring threat models reflect known vulnerabilities and emerging threats relevant to the specific system context.
4. Validation Framework: The study establishes a quantitative framework for evaluating AI-generated threat models against expert benchmarks using semantic similarity and framework mapping metrics.

Results

The evaluation of AegisShield across 15 case studies yielded statistically significant results supporting three primary hypotheses:

• H1: Reduced Complexity: AegisShield-generated threats were significantly more readable than expert-generated threats.
  • Median Flesch-Kincaid Score: Tool-generated (12.7) vs. Expert-generated (13.5).
  • Statistical Significance: Mann-Whitney U test confirmed a significant difference (p < 0.001), indicating the tool produces more accessible descriptions without sacrificing coverage.
• H2: Semantic Similarity: The tool generated outputs semantically aligned with expert models.
  • Success Rate: In 14 out of 15 case studies, at least one threat per batch achieved a cosine similarity score ≥ 0.7 in over 50% of the runs (statistically significant, p < 0.05).
  • High-Performance Cases: Specific cases (e.g., Voice-based Applications, Social Media) achieved similarity scores as high as 0.94, demonstrating the model's ability to replicate expert-level insights.
• H3: MITRE ATT&CK Mapping: The tool successfully mapped threats to established frameworks.
  • Overall Success Rate: 85.4% of the 8,100 generated threats were successfully mapped to MITRE ATT&CK TTPs.
  • Statistical Significance: This rate exceeded the 80% benchmark with statistical significance (p < 0.001).
  • Category Performance: Mapping was highest for Spoofing (94.0%) and Elevation of Privilege (91.6%), while Repudiation (73.5%) showed lower success, often due to the conceptual nature of repudiation threats versus the technical specificity required by the framework.

Significance and Claims

The paper claims that AegisShield represents a viable solution to the "democratization" of cybersecurity. By automating the initial phases of threat modeling, the tool allows organizations to adopt secure-by-design principles without the prohibitive costs of hiring senior architects or engaging in labor-intensive manual processes.

The research posits that while traditional methods are often too complex and slow for modern, agile development lifecycles, AegisShield offers a scalable, cost-effective alternative. The study highlights that the tool not only accelerates threat identification (reducing time from days to minutes) but also produces outputs that are statistically comparable to expert models in terms of semantic meaning and framework alignment.

However, the author maintains a modest tone regarding limitations:

• Domain Variability: Performance varied across domains, particularly in emerging technologies (e.g., IoT, 5G) where MITRE ATT&CK mappings may be less defined.
• AI Limitations: The tool relies on the underlying capabilities of the generative AI model (GPT-4o), and issues such as "hallucinations" (though rare, ~1.15% of unmapped threats) and potential biases in training data remain concerns.
• Scope: The study focused on preemptive threat modeling and did not test incident response or post-breach analysis.

Ultimately, the paper concludes that integrating generative AI with structured frameworks and threat intelligence can significantly enhance the accessibility and effectiveness of threat modeling, thereby strengthening the overall cybersecurity ecosystem.