Technical Summary: Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities

Problem Statement

Connected and Autonomous Vehicles (CAVs) rely on complex interplays of software and hardware components, including sensors, Electronic Control Units (ECUs), and telematics, making them susceptible to security vulnerabilities. These vulnerabilities are currently documented as unstructured plain-text descriptions in the Common Vulnerabilities and Exposures (CVE) database. Security practitioners require structured information regarding affected assets, weakness types, and attack behaviors to effectively mitigate risks. While the Structured Threat Information Expression (STIX) format offers a standardized way to represent this knowledge, converting unstructured CVE descriptions into STIX bundles (comprising STIX Domain Objects [SDO], STIX Relationship Objects [SRO], Common Weakness Enumeration [CWE], and MITRE ATT&CK mappings) is a labor-intensive manual process.

Although Large Language Models (LLMs) possess the capability to understand language and generate structured knowledge, prior studies indicate that LLM-generated STIX outputs often suffer from incomplete objects, incorrect relationships, invalid structures, and hallucinated attributes. Furthermore, existing datasets for STIX generation focus on general cybersecurity text or specific tasks like CWE prediction, lacking a dedicated benchmark for CAV-related CVEs mapped to full STIX bundles. There is a critical need to evaluate open-weight LLMs for this specific domain and to establish a ground-truth dataset to facilitate reliable automated threat intelligence generation.

Methodology

Dataset Construction (CAV-STIXGen)

To address the lack of domain-specific benchmarks, the authors constructed the CAV-STIXGen dataset through a six-step process:

1. Keyword Construction: A domain-specific keyword list was generated using LLMs and refined by the authors to cover CAV components (e.g., LiDAR, CAN bus, V2X) and manufacturers.
2. CVE Collection: 445 candidate CVE records were retrieved from the National Vulnerability Database (NVD) using the keywords.
3. Relevance Filtering: Records were filtered based on inclusion criteria (vehicle components, EV charging, infotainment, etc.) and exclusion criteria (generic web/software vulnerabilities). This resulted in 183 unique, relevant CAV-related CVEs.
4. Annotation: Two authors manually annotated SDOs and SROs for each CVE according to STIX 2.1 specifications, achieving 94% inter-rater agreement.
5. Mapping: CVEs were mapped to CWE categories and MITRE ATT&CK techniques. NVD-provided mappings were verified, and manual analysis was conducted where NVD data was missing.
6. Conversion: Validated annotations were converted into ground-truth STIX 2.1 JSON bundles.

LLM Evaluation

The study evaluated 11 open-weight LLMs (ranging from 4B to 120B parameters) across four categories: general-purpose, cybersecurity-specialized, code-oriented, and large-capacity models. The evaluation involved:

• Input: CVE identifiers and descriptions.
• Strategies: Three prompting strategies were tested:
  • Contextless: Minimal instructions.
  • STIX-guided: Explicit instructions for object types, relationships, and mappings.
  • Dynamic Few-Shot: Guided prompting augmented with five similar ground-truth examples selected by similarity score.
• Parameters: Experiments were run at temperatures of 0, 0.25, 0.75, and 1.0.
• Metrics: Micro-precision, recall, and F1 scores were used for SDO, SRO, and CWE extraction. For MITRE ATT&CK, both Match@1 (at least one correct technique) and Match@All (all correct techniques) were reported.
• Multi-Agent Setup: A configuration using Gemma-4-31B and Codestral-22B was tested to examine if role-specific decomposition (separating extraction, mapping, and validation tasks) improved performance.

Threat Pattern Analysis

The ground-truth dataset was analyzed to identify frequent CWE categories and MITRE ATT&CK techniques. Co-occurrence analysis was performed using FP-Growth (minimum support 0.02) and association rules (minimum confidence 0.30) to identify recurring attack patterns.

Key Contributions

1. CAV-STIXGen Dataset: A manually annotated benchmark dataset mapping 183 CAV-related CVEs to ground-truth STIX 2.1 JSON bundles, including SDOs, SROs, CWE, and MITRE ATT&CK mappings.
2. LLM Evaluation: A comprehensive evaluation of 11 open-weight LLMs for generating structured threat intelligence from unstructured CAV vulnerability descriptions.
3. Prompting Strategy Analysis: A comparative study demonstrating how different levels of task guidance (contextless, guided, few-shot) affect generation quality across different model sizes and types.
4. Threat Pattern Identification: An analysis of the ground-truth data revealing frequent CWE and MITRE ATT&CK co-occurrence patterns specific to the transportation security domain.

Results

Dataset Characteristics

The final dataset contains 183 CVEs, comprising 1,383 SDO instances and 1,395 SRO instances. The most frequent SDO types are infrastructure, attack-pattern, vulnerability, identity, and threat-actor. On average, each CVE contains 7.56 SDOs, 7.62 SROs, 1.15 CWE mappings, and 1.61 MITRE ATT&CK mappings.

LLM Performance

• Prompting Strategy: Dynamic few-shot prompting consistently yielded the best results across all tasks. Contextless prompting resulted in poor performance, particularly for implicit objects and relationships.
• Task Performance:
  • SDO Extraction: High performance was achieved, with top models (e.g., Phi-4, LLaMA-70B, Codestral-22B) reaching F1 scores of 0.94 under dynamic few-shot prompting.
  • SRO Extraction: Performance was lower due to the complexity of identifying correct source/target pairs and relationship types. The best result was an F1 score of 0.63 (Qwen-Coder).
  • CWE Mapping: High accuracy was observed, with several models achieving F1 scores near 0.99 under dynamic few-shot prompting.
  • MITRE ATT&CK Mapping: This remained the most challenging task. While Match@1 scores improved (up to 0.68 for Gemma-4-31B), Match@All scores remained low, indicating difficulty in identifying all valid techniques for a single CVE.
• Model Comparison: No single model excelled at all subtasks. Code-oriented models (Qwen-Coder, Codestral) and large general-purpose models (LLaMA-70B, GPT-120B) generally performed best.
• Multi-Agent Setup: The multi-agent configuration improved SDO extraction for Gemma-4-31B (F1 0.91) but did not consistently outperform the best single-model few-shot results for SRO extraction (Codestral-22B achieved 0.43 F1).

Threat Pattern Analysis

• CWE: The most frequent weaknesses were CWE-787 (Out-of-bounds Write), CWE-284 (Improper Access Control), and CWE-20 (Improper Input Validation). Five of the top 10 CWEs in the dataset overlapped with the 2025 CWE Top 25 list.
• MITRE ATT&CK: The most frequent techniques were T1499 (Endpoint Denial of Service), T1203 (Exploitation for Client Execution), and T1210 (Exploitation of Remote Services).
• Co-occurrence: The most frequent technique pair was T1203–T1499 (appearing in 10 CVEs), followed by T1210–T1499 and T1068–T1548.

Significance and Claims

The paper claims that the CAV-STIXGen dataset provides a necessary foundation for evaluating AI-assisted threat intelligence generation in the transportation domain. The results demonstrate that while open-weight LLMs can effectively extract explicit vulnerability objects and map weaknesses (CWE), they struggle with the graph-level reasoning required for complex relationship extraction and comprehensive attack technique mapping (MITRE ATT&CK).

The authors assert that prompt guidance is critical; specifically, dynamic few-shot prompting significantly outperforms contextless or purely guided approaches by providing target-relevant examples. They conclude that while LLMs can automate parts of the CVE-to-STIX translation, current models require external validation and potentially multi-agent or retrieval-augmented approaches to handle the full complexity of structured threat intelligence, particularly for relationship construction and multi-technique attack mapping. The study positions the dataset and findings as a resource for prioritizing defense in transportation security by identifying recurring threat patterns.