Technical Summary: SAMD – A Tool for Identifying False Data Injection Scenarios in AI/ML-enabled Medical Devices

1. Problem Statement

The integration of Artificial Intelligence (AI) and Machine Learning (ML) into medical devices offers significant benefits, such as automated remote monitoring and enhanced diagnostic capabilities. However, this integration introduces critical security risks, specifically false data injection attacks during the inference phase. Unlike traditional safety-critical domains (e.g., automotive or industrial control) where a single manufacturer typically controls the entire system stack, AI/ML-enabled medical systems often rely on heterogeneous, multi-vendor peripheral devices (e.g., glucose meters, cameras) that feed data into a central ML model.

The core challenge is that the final integration of ML software with these peripherals often occurs post-deployment by users or healthcare providers who lack cybersecurity expertise. Consequently, vulnerabilities in peripheral devices (e.g., communication protocols, firmware, or operating systems) can be exploited to inject malicious data, leading to incorrect ML predictions, misdiagnoses, and potentially fatal treatment errors. Existing security efforts focus on hardening the ML model itself or analyzing the software stack, but few address the systemic risk of exploiting known vulnerabilities in interconnected peripheral devices to compromise the ML engine. Furthermore, manual identification of these cross-component attack paths is difficult due to the complexity of the attack surface and the limited cybersecurity expertise among medical ML developers.

2. Methodology: SAMD

To address these challenges, the authors introduce SAMD (STPA-Sec for AI/ML-enabled Medical Devices), an automated tool designed to perform System Theoretic Process Analysis for Security (STPA-Sec). SAMD models the medical system as a control structure, treating all components as potential vectors for false data injection. The tool automates the discovery of vulnerabilities and the generation of attack scenarios through a three-stage workflow:

A. Technology Identifier

This module extracts information about the technologies used by system components (hardware, software, and peripherals) from publicly available device documentation (e.g., FDA submissions, user guides).

• Technique: It employs a combination of Named Entity Recognition (NER) models (GliNER and GPT-4o-based NER) to identify specific technology factors such as communication protocols, encryption methods, firmware, operating systems, and external libraries.
• Hallucination Mitigation: To ensure accuracy, the module performs exact word-matching checks against the source documents, merging outputs from both NER models and removing any keywords not explicitly present in the documentation.

B. Vulnerability Finder

Once technologies are identified, this module queries the MITRE Common Vulnerability Enumeration (CVE) database to retrieve the top-$N$ (set to 10) most recent CVE records associated with those technologies.

• Relevance Filtering: An LLM is prompted with the device description and the CVE description to determine if the vulnerability could plausibly be exploited to inject, modify, or spoof data used by the ML component. The LLM is instructed to output a strict "YES" or "NO" based on whether the vulnerability allows data tampering, filtering out those limited to information disclosure or Denial of Service (DoS).

C. Attack Scenario Generator

This module synthesizes the filtered vulnerabilities, system control structures, and known adversarial steps to generate detailed attack scenarios.

• Prompt Engineering: The LLM is assigned the role of a security engineer and provided with a structured prompt containing the system description, data flow, the specific ML technique, the targeted technology, and the CVE details.
• Structured Output: To reduce variability and hallucinations, the LLM is constrained to generate attack steps following a specific five-stage chain-of-thought framework:
  1. Reconnaissance
  2. Gaining access
  3. Privilege escalation
  4. Attack execution
  5. Impact
• Goal: The output details how an adversary exploits a specific vulnerability in a peripheral device to manipulate data entering the ML engine, resulting in specific patient safety impacts (e.g., incorrect insulin dosage).

3. Key Contributions

The paper outlines three primary contributions:

1. Development of SAMD: The first automated tool to assist STPA-Sec analysis for AI/ML-enabled medical devices. It automatically identifies target device technologies, retrieves associated vulnerabilities from public databases, and generates specific attack scenarios with detailed execution steps.
2. Empirical Evaluation: The authors implemented SAMD using three state-of-the-art LLMs (GPT-4, GPT-4o, and Llama 3) and applied it to five real-world, FDA-cleared medical devices across diverse specialties:
   • d-Nav: Insulin dose recommendation app.
   • ABMD: Bone mineral density measurement from CT scans.
   • IDx-DR v2.3: Diabetic retinopathy detection from retinal images.
   • KIDScore D3: Automated embryo analysis for IVF.
   • Oxehealth Vital Signs: Non-invasive pulse and breathing rate measurement via video.
3. Rigorous Assessment: The tool's performance was evaluated through manual review and an "LLM-as-a-judge" method to assess the correctness and relevance of generated attack steps.

4. Experimental Results

The evaluation of SAMD yielded the following quantitative results:

• Technology Identification: SAMD achieved 100% precision in identifying device technologies mentioned in the input documentation. (Note: The tool can only identify technologies explicitly present in the provided documents; confidential manufacturer data not in public docs was not identified, but this is a limitation of the input, not the tool).
• Vulnerability Discovery: SAMD retrieved known vulnerabilities linked to the identified technologies with 63.2% precision. Out of 165 extracted CVEs, 36 were manually verified as truly exploitable for false data injection.
• Attack Scenario Generation: The generated attack scenarios demonstrated 95.3% average accuracy (measured by two independent LLM judges, GPT-o3 and Gemini-2.5-Pro) in terms of correctness, detail, and relevance to the system.
• LLM Performance: Among the tested models, GPT-4o consistently outperformed GPT-4 and Llama 3, providing the most correct steps (96.2%), highest level of detail, and best additional insights regarding security recommendations and patient impact.
• Efficiency: SAMD significantly reduced analysis time. For a device with 54 CVEs (22 relevant), manual analysis was estimated to take ~5.5 hours, whereas SAMD completed the task in approximately 3 minutes (maximum runtime observed was 191.64s for IDx-DR v2.3).

5. Significance and Claims

The authors position SAMD as a proactive security tool for the design and post-deployment phases of medical device lifecycles.

• For Manufacturers: SAMD enables the identification of potential attack paths and vulnerable peripheral technologies before commercialization, allowing for the implementation of countermeasures or the issuance of advisories to end-users regarding secure deployment environments.
• For Security Analysts: It provides a systematic method to analyze publicly available information on medical devices to generate security recommendations and identify hazards that might otherwise be overlooked due to the complexity of multi-vendor systems.
• Limitations and Modesty: The paper explicitly states that SAMD is intended to support, not replace, human security analysts. Due to the high safety-criticality of healthcare, the authors recommend a final human review of the generated scenarios. They acknowledge that the tool's effectiveness is bounded by the completeness of the input documentation and the current capabilities of LLMs, though they note that hallucination rates were low in the tested models (specifically GPT-4o).

In summary, SAMD bridges the gap between peripheral device vulnerabilities and ML model safety, offering an automated, structured approach to anticipating false data injection attacks in complex, heterogeneous medical ecosystems.