LLM-Assisted Detection and Repair of Hardware Security Vulnerabilities in Verilog Designs

Technical Summary: LLM-Assisted Detection and Repair of Hardware Security Vulnerabilities in Verilog Designs

Problem Statement
Hardware designs, particularly those described in Verilog at the Register Transfer Level (RTL), are susceptible to security vulnerabilities that, once fabricated into silicon, become permanent and unpatchable. Unlike software, where bugs can be updated, hardware flaws can lead to unauthorized data exposure, privilege escalation, and unauthorized access. While Large Language Models (LLMs) have shown promise in assisting with RTL code repair and testbench creation, they currently face significant challenges in hardware security analysis. These include a lack of domain-specific knowledge, inherent biases against hardware design, and the "under-constrained repair" problem, where LLMs propose fixes that are syntactically correct but functionally irrelevant or alter the intended design behavior. Furthermore, existing methods often struggle with long context processing and the complexity of tracing vulnerabilities over time.

Methodology
The authors propose a structured, iterative framework to leverage LLMs for detecting and repairing Common Weakness Enumerations (CWEs) in single-module Verilog designs. The methodology targets specific hardware weaknesses identified in MITRE's 2025 list of most important hardware vulnerabilities (excluding microarchitectural issues). The process follows a seven-stage pipeline:

1. Module Classification: The LLM identifies the module type and features (e.g., JTAG interfaces, debug modes) to narrow the scope of potential CWEs, preventing information overload.
2. Asset Identification: The model identifies critical assets, such as cryptographic keys, privilege boundaries, clock/reset schemes, and Finite State Machine (FSM) states.
3. Dependency Graph Analysis: The LLM generates a Program Dependency Graph (PDG) to model data and control flow. It performs reachability, dominance, and taint propagation analyses to understand design behavior and potential data leaks.
4. CWE-Driven Review: Using the outputs from previous steps and specific CWE guides (containing descriptions, common causes, and checklists), the LLM performs a targeted review to identify specific vulnerabilities.
5. Testbench Generation: Based on identified vulnerabilities and CWE-specific secure design rules, the LLM generates a testbench to verify the presence of flaws.
6. Simulation: The generated testbench is compiled and run against the RTL design.
7. Code Repair: If tests fail, the LLM attempts to repair the code using the failed test cases and CWE design rules as constraints. This cycle repeats up to three times.

The framework was evaluated on a dataset of 32 single-module Verilog designs, including 27 with known vulnerabilities and 5 intentionally secure modules. Microsoft Copilot was used as the LLM for the evaluation.

Key Results
The study yielded mixed results, highlighting both the potential and current limitations of LLMs in this domain:

• Strengths: The LLM demonstrated strong capabilities in module classification, asset identification, and reasoning about structural and behavioral aspects of the design (via PDG analysis). It correctly identified the functional type of modules in nearly all cases and often recognized vulnerabilities before the formal CWE-driven review.
• Weaknesses:
  • Testbench Generation: This was the greatest weakness. Generated testbenches frequently failed to validate security properties, often due to incorrect Device Under Test (DUT) usage, incomplete test cases, or failure to reset the DUT between tests.
  • False Positives on Secure Designs: In the five non-vulnerable modules, the LLM exhibited a high false-positive rate, incorrectly flagging secure designs as vulnerable. It often recommended unnecessary security features (e.g., lock bits, privilege controls) that altered the intended functionality.
  • Repair Limitations: While the LLM could generate syntactically correct patches, repairs sometimes modified the original design's intended functionality. The model tended to favor security enhancements over preserving original behavior, leading to "over-engineering" (e.g., adding access control to modules that did not require it).
• Success Rate: Out of 32 total tests, 27 resulted in a pass, yielding an 84% success rate. This success rate primarily reflects the 27 modules containing known vulnerabilities that were successfully identified and repaired. However, the 5 non-vulnerable modules, which were intended to test the AI's ability to distinguish secure designs, failed this specific objective. The AI incorrectly identified vulnerabilities in all five secure modules, resulting in a high false-positive rate. For these non-vulnerable cases, code repair was not performed because the identified issues were not actual vulnerabilities; modifying these designs would have altered their intended functionality. Several failures in the vulnerable set were also attributed to the inability to generate meaningful verification results (e.g., uninitialized outputs) or the failure to correctly classify the module's primary function, leading to the omission of relevant CWEs.

Significance and Claims
The paper claims that its proposed methodology demonstrates the potential of LLMs to augment traditional hardware security analysis by providing automated, scalable assistance during the design process. The authors emphasize that their approach helps bridge the gap between LLM capabilities and the rigorous requirements of hardware security by structuring the analysis into manageable steps (classification, asset identification, graph analysis).

However, the authors are modest in their conclusions, acknowledging that the current methodology is not a fully autonomous solution. They state that the results highlight the necessity for:

1. Refined Prompting: To reduce hallucinations and constrain repairs to the specific vulnerability without altering intended functionality.
2. Explicit Context: Providing the LLM with explicit information regarding the module's purpose and expected behavior to improve classification and repair accuracy.
3. Further Evaluation: The need to test the methodology on a larger, more diverse set of CWEs and RTL designs to assess generalizability.

The paper concludes that while LLMs show promise in interpreting module behavior and identifying assets, their reliability in RTL reasoning, testbench generation, and vulnerability verification remains limited by factors such as domain knowledge gaps and under-constrained repair prompts. The work serves as a guide for developing and fine-tuning future LLM-based hardware security tools.