Technical Summary: A Prompt-Based Framework for Loop Vulnerability Detection Using Local LLMs

1. Problem Statement

Loop vulnerabilities represent a critical class of risks in software development, capable of causing infinite execution, resource exhaustion, and logical errors that degrade performance and compromise security. Traditional static analysis tools (e.g., SonarQube, PyLint) and dynamic analysis tools often fail to detect these issues effectively. Static tools rely heavily on syntactic pattern matching, lacking the semantic understanding required to identify context-sensitive flaws such as off-by-one errors, control flow misuse, or subtle logic errors that only manifest under specific runtime conditions. Conversely, dynamic tools require extensive test data and execution environments, leading to high computational costs.

Furthermore, while commercial Large Language Models (LLMs) like ChatGPT and Gemini offer advanced contextual reasoning, their use introduces significant concerns regarding code privacy, latency, and dependency on external APIs. This is particularly problematic for security-critical sectors (e.g., defense, fintech, health) where transmitting source code to third-party vendors is restricted. There is a gap in accessible tools that combine the semantic reasoning of LLMs with the data privacy and operational autonomy of local deployment, specifically tailored for detecting loop-related vulnerabilities through structured prompt engineering.

2. Methodology

The study proposes and evaluates a prompt-based framework designed to detect loop vulnerabilities in Python 3.7+ code using locally deployed LLMs. The research focuses on two small-parameter models: LLaMA 3.2 (3B) and Phi 3.5 (4B). The methodology follows a three-process approach:

Process 1: Manual Baseline Creation

To establish a ground truth, two experienced Python developers independently assessed a set of Python programs containing loop-related vulnerabilities. They focused on three categories:

1. Loop Control and Logic Errors: Including infinite loops, off-by-one errors, control flow misuse, variable reassignment, and dead code.
2. Security Risks Inside Loops: Including data leakage, timing-based side-channels, missing authorization, insecure deserialization, unvalidated loop bounds, and unsafe file/network operations.
3. Resource Management and Efficiency Issues: Including recomputing invariant values, unnecessary object creation, inefficient string concatenation, and redundant I/O.

Discrepancies between the developers were reconciled to create a validated baseline, minimizing cognitive bias and ensuring high accuracy for subsequent evaluation.

Process 2: Automated Vulnerability Extraction

The selected local LLMs were guided via iterative prompt engineering to function as code optimization assistants. The framework utilizes a structured system prompt divided into five sections (S1–S5) covering:

• System identity and capabilities.
• Core functional responsibilities.
• Constraints and guardrails (including hallucination prevention).
• Target detection categories.
• Output format definition.

Key safeguarding features in the prompt design included language-specific awareness (Python), code-aware grounding, version sensitivity, and explicit instructions to prevent hallucinations. The models were tasked with identifying vulnerabilities within specific code blocks, with outputs captured as raw detection results (DR1 for LLaMA, DR2 for Phi).

Process 3: Output Validation

The detection results from both models were validated against the manually established baseline by the same two developers. Results were classified into:

• True Positives (TP): Detected issues matching the baseline in location and type.
• False Positives (FP): Detected issues not present in the baseline.
• False Negatives (FN): Baseline issues missed by the model.

3. Key Contributions

The study makes two primary contributions:

1. Structured Framework: It introduces a structured, prompt-based methodology for utilizing local LLMs to detect loop-related code vulnerabilities, addressing the privacy and latency limitations of commercial models.
2. Performance Evaluation: It provides an empirical evaluation of small-parameter local models (LLaMA 3.2 and Phi 3.5) across distinct categories of loop vulnerabilities, demonstrating the efficacy of iterative prompt engineering in enhancing detection accuracy.

4. Results

The study evaluated the models using Precision, Recall, and F1-score metrics. The results indicate that Phi 3.5 (4B) consistently outperformed LLaMA 3.2 (3B) across all three vulnerability categories:

• Loop Control and Logic Errors: Phi achieved an average F1-score of 0.90, compared to LLaMA's 0.83.
• Security Risks Inside Loops: Phi achieved an average F1-score of 0.90, compared to LLaMA's 0.86.
• Resource Management and Efficiency Issues: Phi achieved an average F1-score of 0.95, compared to LLaMA's 0.90.

Specifically, Phi demonstrated superior precision and recall, particularly in detecting efficiency-related issues. While LLaMA performed competitively, it showed marginally reduced precision and recall compared to Phi.

5. Significance and Limitations

The findings emphasize that effective prompt engineering is critical for enabling local LLMs to perform secure and accurate code vulnerability analysis. The study demonstrates that local models can offer a viable alternative to commercial APIs, providing data privacy and offline analysis capabilities while maintaining high detection accuracy for semantic loop flaws.

The authors acknowledge specific limitations:

• Concurrency Issues: The current framework cannot detect concurrency and synchronization issues, which require temporal reasoning and dynamic code analysis techniques not suitable for the static, prompt-based approach used.
• Scope: The study is limited to Python 3.7+ and does not yet support other programming languages.
• Future Work: The authors suggest future research could focus on detecting concurrency issues, supporting additional languages, comparing against larger code-specific models (e.g., CodeBERT, CodeT5), and integrating the framework into real-time IDEs for practical deployment.

In conclusion, the paper posits that while traditional tools struggle with semantic loop errors, a well-structured prompt-based framework using local LLMs can significantly improve the early detection of subtle but critical coding issues in a secure and efficient manner.