💻 computer science

Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability Detection

This paper demonstrates that static analysis tools like CodeQL and Semgrep exhibit significant discrepancies when compared to human-validated ground truth for LLM-generated code, thereby motivating a proposed framework that integrates persistent human feedback into a dynamic retrieval-augmented generation pipeline to enhance secure code generation and vulnerability detection.

Original authors: Ehsan Firouzi, Mohammad Ghafari

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ehsan Firouzi, Mohammad Ghafari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very talented, fast, but inexperienced apprentice chef (the LLM) to cook a complex meal. You want the food to be safe to eat (secure code), so you ask the chef to cook it and then check it against a rulebook.

Here is the story of what the researchers found and what they propose to fix it, told through simple analogies.

1. The Problem: The "Rulebook" vs. The "Expert"

In the world of coding, we have two main ways to check if a recipe is safe:

The Rulebook (Static Analyzers): Tools like CodeQL and Semgrep are like automated spell-checkers or nutrition label scanners. They scan the code for known dangerous ingredients (vulnerabilities).
The Expert Chef (Human Feedback): A real human who tastes the dish, understands the context, and knows that just because an ingredient looks safe on the label, it might still be poisonous if mixed a certain way.

The Study:
The researchers asked an AI chef to cook 1,080 different dishes (code samples) designed to be safe. They then had a human expert taste-test every single one to create the "Truth."

The Truth: 61% of the dishes were actually safe.
The Rulebook's Verdict:
- Semgrep said 60% were safe. (This looks good on paper!)
- CodeQL said 80% were safe. (This looks even better!)

The Twist:
When the researchers looked at individual dishes, the Rulebook was often wrong.

Semgrep only agreed with the human expert on 65% of the cases.
CodeQL only agreed on 61%.

The Metaphor:
Imagine a security guard (the tool) checking people entering a building.

The guard might let a dangerous person in because they are wearing a nice suit (a False Negative).
Or, the guard might stop a harmless person because they are carrying a bag that looks suspicious (a False Positive).

The paper argues that relying only on the security guard (the static analyzer) is dangerous. Even if the guard gets the "average" number right, they are making mistakes on specific, critical cases. We need the human expert to step in.

2. The Gap in Current Research

The researchers looked at hundreds of other studies about AI and security. They found a pattern:

Language Bias: Most studies only test the AI on Python (for cooking recipes) or C/C++ (for building engines). They rarely test other languages.
Tool Reliance: Almost everyone uses the "Rulebook" (static tools) to grade the AI. Very few studies actually have a human taste-test the food.
The "Memory" Problem: When a human expert does catch a mistake, they fix it for that one dish. But once the expert leaves, the AI forgets the lesson. The next time the AI cooks a similar dish, it makes the same mistake again.

3. The Solution: The "Smart Library" Framework

The authors propose a new way to work, which they call a Human-in-the-Loop (HIL) Framework. Think of this as giving the AI chef a Smart Library that never forgets.

Here is how the new system works:

The Prompt Agent (The Order Taker): When you ask the AI to cook, this agent first checks the Smart Library. It asks, "Have we cooked this before? Did the human expert say anything about this specific ingredient?" It adds that advice to the chef's instructions.
The Security Agent (The First Taste): The AI cooks the dish. The Security Agent (the Rulebook) scans it first.
The Human Agent (The Head Chef): This is the crucial step. A human expert reviews the Security Agent's report.
- If the Security Agent says "Safe" but the Human says "Dangerous," the Human fixes it.
- The Magic Step: The Human's feedback isn't just a one-time fix. It is permanently stored in the Smart Library.
The Trust Score: Not all advice in the library is equal.
- If two experts agree on a piece of advice, it gets a high "Trust Score."
- If the advice has been used successfully many times in the past, its score goes up.
- If the advice is new, it waits in a "staging area" until two experts sign off on it. This prevents bad advice (or "poisoned" advice) from entering the library.

4. Why This Matters

The paper concludes that we cannot just trust the automated tools (the Rulebook) to tell us if AI-generated code is safe. They are too often wrong on the details.

Instead, we need a system where:

Humans are the final judges.
Human lessons are saved forever.
The AI learns from past human corrections so it doesn't make the same mistake twice.

In short: The AI is fast, the tools are good at scanning, but the human expert is the only one who truly understands the context. By building a system that remembers what the human expert says, we can make AI-generated code much safer.

Technical Summary: Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability Detection

Problem Statement
The paper addresses the growing reliance on Large Language Models (LLMs) for secure code generation and vulnerability detection, highlighting a critical gap in current evaluation methodologies. While LLMs are increasingly used in software development, they frequently generate insecure code even when explicitly instructed to do otherwise. Existing literature predominantly evaluates these models using automated static analysis tools (e.g., CodeQL, Semgrep) as the sole ground truth. The authors argue that this approach is flawed because static analyzers exhibit high false-positive rates and often fail to align with human expert judgment regarding actual security vulnerabilities. Furthermore, current frameworks lack mechanisms to persistently store and reuse expert feedback, limiting the ability of LLMs to learn from past corrections.

Methodology
The study employs a two-pronged methodology: a comprehensive literature review and an empirical validation study.

Literature Review: The authors surveyed 2025 publications from top-tier software engineering venues (A/A*) to characterize current research trends. They analyzed benchmarks, target programming languages, model families, and evaluation tools used in secure code generation and vulnerability detection studies.
Empirical Validation:
- Dataset Construction: Using the Recursive Critiques Improvement (RCI) prompting technique with GPT-4o, the authors generated 1,080 code samples based on 108 unique tasks mapped to the CWE Top 25.
- Ground Truth Creation: Two trained reviewers (a Ph.D. and an M.Sc. student), under faculty supervision, manually inspected the samples to label them as vulnerable or secure regarding their specific target CWE. Disagreements were resolved through discussion or expert arbitration, achieving a high inter-rater agreement (Cohen's $\kappa = 0.94$ ).
- Tool Comparison: The manually validated ground truth was compared against the outputs of two widely used static analysis tools, CodeQL and Semgrep.
- Framework Proposal: Based on the findings, the authors propose a conceptual framework integrating human feedback into a Retrieval-Augmented Generation (RAG) pipeline.

Key Results
The empirical study revealed significant discrepancies between automated tools and human experts:

Ground Truth: 61% of the generated samples were genuinely secure, while 39% contained vulnerabilities.
Tool Performance: Semgrep classified 60% of samples as secure, and CodeQL classified 80% as secure. While Semgrep's aggregate rate appeared close to human judgment, per-sample analysis showed substantial disagreement.
Accuracy Metrics: Only 65% of Semgrep's reports and 61% of CodeQL's reports correctly matched the ground-truth labels.
CWE Coverage: Both tools failed to detect target CWEs in a substantial fraction of ground-truth cases and frequently reported findings for non-target CWEs (whose validity remained unknown in this study).
Self-Critique: The study observed that LLMs providing "specific" self-critiques (targeted, actionable feedback) produced more secure code (64% secure) compared to "general" critiques (56% secure).

Proposed Framework: Persistent Human Feedback
To address the limitations of static tools and the transient nature of current feedback loops, the authors propose a conceptual framework (illustrated in Figure 3) that integrates a Human-in-the-Loop (HIL) agent into a dynamic RAG pipeline. The framework consists of four agents:

Prompt Agent: Orchestrates code generation by retrieving relevant security guidance from a Dual-Source RAG (Human Feedback and External Vulnerability Knowledge) to refine the developer's prompt.
Security Agent: Evaluates generated code using static analyzers and LLMs, producing machine-readable and human-readable reports.
Human-in-the-Loop (HIL) Agent: Acts as the final decision-maker. Experts review the Security Agent's reports. In cases of disagreement, expert feedback is persistently stored in the RAG.
- Trust Weighting: The system assigns dynamic trust weights to feedback based on reviewer consensus and historical empirical success (using an Exponential Moving Average).
- Dual Review: To prevent data poisoning, new feedback requires approval from two distinct reviewers before entering the active retrieval pool.
Dual-Source RAG: A shared knowledge service that prioritizes human-vetted feedback (HF) over general external knowledge (EVK) using a composite scoring function that penalizes contradictions between general advice and expert feedback.

Significance and Claims
The paper claims that relying solely on static analysis tools for evaluating LLM-generated code is unreliable and poses risks to the validity of security research. The authors assert that expert feedback is not only necessary for individual instances but can generalize to similar future cases. Consequently, the proposed framework aims to enable LLMs to reuse prior human feedback dynamically, improving both secure code generation and vulnerability detection. The study concludes that the software engineering community must move beyond automated-only evaluation and incorporate persistent, expert-driven feedback loops to ensure the reliability of LLMs in security-critical contexts.

Limitations (Threats to Validity)
The authors acknowledge several limitations:

Construct Validity: The literature review was restricted to 2025 publications in software engineering venues, potentially missing patterns in security-specific venues (e.g., USENIX Security, CCS).
Internal Validity: Ground-truth labels rely on manual assessment, which is subject to human bias despite high inter-rater agreement.
External Validity: The experiments focused on Python code generated by GPT-4o using RCI prompting. The results may not generalize to other languages (particularly low-level languages like C/C++), different domains, or other LLMs and static analysis tools.

1. The Problem: The "Rulebook" vs. The "Expert"

2. The Gap in Current Research

3. The Solution: The "Smart Library" Framework

4. Why This Matters

Technical Summary: Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability Detection

More like this