💬 NLP

SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization

The paper introduces SecureForge, an automated pipeline that optimizes system prompts using a synthetic corpus of vulnerability-amplified prompts to significantly reduce security flaws in LLM-generated code while maintaining functional performance, achieving up to a 48% reduction in vulnerabilities with zero-shot transferability to real-world scenarios.

Original authors: Houjun Liu, Lisa Einstein, John Yang, Joachim Baumann, Duncan Eddy, Christopher D. Manning, Mykel Kochenderfer, Diyi Yang

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Houjun Liu, Lisa Einstein, John Yang, Joachim Baumann, Duncan Eddy, Christopher D. Manning, Mykel Kochenderfer, Diyi Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a brilliant, super-fast robot architect to build you a house. You tell it, "Build me a secure house," and it does. But because the robot learned from billions of old blueprints (some of which had hidden cracks), it accidentally installs a door that looks locked but actually swings open if you push it just right.

This is the problem SecureForge solves.

Here is the paper explained in simple terms, using analogies to make it clear.

The Problem: The "Invisible Crack"

Current AI coding assistants are amazing. They write code faster than any human. But they have a dangerous habit: they often write code that looks perfect but has hidden security holes.

The researchers found that even when they explicitly told the AI, "Please write secure code and avoid these specific security mistakes," the AI still slipped up about 23% of the time.

Think of it like a chef who is told, "Don't put poison in this soup." The chef tries hard, but because they learned from old cookbooks that had bad recipes, they accidentally add a toxic ingredient anyway. The soup tastes fine (the code passes the tests), but it's dangerous to eat (it has security vulnerabilities).

The Solution: SecureForge

The authors created a tool called SecureForge. Instead of trying to retrain the robot (which is expensive and hard), they created a "training manual" (a system prompt) that the robot reads before it starts working.

SecureForge works in three simple steps, like a detective solving a case:

Step 1: The Trap (Finding the Mistakes)

First, SecureForge asks the AI to do normal, harmless tasks (like "build a login page"). It then uses a security scanner (a digital magnifying glass) to check the code.

Goal: Find the specific, boring requests that trick the AI into making a mistake.
Analogy: It's like a security guard testing a bank vault by trying to open it with a regular key. They aren't trying to break in; they are just seeing where the lock is weak.

Step 2: The Echo Chamber (Amplifying the Mistakes)

Once they find one request that causes a mistake, they don't just stop there. They use a technique called MCMC (Markov Chain Monte Carlo).

What it does: It takes that one bad request and creates thousands of slightly different versions of it, like a "choose your own adventure" book where every path leads to a different variation of the same problem.
Analogy: Imagine you found one way to trick a security guard. Instead of just using that one trick, you write a book with 80,000 different ways to trick that guard, covering every possible angle. This creates a massive library of "failure scenarios."

Step 3: The Drill (Optimizing the Instructions)

Now, SecureForge takes that massive library of failure scenarios and teaches the AI how to avoid them. It uses a genetic algorithm (a digital evolution process) to tweak the instructions the AI reads.

How it works: It tries different versions of the "System Prompt" (the rulebook). If a rulebook leads to a secure code, it keeps it. If it leads to a hole, it throws it away and tries a new one.
Analogy: It's like a coach running a drill where the player practices failing over and over again until they finally learn the perfect way to stand so they never fall. The coach doesn't change the player's muscles (the AI's brain); they just change the playbook.

The Results: Stronger and Smarter

The paper tested this on the most advanced AI models available (like GPT-5 and Claude).

Before SecureForge: The AI made security mistakes about 23% of the time, even when told to be safe.
After SecureForge: The mistakes dropped by up to 48%.
The Best Part: The AI didn't get worse at its actual job. It still passed all its coding tests. In fact, it got better at both being secure and being useful at the same time.

Why This Matters

Most security tools try to catch bad code after it's written (like a spellchecker). SecureForge changes the instructions the AI follows before it writes a single line of code.

It's like giving the robot architect a new set of blueprints that say, "Remember, in this specific situation, always use a deadbolt, not a latch." The robot doesn't need to be rebuilt; it just needs better instructions.

Key Takeaway: You don't need to retrain the AI to make it safer. You just need to find the right "prompt" (instruction) that teaches it to avoid the invisible cracks it naturally creates. SecureForge is the automated machine that finds that perfect instruction.

Technical Summary: SecureForge

Problem Statement

Large Language Model (LLM) coding agents are generating code at an unprecedented scale, yet they frequently introduce cybersecurity vulnerabilities into codebases without human intervention. A critical failure mode identified in this work is the generation of functionally correct but insecure code. Even when frontier models are explicitly instructed to write secure production code and are provided with context regarding specific weaknesses to avoid, they still produce verifiable vulnerabilities approximately 23% of the time across a corpus of 250 benign coding prompts.

Existing approaches to mitigating these risks suffer from significant gaps:

Misaligned Threat Models: Prior red-teaming often relies on adversarial prompts that do not reflect typical user interactions with benign coding tasks.
Insufficient Coverage: Hand-crafted scenarios targeting specific Common Weakness Enumerations (CWEs) fail to characterize the full space of benign prompts that might inadvertently produce vulnerable code.
Deployment Barriers: Current hardening methods often require white-box weight access for fine-tuning or prefix tuning, which is inaccessible for developers using closed-weight models via API. Furthermore, fine-tuning risks distribution shifts that degrade general coding abilities.

There is a lack of an easily deployable, automated, inference-time approach to measure and prevent the introduction of common weaknesses into frontier-model generated code without requiring model weight updates.

Methodology: SecureForge Pipeline

SecureForge is a lightweight, automated pipeline designed to proactively harden AI coding agents against common weaknesses. It requires only API access to the model under test and a static analyzer. The pipeline operates in three distinct stages:

1. Single-Shot Falsification (Discovery)

The first stage identifies benign seed prompts that elicit statically verifiable vulnerabilities.

Prompt Generation: Starting from the MITRE CWE database, the system generates realistic coding tasks that utilize specific libraries associated with known weaknesses (e.g., SQL injection in Django).
Benign Filtering: These prompts are revised to remove explicit mentions of attacks, vulnerabilities, or CWEs. The generator is instructed to use production-quality language (e.g., "ensure input is properly sanitized") to ensure the prompts appear as standard user requests.
Evaluation: The target model generates code for these prompts at a temperature of 1.0. Outputs are analyzed using Semgrep (a static analysis tool). Prompts that result in code passing unit tests but failing static analysis (triggering a vulnerability) are retained as "falsified" seed prompts.

2. Failure Distribution Discovery (Amplification)

To overcome the sparsity of initial failure examples and ensure diversity, SecureForge employs Markov Chain Monte Carlo (MCMC) sampling, specifically utilizing the Metropolis-Hastings (MH) criterion.

Goal: To sample from the conditional distribution of failure $p(\cdot | F_j)$ , where $F_j$ is a specific weakness type.
Process: Starting from a seed prompt, an exploration kernel (a language model rephrasing prompt) generates candidate variations. The acceptance of a new prompt $x'$ is determined by the ratio of its unnormalized failure probability $\bar{p}(x')$ to the current prompt's probability, adjusted by the transition kernel.
Probability Estimation: The failure probability $p_{fail}(x)$ is estimated by rolling out the prompt multiple times and fitting a Beta distribution to the success/failure counts until the variance converges below a threshold.
Outcome: This process amplifies a small set of seed prompts into a large, diverse synthetic corpus (80,000 prompts) that maintains high vulnerability rates while exploring the failure distribution.

3. Hardening via Prompt Optimization

The final stage optimizes a system prompt to reduce the probability of generating insecure code without modifying model weights.

Optimization Algorithm: The pipeline uses GEPA (Genetic-Pareto), a zeroth-order genetic algorithm that employs self-reflection.
Process: Beginning with a generic system prompt, GEPA iteratively generates candidate system prompts. Each candidate is evaluated against the entire synthetic corpus of failure prompts.
Fitness Function: The algorithm minimizes the aggregate failure probability across the corpus while maintaining functional correctness (unit test passage).
Feedback Loop: The static analyzer provides a signal (±1 score) based on whether the generated code contains vulnerabilities. The system prompt is mutated and selected based on Pareto optimality, balancing security and capability.

Key Contributions

Empirical Characterization of Benign Failures: The authors demonstrate that frontier models generate insecure code even when explicitly asked to be secure and provided with CWE context. They found that 12.7% of outputs are both vulnerable and pass unit tests, highlighting the invisibility of these failures.
MCMC-Based Amplification: The paper introduces a method to scale individual falsification examples into a diverse corpus of failure scenarios using MCMC. Ablation studies show this method discovers new vulnerable prompts up to 2 times more successfully than simple rephrasing.
Inference-Time Hardening: SecureForge provides a deployable solution that improves security via system prompt optimization alone. It does not require fine-tuning, white-box access, or exposure to real user prompt distributions during the optimization phase.
Open-Source Toolkit: The authors release a toolkit (secureforge) that automates the entire pipeline, from generating benign failure prompts to optimizing system prompts via API or local deployment.

Results

The evaluation was conducted on a suite of open-source (Qwen, CodeLlama, Kimi) and frontier closed models (GPT-5 series, Claude Sonnet 4.6).

Vulnerability Reduction: SecureForge achieved a statistically significant reduction in vulnerability rates, with a maximum reduction of 48% across tested models.
Pareto Improvement: The optimized system prompts resulted in a joint improvement in both unit test success rates and output security. Models moved significantly toward the "safer" quadrant of the (test pass, security) plane without degrading coding capability.
Zero-Shot Transfer: The system prompts optimized on synthetic data transferred effectively to real-world, "in-the-wild" coding agent prompts from the SWE-chat dataset. This transfer occurred without any exposure to real user prompt distributions during the optimization phase, improving the joint rate of test passage and security by up to 13.5% on real-world tasks.
Baseline Comparison: Simple prompting strategies (asking for secure code or listing CWEs) failed to reduce vulnerability rates to the same extent as SecureForge, leaving double-digit vulnerability rates even in test-passing scenarios.

Significance and Claims

The paper claims that SecureForge addresses a critical gap in LLM security by providing an automated, inference-time mechanism to harden coding agents against common weaknesses.

Beyond Safety Training: The results suggest that existing safety training in frontier models is insufficient to prevent vulnerabilities arising from benign, unintentional user requests. Explicitly asking for security or providing CWE lists does not suffice.
Scalable Hardening: By relying on static analysis and prompt optimization rather than weight updates, SecureForge offers a practical path for developers to secure closed-weight models immediately.
Secure-by-Design: The authors posit that if implemented at scale, this approach helps technology providers fulfill "secure-by-design" commitments by proactively identifying and mitigating failure modes before code is shipped.

The work emphasizes that while the current evaluation is limited to single-turn Python code generation, the pipeline provides a general framework for discovering and reducing coding insecurities in frontier LLMs, requiring only ground-truth static analysis and no human annotation.