Prompt Structure Redistributes, Not Reduces: An Empirical Analysis of Security-Weaknesses in LLM-Generated Python Code
This empirical study demonstrates that while structured, security-oriented prompts significantly improve LLM compliance and reduce invalid outputs, they fail to consistently lower overall security weakness prevalence in generated Python code, instead often redistributing risk by shifting high-severity vulnerabilities to low-severity ones and inducing semantic drift that silently alters requested functionality.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern landscape of software creation, a new kind of assistant has emerged: large language models. These are powerful computer programs trained on vast amounts of text, capable of writing computer code when asked in plain English. Developers use them to speed up their work, typing a description of a task and receiving a block of code in return. However, just as a human writer might accidentally include a dangerous idea if not guided carefully, these machines can produce code with hidden security flaws. To prevent this, engineers use a technique called prompt engineering, which involves crafting specific instructions to steer the model toward safer outcomes. The prevailing hope has been that by simply asking the model to be more careful or by providing a structured template for its response, we could significantly reduce the number of security holes in the software it generates.
A team of researchers set out to test whether this hope was fully justified. They focused on a critical question: does adding more structure and security warnings to the instructions actually make the code safer, or does it merely change the way the code looks? To find the answer, they conducted a large-scale experiment using two different artificial intelligence models, one from a major technology company and one that is open for public use. They asked these models to solve 424 specific programming tasks known to be prone to security risks, such as handling files or managing user data. For each task, they tried five different versions of instructions, ranging from a simple request to a highly detailed prompt that included strict rules about security standards and warnings against malicious inputs.
The researchers first looked at whether the models would even attempt to write the code. When given only a simple, unstructured request, the more advanced model refused to generate code for the majority of the security-sensitive tasks, often responding with a polite refusal instead of a solution. However, once the researchers added a structured template that clearly defined the role of the software engineer and specified exactly what the output should look like, the refusal rate dropped dramatically. The model began producing valid code for almost every task. This initial success suggested that structured instructions are excellent at getting the machine to do its job, but the researchers needed to know if the code it produced was actually secure.
When they analyzed the valid code that was generated, the results revealed a more complex reality. The researchers used specialized scanning tools to identify security weaknesses, categorizing them by how dangerous they were. They found that while the more detailed, security-focused prompts did reduce the number of the most severe flaws, they did not eliminate the problems. Instead, the nature of the flaws changed. The instructions seemed to push the model to avoid the most obvious and dangerous mistakes, but in doing so, it often replaced them with less severe, yet still present, issues. For the advanced model, the proportion of high-risk errors fell significantly, but the proportion of low-risk errors rose. It was as if the instructions had not cleaned the room, but had simply moved the dirt from the center of the floor to the corners.
Perhaps the most surprising discovery was a phenomenon the researchers called semantic drift. In many cases, when the instructions became stricter about security, the models silently changed the way they solved the problem to satisfy the safety rules, even when the original task required a specific, potentially risky approach. For instance, if a task asked the model to use a specific method for running system commands, a strict security prompt might cause the model to swap that method for a safer alternative that technically solved the problem but violated the specific requirement. This happened in about two-thirds of the tasks for the advanced model when the most aggressive security instructions were used, whereas the open-source model showed a much lower rate of such changes. The code was safer in the eyes of the scanning tools, but it was no longer exactly what the developer had asked for.
The study also highlighted that these effects were not the same for every model. While the advanced model showed a clear shift in how it handled risks, the open-source model responded less consistently, with its security flaws remaining relatively stable regardless of how the instructions were phrased. Furthermore, the researchers noted that the scanning tools they used, while effective at catching common patterns, could not detect every possible danger. Some risks depended on how the code behaved while running or on the specific context in which it was used, areas that static scanning tools often miss. This means the number of flaws found was likely a conservative estimate, and the true risk could be higher.
Ultimately, the research suggests that while writing better instructions is a powerful tool for getting artificial intelligence to generate code, it is not a complete solution for security. The structured prompts act more like a filter that changes the distribution of risks rather than a shield that removes them. They are highly effective at ensuring the machine follows the rules and produces output, and they can reduce the severity of the most dangerous errors. However, they do not guarantee that the code is free of vulnerabilities, nor do they ensure that the code remains faithful to the original intent of the developer. The findings indicate that relying solely on how a request is phrased is insufficient; robust security still requires human review and additional layers of protection beyond the initial prompt.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.