Evaluating and Preventing Security Smells in AI-Generated Ansible Code
This paper reveals that AI-generated Ansible code inherently contains security vulnerabilities but demonstrates that integrating security benchmarks into prompts via an extended CO-STAR framework can significantly improve compliance and code quality, with top models achieving near-perfect security standards without retraining.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital world, the backbone of almost every online service is a vast, invisible layer of computers and software known as infrastructure. For decades, setting up this infrastructure required teams of engineers to manually configure servers, databases, and security settings, a slow and error-prone process. To fix this, the industry adopted a method called "Infrastructure as Code," where the entire setup is written out in text files, much like a detailed recipe or a blueprint. These text files tell computers exactly how to build and secure a system, allowing for rapid, consistent, and automated deployment. However, just as a recipe with a missing ingredient can ruin a meal, a single mistake in these code files can leave a system wide open to hackers. If the code contains hidden weaknesses, such as unencrypted passwords or overly permissive access rules, those flaws are instantly transferred to the live system the moment it is turned on.
Recently, a new tool has entered this field: artificial intelligence coding assistants. These programs can read a simple request in plain English and automatically write the complex code needed to build these systems. While this promises to speed up development, it raises a critical question that had never been answered until now: does the code these machines write actually keep systems safe? The researchers at the University of Waikato in New Zealand set out to find the answer. They were not just looking for bugs; they were investigating whether the code generated by these AI tools met strict, real-world security standards required by governments and industries to protect data. Their work reveals a startling gap between what these AI tools can do and what they actually do when left to their own devices, and it offers a clear path to fixing the problem before a single line of code is deployed.
The team began by testing the default behavior of sixteen different artificial intelligence models. They asked each model to write a specific set of instructions, known as an Ansible role, to set up two common types of software: a web server called Apache Tomcat and a database system called MongoDB. They gave the models no special security advice, no warnings about what to avoid, and no examples of good code. They simply asked the machines to do the job. The results were immediate and concerning. Every single one of the sixteen models produced code that contained security flaws. These flaws included hard-coded passwords that anyone could read, missing protections for sensitive files, and a lack of error handling that could cause the system to crash or behave unpredictably. When the researchers compared this AI-generated code to code written by human developers from public repositories, the AI code performed worse. It was not just slightly flawed; it was fundamentally insecure, failing to meet basic safety requirements that are standard in the industry.
The researchers then investigated why this was happening. They discovered that the problem was not necessarily that the AI models lacked the ability to write code, but that they lacked the ability to follow complex instructions regarding safety. In a second phase of the study, the team changed their approach. Instead of just asking for code, they provided the models with a highly structured set of rules. They used an expanded version of a prompting framework that explicitly listed security best practices and specific government safety standards. They told the models exactly which permissions to set, how to manage passwords securely, and what documentation to include, treating these requirements as mandatory constraints rather than suggestions. This shift from a simple request to a detailed, rule-based instruction changed the outcome dramatically.
When the researchers applied this structured approach, the results improved significantly. Four of the sixteen models were able to follow the complex instructions and generate code that was free of the security flaws seen in the first round. The best-performing model produced code that met between ninety-five and one hundred percent of the strict security benchmarks, a massive leap from the baseline. In fact, this top AI model outperformed the average human-written code, which only met between twenty-three and forty-three percent of the same standards. The study showed that for these capable models, the issue was not a lack of knowledge, but a failure to apply that knowledge when the instructions were vague. When the rules were clear and the constraints were explicit, the AI could synthesize secure, high-quality code in a single attempt, eliminating the need for time-consuming fixes after the code was already written.
The study also highlighted a crucial distinction between the models. The four successful models were all closed-source systems, meaning their internal workings are not public, while many of the open-source models failed to follow the complex instructions. This suggests that the ability to generate secure code depends heavily on how the model was trained and the specific capabilities it developed during that training, rather than just the size of its memory or the number of parameters it has. The researchers found that the models that succeeded were able to parse the multi-layered instructions, understand the difference between mandatory rules and recommended practices, and apply them consistently across the code. The models that failed, even those with high scores on other coding tests, simply could not hold onto the multiple constraints required to build a secure system.
This work challenges the current way the industry handles security. Traditionally, security experts wait until code is written and then scan it to find and fix mistakes, a process known as detection. The researchers argue that for AI-generated code, this approach is insufficient because the flaws are introduced the moment the code is created. Instead, they propose a method of prevention, where security requirements are built into the generation process itself. By embedding safety rules directly into the instructions given to the AI, organizations can ensure that the code is secure from the start. This approach does not require retraining the AI models or changing their underlying architecture; it only requires a change in how humans talk to them. The study concludes that while AI coding assistants hold great promise, they cannot be trusted to generate secure infrastructure without clear, explicit guidance. With the right prompts, however, they can produce code that is not only functional but also safer than what many human developers produce today.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.