Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation
This paper benchmarks seven large and small language models on generating secure AWS Terraform code, revealing that syntactic validity and security compliance are largely orthogonal properties and that automated multi-tool scanning remains essential regardless of model performance or prompt engineering strategies.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are building a massive, invisible city in the sky, made entirely of digital bricks. In the real world, if you build a house with a wobbly foundation or leave the front door wide open, it's a disaster. In the digital world, this "city" is called the Cloud, and the blueprints used to build it are written in a special language called Infrastructure-as-Code (IaC). Think of IaC as a super-precise recipe that tells computers exactly how to set up servers, storage, and security locks. For a long time, humans wrote these recipes, but they often made mistakes, leaving digital doors unlocked and letting bad guys sneak in.
Recently, a new kind of helper arrived: Artificial Intelligence. Specifically, Large Language Models (LLMs) and their smaller, faster cousins, Small Language Models (SLMs). These are like super-smart robots that can read millions of recipes and try to write new ones for you in seconds. The big question on everyone's mind was: Can these AI robots write recipes that are not just correct (so the building doesn't collapse) but also secure (so no one can break in)? It's a bit like asking if a robot chef can bake a cake that tastes good and doesn't accidentally contain a hidden bomb. If the AI gets the syntax right but forgets the locks, the whole city is in danger. This is the exact puzzle a team of researchers from Brazil set out to solve.
The Great AI Chef Competition
The researchers decided to put seven different AI chefs to the test in a giant kitchen called the "Cloud." They didn't just ask the robots to write a recipe; they asked them to write 17 different types of recipes for building secure digital structures using a tool called Terraform. They tested three "Big Brain" models (the closed, expensive ones like Claude Opus 4, GPT-5.4, and Gemini 2.5 Pro) and four "Pocket Brain" models (the open-source, smaller ones like WizardCoder and CodeLlama).
To see if the recipes were any good, the team didn't just look at them with their eyes. They built a super-strict automated inspection line. First, they checked if the recipe was grammatically correct (Syntactic Validity). If the recipe had typos, the building wouldn't even start. But the real test came next: they ran two different security scanners, Checkov and Trivy, which act like digital security guards looking for open windows, unlocked doors, and hidden traps. They also tested if the robots could fix their own mistakes when told, "Hey, you left a door open," and if giving them more detailed instructions about security actually helped.
The Shocking Results: Good Grammar, Bad Locks
Here is the twist that the researchers found, and it's a bit of a plot twist: Writing a perfect recipe and writing a safe recipe are two completely different skills.
They discovered that just because an AI can write a Terraform recipe that is grammatically perfect and runs without crashing, it does not mean the recipe is secure. In fact, the two skills are almost unrelated. One of the smaller models, WizardCoder-33B, was a grammar champion. It wrote valid recipes 77.8% of the time! But when the security guards checked those same recipes, the model failed every single security check. It was like a chef who made a beautiful, perfectly shaped cake but forgot to put the lid on the jar of poison inside.
On the other hand, the "Big Brain" models did much better, but they still needed a lot of help. The best performer, Claude Opus 4, only managed to write a fully secure recipe about 23% of the time when given basic instructions. However, when the researchers gave it a very specific, detailed security checklist (telling it exactly which locks to install and which alarms to set), its performance jumped to 92.5% for one type of security scanner. This proved that while the big brains are capable, they won't do the right thing unless you tell them exactly what to do.
The "Pocket Brains" Hit a Wall
The smaller models (SLMs) hit a hard wall. No matter how much the researchers tried to give them detailed security instructions, the small models just couldn't follow through. They would either write a recipe that was broken, or they would write a valid one that was completely insecure. The researchers found that for these smaller models, the problem wasn't the instructions; it was that the models simply didn't have the "brain power" to understand complex security rules. They are great at copying patterns, but they struggle to invent secure solutions on their own.
Can They Fix Their Own Mistakes?
The team also tested if the robots could learn from their errors. They let the robots write a recipe, scanned it, and then showed the robot the list of mistakes (like "You forgot to encrypt this file") and asked it to try again.
- The Good News: The robots were surprisingly good at fixing simple, specific mistakes found by the Trivy scanner (like a missing lock on a specific door). Many models improved their scores significantly after seeing the error list.
- The Bad News: They were terrible at fixing the big, structural problems found by the Checkov scanner. These were issues like "You need to build a whole new security system for this building." The robots couldn't re-architect their own work in one go. They could tighten a loose screw, but they couldn't redesign the foundation.
The Bottom Line
The most important takeaway from this study is a warning for anyone thinking about letting AI build their digital cities: You cannot trust the AI to be the security guard.
The researchers found that between 2024 and 2026, AI models got much better at writing code that looks right, but they did not get any better at writing code that is safe. In fact, the gap between "looks good" and "is safe" has actually gotten wider.
The study concludes that you cannot just ask an AI to "make it secure" and hope for the best. Even the smartest AI models need a human (or a very strict automated system) to double-check their work. You must run multiple security scanners on every piece of code the AI generates, no matter how good the model is or how detailed your instructions were. The AI is a powerful assistant, but in the world of cloud security, it is not yet a replacement for a safety inspector. If you skip the inspection, you might end up with a beautiful, perfectly built digital city that has no locks on the doors.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.