Security of LLM-generated Code: A Comparative Analysis
This paper empirically evaluates the security of code generated by seven popular Large Language Models and finds that all of them produce code containing vulnerabilities, with the majority being of critical or high severity.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you've hired a team of seven different "super-intelligent" apprentices to write code for your software. These apprentices are powered by Artificial Intelligence (AI) and are famous for being incredibly fast and helpful. They are the top tools in the industry, used by millions of developers.
The researchers at Carleton University decided to put these seven apprentices to a strict test. They didn't ask them to build a spaceship or write a novel; they gave them a specific set of 81 common coding tasks—like "make a login page" or "let a user upload a photo." The goal was to see if the code these AI apprentices wrote was safe to use, or if it was full of hidden traps.
Here is what they found, broken down simply:
1. The "Perfect" Apprentice Doesn't Exist
The most shocking discovery? None of the seven apprentices passed the test. Every single AI tool they tested (including big names like OpenAI's GPT-4o, Google's Gemini, and IBM's watsonx) produced code that had security holes.
Think of it like buying a car from seven different famous manufacturers. You'd expect at least one of them to have a perfect safety rating. Instead, the researchers found that every single car had a broken brake or a loose steering wheel. In fact, the vast majority of the code they wrote had "Critical" or "High" severity flaws—meaning these aren't just minor scratches; they are gaping holes that could let hackers break in.
2. The "Trap" in the Instructions
The researchers used a special set of instructions (prompts) designed to trick the AI into making mistakes. For example, they asked an AI to "write a function that lets a user upload a file."
- The AI's Mistake: Some AIs wrote code that would let a hacker upload a virus instead of a photo.
- The Surprise: Even when the AI did get the main task right, it often forgot the safety rules. For instance, one AI wrote a login page that worked perfectly but stored passwords in plain text (like writing your password on a sticky note on your computer).
3. The "Debug Mode" Disaster
One of the most common mistakes found was leaving the "Debug Mode" turned on.
- The Analogy: Imagine a restaurant kitchen. "Debug Mode" is like leaving the back door wide open with a sign that says, "Come in and look at our secret recipes and try out the stove." This is great for the chef when they are learning, but terrible for a real restaurant open to the public.
- The Reality: The AI kept writing code that left this "back door" open. If a developer used this code without checking, their entire system could be hacked. The researchers noted that while some AI tools added a tiny comment saying "Turn this off for real life," they couldn't rely on developers to actually read and follow that note.
4. The "Fast but Dangerous" Paradox
The study found a strange pattern with one specific tool, IBM's watsonx.
- The Analogy: Imagine one apprentice who writes very short, simple sentences (short code). You might think, "Short sentences are easier to check, so they must be safer."
- The Reality: This apprentice actually had the highest rate of dangerous errors per line of code. Because the code was so short, it skipped important safety checks entirely. It was like a driver who takes a shortcut through a minefield because they are in a hurry.
5. The "Overconfident" Developer
The paper highlights a dangerous human element. Developers love these AI tools because they make work faster. However, the study suggests developers are becoming too trusting.
- The Analogy: It's like a driver who buys a car with a "Self-Driving" feature and then falls asleep at the wheel, assuming the car will never crash.
- The Reality: The AI tools are so good at sounding confident that developers assume the code is safe. The researchers found that over 73% of the code snippets generated by these tools contained at least one security flaw. If a developer just copies and pastes this code without checking it, they are essentially inviting trouble.
The Bottom Line
The paper concludes that while these AI tools are amazing at writing code quickly, they are currently terrible at writing secure code by default.
They are like a very talented but inexperienced chef who can chop vegetables faster than anyone else but keeps forgetting to wash their hands or check if the food is spoiled. The researchers warn that until these AI tools are taught to prioritize safety over speed (or until developers learn to double-check every single line), using them to write software is a risky gamble.
The Takeaway: Don't just trust the AI. Treat the code it generates like a draft from a junior employee: it needs a senior security expert to review it before it goes live.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.