WildCode Revisited: A Comprehensive Empirical Study on the Security of LLM-Generated Code
This study empirically validates that real-life code generated by ChatGPT frequently lacks security, confirming prior findings from synthetic experiments while revealing that users rarely inquire about security concerns when requesting code.
Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a very talented, incredibly fast, but somewhat naive assistant who can write computer code for you. You ask for a recipe, and they whip up a meal in seconds. But here's the catch: they learned to cook by reading millions of cookbooks, but they've never actually been in a kitchen, and they don't really understand why certain ingredients might be dangerous if mixed wrong.
This paper, "WildCode Revisited," is like a massive, real-world health inspection of the food this assistant serves up. Instead of asking the assistant to cook a specific test meal in a lab (which is what most previous studies did), the researchers went into the "wild." They looked at over 82,000 real conversations where regular people asked this AI (ChatGPT) to write code for them.
Here is what they found, broken down into simple analogies:
1. The "Wild" Kitchen vs. The Test Kitchen
Most previous studies were like a cooking show where the chef is asked, "Make a perfect lasagna." The chef knows it's a test, so they try their best.
This study looked at the WildChat dataset, which is like a 24-hour diner where people order whatever they want, in whatever language they speak. The researchers found that when people just ask for code without specifying safety, the AI often serves up dishes that are full of holes. It's like the assistant forgetting to lock the back door, leaving the house wide open to burglars (hackers).
2. The Security "Leak"
The researchers acted like security inspectors, scanning the code for common mistakes. They found that the AI frequently serves up:
- Weak Locks: Using old, easily picked locks (weak encryption) instead of modern, unbreakable ones.
- Open Windows: Leaving windows open for strangers to walk right in (SQL injection vulnerabilities).
- Fake Ingredients: The AI sometimes invents "ingredients" (software libraries) that don't actually exist. If you try to use them, your program crashes. Worse, a bad actor could create a fake ingredient with the same name and sneak in a virus.
- Fragile Glass: The code is often made of glass that shatters if you look at it the wrong way (memory safety issues), especially in languages like C/C++.
The Big Surprise: Even though the code looks like it works (it runs), it's often full of security traps. The AI is great at making things work, but terrible at making them safe.
3. The Customers Don't Ask for Safety
The researchers also looked at what the customers (the users) were asking for.
- The "Just Fix It" Crowd: Most people just wanted the code to work or wanted to fix a bug. They were like customers saying, "Make me a sandwich," without caring if the bread had mold on it.
- The Silence on Safety: Very few people asked, "Is this code safe?" or "Does this have a backdoor?" It's as if customers are ordering food but never asking, "Is this fresh?" or "Is there a health inspector here?"
- The "Security" Glitch: Even when users did ask for secure code, the AI didn't always listen, and the users didn't always follow up to check if the AI actually made it secure.
4. Can We Train the Assistant Better? (The "Prompt" Experiment)
The researchers tried a few tricks to see if they could get the AI to cook safer meals. They tried:
- Role-Playing: Telling the AI, "Act like a security expert." (Result: Not very effective. The AI just pretended to be an expert but still made mistakes.)
- Warning Labels: Telling the AI, "This code has security issues." (Result: A little better, but not a miracle.)
- The "Cheat Sheet": Giving the AI a list of specific errors found by a security scanner and saying, "Fix these." (Result: This worked best. When the AI was shown exactly what was wrong, it fixed the code much better.)
5. The "Refusal" Game
Sometimes, the AI refuses to write code if it thinks the request is dangerous (like writing a virus). The researchers found that users could sometimes trick the AI into writing dangerous code by asking the same question in a slightly different way. It's like a child asking, "Can I have a cookie?" and getting a "No," but then asking, "Can I have a cookie if I'm hungry?" and getting a "Yes." The AI's safety filters aren't as strong as we'd hope.
The Bottom Line
This study is a reality check. It shows that while AI is amazing at generating code quickly, it is currently not a safe substitute for a human security expert.
- The Code: Often has hidden traps and weak spots.
- The Users: Rarely ask about safety, assuming the AI knows what it's doing.
- The Fix: We can't just tell the AI to "be safe." We need to give it specific feedback on what's wrong (like a security scanner) to help it fix the holes.
The paper concludes that we are in a rush to use these tools, but we are doing so without checking the locks on the doors. We need to be much more careful and proactive about security before we let AI write the code that runs our world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.