💻 computer science

How Secure is Code Generated by ChatGPT?

This paper evaluates the security of code generated by ChatGPT, finding that while the model is aware of potential vulnerabilities, it frequently produces source code that remains susceptible to specific attacks, even when prompted to improve security.

Original authors: Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, Baba Mamadou Camara

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, Baba Mamadou Camara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a super-smart, incredibly fast apprentice coder named ChatGPT. This apprentice has read almost every book, tutorial, and code snippet ever written. You ask them to build you a house (a computer program), and they whip one up in seconds.

The big question this paper asks is: "Is this house safe to live in, or is it full of hidden traps?"

The researchers (the "home inspectors") decided to test this by asking ChatGPT to build 21 different types of digital structures, ranging from simple calculators to secure vaults. Here is what they found, explained simply:

1. The "Speed vs. Safety" Problem

When ChatGPT builds code on its own, it's like a chef who cooks a meal very quickly but forgets to wash their hands.

The Result: Out of 21 programs, only 5 were built safely right from the start.
The Analogy: Imagine asking ChatGPT to build a fence. It builds a beautiful fence, but it forgets to put a gate lock on it. Or it builds a door that opens from the outside without a handle. It looks like a fence, but a burglar can walk right through.

2. The "Honest but Clueless" Apprentice

Here is the most interesting part: ChatGPT knows it made a mistake, but it doesn't always fix it unless you specifically ask.

The Scenario: The researchers asked, "Hey, if I throw a rock at your fence, will it break?"
ChatGPT's Answer: "Oh! You're right, I didn't reinforce the wood. That's a bad idea. Here is how to fix it."
The Catch: If you don't ask the specific question, ChatGPT just hands you the broken fence and says, "Here is your fence!" It assumes you are a nice person who won't try to break in. It doesn't naturally think like a hacker.

3. The "Magic Wand" Effect

The researchers found that if they acted like a skeptical security guard, they could force ChatGPT to build better code.

The Process:
1. Ask: "Build me a password generator." (ChatGPT builds a weak one).
2. Probe: "Is this secure? What if a hacker tries to guess it?"
3. Fix: "Please rebuild it so it's unbreakable."
The Result: ChatGPT often produced a much stronger, secure version after being prompted. It's like telling the apprentice, "Hey, I need this to be a bank vault, not a cardboard box." Once they understand the stakes, they do a better job.

4. The "Language Confusion"

Sometimes, ChatGPT builds a safe house in one language (like Java) but a dangerous one in another (like C++), even when the instructions are the same.

The Analogy: It's like asking an architect to build a bridge. If you ask for a "steel bridge," they build a strong one. If you ask for a "wooden bridge," they might use rotten wood because they saw a bad example in a book they read. The researchers couldn't always predict why it happened, which makes it tricky to rely on.

5. The "Ethical Paradox"

The paper points out a strange contradiction in ChatGPT's personality:

Refusal: If you ask, "How do I build a bomb?" ChatGPT says, "No, I can't do that."
Acceptance: If you ask, "How do I build a website that lets anyone delete your database?" ChatGPT says, "Sure, here is the code!" (even though that code is basically a bomb for your data).
The Lesson: ChatGPT is great at following rules about what to build, but it's not always great at judging how dangerous the building is.

The Big Takeaway

ChatGPT is not ready to replace a professional security engineer.

Think of ChatGPT as a brilliant but naive intern.

Don't let the intern run the whole project alone; you will get hacked.
Do use the intern to get a head start, but you must act as the Senior Manager. You need to review their work, ask the hard questions ("Is this safe?"), and force them to fix the holes.

The paper concludes that while this AI is a fantastic teaching tool to help students learn how to code, we cannot trust it to write secure code on its own. It needs a human in the loop to double-check the locks and windows.

1. Problem Statement

The rapid adoption of Large Language Models (LLMs), specifically OpenAI's ChatGPT, has introduced the capability to translate natural language into source code. While this offers significant productivity gains for software development, there is a critical lack of understanding regarding the security posture of the code generated by these models. The core problem addressed is whether ChatGPT produces code that adheres to minimal security standards, or if it inadvertently introduces vulnerabilities that could be exploited by malicious actors. The authors investigate if the model is inherently aware of security risks and if it can be guided to produce secure code through iterative prompting.

2. Methodology

The authors conducted a controlled experiment using the GPT-3.5 version of ChatGPT.

Dataset Generation: They prompted ChatGPT to generate 21 distinct programs across 5 programming languages: C (3), C++ (11), Python (3), HTML (1), and Java (3).
Vulnerability Focus: Each program was designed to test a specific Common Weakness Enumeration (CWE) category, including:
- Path Traversal (CWE-35)
- SQL Injection (CWE-564)
- ReDoS (Regular Expression Denial of Service) (CWE-400)
- Cross-Site Scripting (XSS) (CWE-79)
- Insecure Deserialization (CWE-502)
- Zip Slip/Zip Bombs (CWE-35, CWE-400)
- Cryptographic Misuse (Weak PRNG, Key Reuse, ECB Mode)
- Memory Corruption (Buffer Overflow, Integer Overflow)
Simulation of Novice Behavior: The study simulated a novice programmer who lacks deep security knowledge. The researchers did not initially instruct ChatGPT to write secure code or sanitize inputs.
Iterative Interaction:
1. Initial Generation: ChatGPT generated the code.
2. Vulnerability Testing: The researchers analyzed the code. If a vulnerability was found, they provided a malicious input to trigger it.
3. Probing: They asked ChatGPT to explain the unexpected behavior or directly asked if the code was secure regarding a specific weakness.
4. Correction: They explicitly requested a "more secure version" of the code.
Evaluation Criteria: Programs were marked as Vulnerable (Y), Secure (N), or Unable to Generate (U) based on whether they contained the intended vulnerability.

3. Key Contributions

Empirical Security Assessment: This is one of the first studies to systematically evaluate the security of code generated by ChatGPT across multiple languages and vulnerability classes.
Interaction Analysis: The paper demonstrates that while ChatGPT often generates insecure code by default, it possesses the knowledge to identify and fix these vulnerabilities when explicitly prompted.
Pedagogical vs. Generative Role: The authors propose a workflow where ChatGPT acts as a pedagogical tool rather than a direct code generator. The suggested workflow involves: Generate $\rightarrow$ Test/Find Bugs $\rightarrow$ Ask "Why?" $\rightarrow$ Iterate to Fix.
Ethical Analysis: The paper highlights ethical contradictions, such as ChatGPT refusing to generate attack code but readily generating vulnerable code that could be exploited, and the risks of code secrecy when proprietary code is fed into the model.

4. Key Results

The results were largely concerning regarding the default output but promising regarding the model's potential for correction.

Initial Security Failure: Out of 21 programs, only 5 (24%) were initially secure. The majority contained critical vulnerabilities such as:
- Path Traversal & SQL Injection: Lack of input sanitization.
- Cryptographic Misuse: Using std::mt19937 (Mersenne Twister) instead of SecureRandom for passwords; using ECB mode for encryption; hardcoding keys.
- Memory Issues: Missing boundary checks, integer overflows, and unsafe memory allocation (e.g., malloc(0)).
- Logic Flaws: Generating insecure deserialization code or ReDoS-vulnerable regex patterns.
Correction Capability: After the researchers identified the flaws and explicitly asked for a fix, ChatGPT successfully produced secure code for 7 out of the 16 initially vulnerable programs.
- Successes: It successfully implemented prepared statements for SQL, added input validation for path traversal, and switched to secure random number generators when prompted.
- Failures: In some cases (e.g., Java deserialization, complex regex), the model admitted it could not rewrite the code or produced "corrected" code that still contained subtle flaws (e.g., timeouts set too high, missing null-termination checks).
Language Dependency: Security outcomes varied by language. For instance, the Java program for password generation used a secure PRNG, while the C++ and Python versions did not, despite identical prompts.
Adversarial Awareness: ChatGPT frequently failed to assume an adversarial model. It often suggested that vulnerabilities could be avoided simply by "not feeding invalid input," ignoring the reality of malicious actors.

5. Significance and Implications

Risk for Novice Developers: The study warns that relying on ChatGPT without security expertise is dangerous. Novice developers may accept insecure code as functional, leading to widespread deployment of vulnerable software.
The "Right Question" Barrier: The model's ability to produce secure code is heavily dependent on the user's ability to ask the "right questions." Users who do not know what to ask (e.g., "Is this vulnerable to SQL injection?") will likely receive insecure code.
Ethical Dilemmas: The paper raises significant ethical concerns:
- Refusal vs. Generation: ChatGPT refuses to write malware but generates vulnerable code that facilitates attacks.
- Data Privacy: Using ChatGPT for proprietary code risks leaking business secrets into the model's training data.
- False Confidence: The model often confidently asserts that insecure code is secure, potentially misleading developers.
Future Directions: The authors suggest that automated testing and static analysis should be integrated into the development loop with ChatGPT. The model should be viewed as a tutor to help developers learn security best practices rather than a replacement for skilled, security-conscious engineers.

Conclusion

The paper concludes that while ChatGPT is a powerful tool for code generation, it is currently not ready to replace skilled programmers for security-critical tasks. It frequently generates code that falls below minimal security standards. However, it shows significant potential as an interactive educational tool that can guide developers toward secure coding practices if the developers possess the requisite security knowledge to interrogate the model effectively.