💻 computer science

Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs

This study reveals that even minute, single-character perturbations in prompts can cause coding LLMs to generate vulnerable code, with input-handling flaws being more predictable from hidden states than secure-defaults errors, thereby expanding the security threat model beyond prompt injection to include ordinary prompt variations.

Original authors: Alexander Sternfeld, Andrei Kucharavy, Ljiljana Dolamic

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Sternfeld, Andrei Kucharavy, Ljiljana Dolamic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very talented, super-fast apprentice coder. This apprentice (an AI) can write entire programs in seconds. You give them a simple instruction, like "Write a function to unzip a file safely." Usually, they do a great job.

But this paper asks a scary question: What if the apprentice is incredibly sensitive to tiny mistakes in how you ask the question?

The researchers found that if you change just one letter in your instruction—maybe a typo, or swapping a word for a similar one—the code the apprentice writes can suddenly go from "safe and secure" to "full of holes that hackers can walk through."

Here is a breakdown of their findings using everyday analogies:

1. The "One-Letter" Domino Effect

Think of the instruction you give the AI as a recipe. The researchers found that if you change just one letter in the recipe (like changing "salt" to "salty"), the resulting dish might not just taste slightly different; it could become poisonous.

The Discovery: They tested three different AI models and five programming languages. They found that changing a single character in the prompt could flip the code from secure to vulnerable.
The Analogy: It's like telling a chef, "Make sure the door is locked," versus "Make sure the door is locked" (with a typo). In this specific case, the AI might completely forget to lock the door, leaving the house wide open.

2. Two Different Types of "Mistakes"

The researchers noticed that not all security holes are created equal. They found two distinct categories, which behave differently:

Type A: The "Missing Guard" (Input Handling)
- What it is: The AI forgets to add a safety check, like a bouncer checking IDs at a club.
- The Finding: The AI's internal "brain" (hidden states) actually shows signs of this mistake before it even writes the code. It's like seeing the chef reach for the wrong ingredient before they start cooking. The researchers could predict these errors with about 75% accuracy just by looking at the AI's thought process.
- Why: The AI has to decide early on to build a whole new "safety structure" into the code. That decision is visible early.
Type B: The "Weak Choice" (Secure Defaults)
- What it is: The AI builds the safety structure, but picks a weak lock (like a password "1234" instead of a complex one).
- The Finding: These are much harder to predict. The AI's internal brain looks fine right up until the very last moment. The decision to pick the weak lock happens so late in the process that the "early warning signal" is missing. The researchers could only predict these about 67% of the time.
- Why: It's like the chef deciding to use a flimsy lock on the door only after the house is already built. The blueprint looked perfect, but the final choice was a bad one.

3. Where the Mistake Happens Matters

The researchers also looked at where in the instruction the typo happened.

The Middle is Critical: They found that typos in the middle of the instruction were the most dangerous.
The Analogy: Imagine a sentence: "Please lock the front door and the back door." If you typo the word "front" in the middle, the AI might get confused about which door to lock. If you typo the very first or last word, the AI is more likely to ignore it or guess correctly. The "meat" of the instruction is where the AI is most fragile.

4. The "Crystal Ball" (Probing)

The team built a "crystal ball" (a mathematical probe) that looks at the AI's internal state right after it reads your prompt but before it writes any code.

The Result: This crystal ball can tell you if the AI is about to write code with "Missing Guard" errors (Type A) quite well.
The Limitation: It struggles to predict "Weak Choice" errors (Type B). This suggests that for some security issues, we can catch the AI before it starts writing, but for others, we might need to check the code while it's being written or after it's done.

The Bottom Line

The paper concludes that we cannot assume our AI coding assistants are robust. A simple typo or a slight rephrasing of a request can accidentally create a security vulnerability.

Good News: We can sometimes detect these risks early by looking at the AI's internal "thoughts."
Bad News: We can't catch all risks this way, especially the ones where the AI makes a single, bad choice at the very end of the process.

Important Note: The researchers emphasize that they did this by making random, accidental-looking typos (like a human might make), not by trying to trick the AI on purpose. This means the danger is real even for normal, everyday developers who just want to get their work done.

Technical Summary: Minimal Prompt Perturbations Lead to Code Vulnerabilities

Problem Statement

While Large Language Model (LLM)-based coding assistants are rapidly adopted to enhance developer productivity, the security of the code they generate remains a critical concern. Prior research has established that minor prompt perturbations (e.g., typos, paraphrasing) can degrade the functional correctness of generated code. However, the security consequences of such perturbations remain unstudied. A natural assumption in development is that surface-level variations in prompts should not alter the security posture of the output. This paper investigates whether minimal, benign prompt mutations can flip LLM-generated code from secure to vulnerable, and whether these vulnerabilities are encoded in the model's internal representations before code generation begins.

Methodology

Benchmark and Models

The study utilizes the CWEval benchmark (Peng et al., 2025), which is distinct from prior security benchmarks (e.g., CyberSecEval, SecurityEval) because it evaluates functionality and security simultaneously via outcome-driven test oracles. The benchmark covers 31 Common Weakness Enumeration (CWE) types across five programming languages (C, C++, Go, JavaScript, Python).

The experiments employ three open-source, code-specialized LLMs:

CodeLlama (70B)
DeepSeek-Coder (33B)
Qwen3-Coder (30B)

Experiments were conducted with a temperature of $T=0$ to ensure stable evaluation, with sensitivity analyses performed at $T=0.8$ .

Perturbation Strategy

The authors applied systematic mutations to the prompts, categorized by increasing semantic disruption:

Single-character changes: Substituting one character with a random ASCII letter.
Three-character changes: Swapping three random characters within a single token.
Whole-token replacements: Replacing a token with a similar random token based on embedding cosine similarity.

Hidden State Probing

To determine if vulnerability signals exist prior to generation, the authors trained linear (L2-regularized logistic regression) and shallow non-linear (two-layer MLP) classifiers on the last-token hidden state of the prompt. These probes predict whether a specific (model, language, CWE, mutation) instance will result in:

Functional code.
Code that is both functional and secure.

Probes were trained per (model, language, CWE, target) cell using an 80/20 train/test split with 5-fold cross-validation for hyperparameter tuning.

Key Contributions

Demonstration of Prompt Fragility: The study shows that mutations as small as a single character can flip generated code from secure to vulnerable (and vice versa), affecting both functionality and security simultaneously.
Decomposition of Mutation Drivers: The research identifies that for some perturbations, the position of the mutation in the prompt is decisive (e.g., mutations in the central part of the prompt are most harmful), while for others, the specific character substitution determines the outcome.
Systematic Variation in Probe Performance: The paper demonstrates that the predictability of vulnerabilities from hidden states varies systematically by CWE category. Vulnerabilities requiring structural changes (input handling) are more predictable than those relying on single local choices (secure defaults).

Results

Impact of Mutations

Prevalence: Mutations affect a large fraction of CWEs across all models and languages.
Directionality: Mutations generally cause more harm than good. For CodeLlama (70B) and DeepSeek-Coder (33B), the rate of deterioration (flipping secure code to vulnerable) significantly exceeds the rate of improvement.
Model Robustness: Qwen3-Coder (30B) exhibited the most robust performance, showing consistently smaller degradation bars compared to the other two models.
Language Differences: Patterns diverge by language. For Python, many CWEs are affected, but the fraction of mutations changing the outcome within each CWE is low. Conversely, for Go, fewer CWEs are affected, but those that are affected respond strongly to mutations.

Probing Analysis

Overall Predictability: The mean Area Under the Curve (AUC) for predicting joint functional and secure outcomes from prompt-end hidden states is approximately 0.70 across models.
CWE-Specific Variance:
- Group I (Input-Handling): Vulnerabilities where untrusted input reaches a sensitive sink (e.g., SQL injection, path traversal) require adding validation logic. These are highly predictable from hidden states (Mean AUC: 0.753).
- Group D (Secure Defaults): Vulnerabilities where the fix involves a single local choice (e.g., selecting a strong algorithm, safe flag, or key size) are less predictable (Mean AUC: 0.674).
Statistical Significance: A one-sided Mann-Whitney U test confirms a statistically significant gap between the two groups ( $p = 0.009$ ).

Interpretation of Probing Results

The authors posit that the prompt-end hidden state encodes decisions the model commits to early (structural changes like adding a validation guard). In contrast, decisions deferred to specific token positions during generation (like choosing a specific algorithm parameter) are dominated by the next-token distribution and are thus less visible in the final prompt representation.

Significance and Claims

The paper claims that the threat model for LLM-assisted coding extends beyond adversarial prompt injection to include ordinary prompt variation (typos, paraphrasing). The findings indicate that:

Input-handling flaws can potentially be detected and mitigated before generation begins by analyzing prompt representations.
Secure-defaults flaws are harder to detect at the prompt stage and may require intervention during the decoding process.

The authors maintain a defensive orientation, emphasizing that their work identifies fragility under benign use rather than providing exploit recipes. They note limitations regarding the computational cost of probing frontier-scale closed models and the manual nature of the CWE grouping, suggesting future work could probe latent representations during generation to capture the "secure-defaults" signal missed at the prompt end.