💻 computer science

A Multi-Language Perspective on the Robustness of LLM Code Generation

This paper presents a comprehensive multi-language evaluation of large language models for code generation, revealing that performance consistently degrades under various prompt perturbations across different languages, with larger model sizes offering no reliable robustness gains and LLM-based docstring repair providing only marginal or even negative benefits.

Original authors: Fazle Rabbi, Zishuo Ding, Jinqiu Yang

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fazle Rabbi, Zishuo Ding, Jinqiu Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, hyper-intelligent assistant who can write computer code just by listening to your instructions. You ask it, "Write a function to sort a list of numbers," and it does so perfectly. You feel confident. But what happens if you make a tiny mistake? What if you accidentally type "sortt" instead of "sort," or describe the task as "arrange numbers in order" instead of "sort a list"?

This paper is essentially a stress test for these AI coding assistants. The researchers wanted to see: How fragile is this assistant? Does a tiny typo or a slightly different sentence make it completely break?

Here is the breakdown of their study using simple analogies:

1. The "One-Language" Trap

The Problem: Previous studies only tested these AI assistants on Python (a popular, easy-to-read programming language). It was like testing a car only on a smooth, sunny highway in California and assuming it would handle the same way on a snowy mountain road in Canada or a dusty desert in Africa.
The Study: The researchers decided to test the AI on three different "terrains":

Java: Strict and structured (like a formal business suit).
C++: Complex and powerful but tricky (like a high-performance race car with a manual transmission).
JavaScript: Flexible and loose (like a comfortable hoodie).

2. The "Twist" Experiments (Perturbations)

The researchers didn't just ask the AI to write code; they deliberately messed up the instructions in four specific ways to see how the AI reacted. Think of this as a game of "Telephone" where the message gets slightly distorted:

The "DocString" Twist (The Description): They changed the natural language description.
- Example: Changing "Sort the list" to "Arrange the numbers in order" or adding a typo like "sortt the list."
- Analogy: Asking a chef to "bake a cake" vs. "bake a cak" vs. "make a sweet dessert."
The "Function Name" Twist: They changed the name of the function the AI was supposed to write.
- Example: Changing calculateTotal to calculateTota1 (with a number 1) or calc_total.
- Analogy: Calling a friend "Bob" vs. "Rob" vs. "B0b."
The "Syntax" Twist: They changed the structure of the code already provided.
- Example: Swapping the order of numbers in a math equation or changing a for loop to a while loop.
- Analogy: Giving a recipe where you swap the order of ingredients but keep the amounts the same.
The "Format" Twist: They changed the spacing, tabs, or line breaks.
- Example: Removing all the spaces or using tabs instead of spaces.
- Analogy: Writing a letter with no spaces between words.

3. The Shocking Results

The researchers found that all the AI models were surprisingly fragile. Even the smartest, largest models (which you'd expect to be like a genius PhD student) failed significantly when the instructions were slightly tweaked.

Size Doesn't Save You: Bigger models (with more "brain power") weren't necessarily more robust. Sometimes, the bigger models were more brittle than the smaller ones. It's like a giant, expensive robot that trips over a pebble, while a small, simple toy car drives right over it.
Language Matters:
- Java was the most resilient (the sturdy sedan).
- C++ was the most fragile (the sensitive race car).
- JavaScript was somewhere in the middle.
- Key Takeaway: You can't assume an AI is good at coding just because it's good at Python. It might crash and burn in Java or C++.
Meaning vs. Spelling: Surprisingly, changing the meaning of a sentence (even slightly) was just as dangerous as changing the spelling. The AI relies heavily on specific keywords and phrasing.

4. The "Fix-It" Attempt

The researchers tried a clever trick: Can we use an AI to fix the broken instructions before the coding AI sees them?

The Idea: If the prompt says "sortt the list," use an AI to correct it to "sort the list" first.
The Result: It worked a little bit for simple typos (like fixing a spelling error), but it failed miserably for deeper changes. If the description was rephrased or the meaning shifted slightly, the "fixer" AI couldn't tell what the original intent was, and sometimes made it worse.
Analogy: If you ask a translator to fix a sentence that was translated poorly, they might fix the grammar, but they might not realize the meaning was already lost in the first translation.

5. Why Should You Care?

This paper is a wake-up call for developers and companies using AI to write code.

Don't Trust the "Pass Rate": Just because an AI passes a test 90% of the time doesn't mean it's reliable. A tiny typo could cause that 90% to drop to 10%.
Be Careful with Prompts: You need to be very precise with your instructions. If you are using AI to write code for a banking app (Java) or a game engine (C++), you can't just assume it will work because it worked on a Python tutorial.
Robustness is a Feature: We need to build AI that is "tougher"—like a car with better suspension that can handle potholes, not just smooth highways.

In a nutshell: AI code generators are incredibly smart but surprisingly sensitive. They are like high-performance sports cars that need perfect fuel and a smooth road. If you give them a slightly different instruction or a different programming language, they might stall out. We need to stop treating them like magic wands and start testing them like the fragile tools they currently are.

1. Problem Statement

While Large Language Models (LLMs) have achieved significant success in code generation, their robustness—the ability to maintain performance under slight input perturbations—remains a critical, under-explored issue.

Limitations of Prior Work: Previous robustness studies (e.g., ReCode) focused almost exclusively on Python, overlooking other widely used languages with distinct syntax and semantics (Java, C++, JavaScript).
Evaluation Gaps: Existing benchmarks often rely on the HumanEval dataset, which has been shown to have insufficient test coverage, potentially masking failures that would be exposed by more rigorous test suites like EvalPlus.
Mitigation Unknowns: It is unclear whether robustness losses caused by perturbed prompts (e.g., typos in docstrings or function names) can be mitigated by automatically repairing the prompt using an LLM.

2. Methodology

The authors propose a comprehensive framework to evaluate and analyze LLM robustness across multiple languages.

A. Dataset Construction: EvalPlus-X

To address the limitations of HumanEval, the authors created EvalPlus-X, a multilingual extension of the EvalPlus benchmark.

Languages: Java, C++, and JavaScript (excluding Python as it was covered in prior work, and Go due to paradigmatic similarity to C++/Java).
Process:
- Translated EvalPlus test cases and canonical solutions from Python to the target languages.
- Manually updated canonical solutions to handle corner cases and edge inputs that the original HumanEval-X solutions missed.
- Ensured type-system compatibility (e.g., converting Python lists to Java ArrayList or C++ vector).
Scale: The resulting dataset contains significantly more test cases per problem (average ~~700–980) compared to the original HumanEval-X (~~7–8), providing a much stricter evaluation standard.

B. Perturbation Strategy

The study applies 29 semantic-preserving perturbations across four categories to the input prompts:

DocString (10 types): Natural language transformations including synonym substitution, back-translation, tense changes, and character-level noise (e.g., "Butter Fingers" typos).
Function Name (7 types): Renaming functions via case changes (snake_case $\leftrightarrow$ camelCase), character swaps, and synonym insertion.
Syntax (6 types): Applied to code completion tasks (partial code), including dead code insertion, loop transformation (for $\leftrightarrow$ while), and operand swapping.
Format (6 types): Changes in indentation, newlines, and comment styles (e.g., converting docstrings to inline comments).

C. Experimental Setup

Models: Six LLMs were evaluated: Incoder (1B, 6B), CodeGen-Multi (2B, 6B), Magicoder-S-DS-6.7B, and Qwen2.5-Coder-7B-Instruct.
Metrics:
- Robust Pass (RPs@k): Percentage of perturbed prompts passing tests.
- Robust Drop (RDs@k): Performance degradation relative to the nominal (unperturbed) prompt.
- Robust Relative (RRs@k): Measures cases where the model fails the nominal prompt but passes the perturbed one (or vice versa).
Mitigation Experiment (RQ3): The authors tested if an LLM (Magicoder-7B) could repair perturbed docstrings before code generation to restore performance.

3. Key Contributions

Multilingual Robustness Framework: Extended the ReCode framework from Python-only to Java, C++, and JavaScript, revealing that robustness is a language-dependent property.
EvalPlus-X Benchmark: Released a rigorous, multilingual dataset with expanded test cases and corrected canonical solutions, addressing the "weak test case" problem of previous benchmarks.
Comprehensive Perturbation Analysis: Conducted the first comparative study evaluating code generation models across multiple languages using 29 distinct perturbation types.
Mitigation Strategy Evaluation: Investigated the efficacy of LLM-based prompt repair, finding it largely ineffective for semantic perturbations.
Open Resources: Publicly released the dataset, perturbation scripts, and evaluation code.

4. Key Results

RQ1: How robust are LLMs across different languages?

Universal Degradation: All models consistently degrade under perturbations across all three languages.
Language Sensitivity:
- Java: Most resilient. Its strict syntax and type system allow models to infer intent despite surface changes.
- C++: Most fragile. Complex syntax and variability lead to the largest performance drops.
- JavaScript: Intermediate resilience; dynamic typing offers some flexibility but still suffers significant drops.
Model Size vs. Robustness: Larger model size does not guarantee better robustness. In several cases, larger models (e.g., 6B/7B) were more brittle than smaller ones under semantic perturbations.
Perturbation Impact: Semantic perturbations (changing meaning slightly via synonyms or tense) are at least as disruptive as syntactic ones (typos).

RQ2: What factors drive robustness differences?

Feature Correlation:
- Java & JavaScript: Highly sensitive to DocString and Function Name changes.
- C++: More sensitive to Syntax-level perturbations.
Failure Patterns: Models generate structurally different code when prompts are perturbed, leading to failures in corner cases that the nominal code might have passed.

RQ3: Can prompt repair mitigate robustness drops?

Partial Success: Using an LLM to repair perturbed docstrings yields marginal gains (approx. 6–12% recovery) for surface-level perturbations (typos, whitespace).
Semantic Failure: The strategy is ineffective for semantic perturbations (e.g., BackTranslation, Synonym Substitution). Since these perturbed texts are often grammatically correct and fluent, the repair model cannot identify an error to fix, and sometimes even degrades performance by altering the intent.

5. Significance and Implications

For Practitioners: High nominal accuracy (pass@1) is a poor predictor of robustness. Developers cannot rely on LLMs for critical code generation without robustness testing, especially when prompts might contain natural variations or typos.
For Model Builders: Scaling up model parameters is not a silver bullet for robustness. Robustness requires targeted training strategies and diverse input exposure. Models must be evaluated per language, as performance in Python does not generalize to C++ or Java.
For Researchers:
- Benchmarks must move beyond Python and incorporate rigorous test suites (like EvalPlus) to avoid masking failures.
- Robustness testing must include diverse perturbation types (semantic, formatting, not just syntax).
- Simple prompt repair is insufficient; future work needs context-aware repair strategies (e.g., using function signatures as anchors) rather than just grammar correction.

Conclusion

This study demonstrates that LLM code generation is inherently fragile across multiple programming languages. Robustness is not a uniform property but varies significantly based on the language's syntax, the model's architecture, and the type of perturbation. The authors provide a new benchmark (EvalPlus-X) and evidence that current mitigation strategies (prompt repair) are limited, calling for more robust training methodologies and evaluation standards in the field.