Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

The Great Code Translator Experiment

Imagine you have a super-smart robot friend who knows how to build anything, from a simple birdhouse to a complex spaceship. This robot, called a Large Language Model (or LLM), has read almost everything ever written on the internet, including millions of lines of computer code. Because it has read so much, it's incredibly good at following instructions to write new code for us. Usually, we talk to this robot in English because that's the language most of the code it learned from was written in. But what if you asked the robot to build a spaceship using instructions in Spanish, Hindi, or Chinese? Would the robot get confused? Would the spaceship fall apart? Or would it build the exact same thing, just with a different accent?

This is the big question behind a new study by a team of researchers. They wanted to see if the language we use to talk to these AI robots changes the quality of the code they write. In the world of software engineering, "code" is the set of instructions that tells a computer what to do. "Correctness" means the code actually works and passes all the tests, like a bridge holding up a car without collapsing. "Quality" is a bit broader; it means the code is easy to read, follows the rules of the neighborhood (like not leaving trash on the sidewalk), and doesn't have hidden traps that could break later. The researchers were curious if asking for a bridge in a language other than English would make the robot build a wobbly bridge, or if the robot is just as good at building in any language.

The Experiment: A Multilingual Coding Challenge

To find out, the researchers set up a massive coding contest. They took 460 different programming tasks—230 for Python and 230 for Java—from a famous benchmark called CoderEval. These weren't just simple "print hello" tasks; they were real-world challenges that required the AI to solve problems, handle data, and build structures.

Then, they did something clever. Instead of just asking the AI in English, they translated these 460 tasks into four other languages: Chinese, Hindi, Spanish, and Italian. But they didn't just use a quick machine translator. They had native speakers who are experts in computer science manually check and fix every single translation. They wanted to make sure the instructions were perfect and natural, just like a human asking a friend for help.

Next, they fed these translated instructions into three of the most powerful AI robots available today: GPT-4o mini, DeepSeek, and Claude. They asked each robot to solve the same 460 problems in all five languages (English plus the four others). In total, they generated thousands of pieces of code to compare.

The Big Surprise: English Isn't Always King

The results were a bit of a shock. The common belief was that if you ask the AI in English, you get the best code because that's the language the AI studied the most. But the study found that this isn't always true.

In fact, for Python tasks, asking the AI in Chinese actually led to better results than asking in English! The code generated from Chinese prompts passed more tests and worked more reliably. It's like asking a chef to cook a dish; sometimes, giving the recipe in a different language makes the chef think of a better way to cook it. However, this "Chinese advantage" didn't happen for every robot or every type of task. For Java, the results were more mixed, and no single language was clearly the winner. The main takeaway is that speaking English to an AI doesn't guarantee the best code, and speaking another language doesn't automatically mean the code will be worse.

The Code Looks Different, But Is It Worse?

The researchers also looked at the "quality" of the code, not just if it worked. They checked things like:

• Complexity: Is the code a tangled mess or a straight line?
• Comments: Did the robot explain what it was doing?
• Warnings: Did the code break any rules or look messy?

They found that the code generated in non-English languages wasn't necessarily "bad." It just looked different. For example, code written from Chinese prompts often had more comments, as if the robot was trying to be extra helpful. Code from Hindi prompts sometimes had more style mistakes, like missing punctuation or weird spacing, but these were usually small issues that could be easily fixed.

One interesting discovery was about Hindi. When the AI was asked to code in Hindi, it sometimes produced code that was a bit riskier, with more potential bugs or confusing logic. But for Spanish and Italian, the code was generally just as good as the English version, sometimes even better in terms of avoiding certain mistakes.

The "Mix-and-Match" Problem

Here is where things get a little messy. While the logic of the code might be great, the words inside the code were often a confusing mix. The researchers found that even when you asked the AI in Spanish or Chinese, the robot often wrote the comments (the notes explaining the code) and the variable names (the labels for data) in English.

It's like asking a chef to write a recipe in Italian, but they write the ingredient list in English and the instructions in a mix of both. The researchers found that the AI is very inconsistent. Sometimes it follows your language, sometimes it doesn't. This is a problem for teams of developers who need their code to be in one consistent language so everyone can understand it. The study suggests that just asking for code in a specific language isn't enough; you might need to be very specific about telling the robot to keep the notes and labels in that same language.

What Does This Mean for Us?

The study concludes that we don't need to be afraid of using our native languages to talk to AI coding assistants. You can ask for code in Spanish, Chinese, or Hindi without worrying that the robot will fail. In some cases, like Python, you might even get better results!

However, there are two things to watch out for:

1. The "Style" Issue: The code might have more small formatting errors (like missing commas or long lines) when written in non-English languages. But these are easy to fix with automated tools.
2. The "Language Mix" Issue: The AI might not keep the comments and labels in the language you asked for. If you need everything in one language, you might need to double-check the code or give the AI extra instructions.

Overall, this research shows that AI is becoming more flexible. It can understand and build things in many languages, breaking down the barrier that used to force everyone to speak English to get good code. But like any new tool, it has its quirks, and we need to learn how to work with them to get the best results.