LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
This paper introduces LangChoiceBench, a benchmark revealing that large language models exhibit a strong, often unjustified preference for Python in project-level code generation, characterized by low consistency between reasoning and implementation, limited language diversity, and a tendency to fabricate contextual evidence to support their choices.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a super-smart, hyper-creative assistant to build a house for you. You tell them, "I need a sturdy, weather-proof home," and they immediately start sketching. But here's the twist: you didn't tell them what to build it with. You didn't say "wood," "brick," or "steel." You just said "build a house."
In the world of artificial intelligence, these assistants are called Large Language Models (LLMs). They are the digital brains behind tools that can write stories, solve math problems, and, increasingly, write computer code. When you ask them to build a software project, they have to make a huge decision right at the start: which programming language should they use? It's like choosing between wood, steel, or glass. Some materials are great for a beach house, while others are perfect for a skyscraper, but using the wrong one can make the whole thing shaky or unsafe.
For a long time, scientists have noticed that these AI assistants seem to have a favorite material. No matter what kind of house you ask for, they almost always reach for Python. It's their go-to tool, their "default setting." But is this always the right choice? What if you asked for a high-speed race car (which needs lightweight, strong materials) and the AI built it out of heavy, slow wood just because it's familiar? That's the big question researchers are asking: Are these AI assistants making smart choices, or are they just blindly grabbing their favorite tool and hoping for the best?
The Great Language Heist: What LANGCHOICEBENCH Found
To find the answer, a team of researchers built a special testing ground called LANGCHOICEBENCH. Think of it as a giant, high-stakes video game level designed specifically to trick the AI. They created 28 different "missions" representing real-world software projects where Python is actually a terrible choice.
Imagine asking the AI to build a tiny robot that needs to react in a split second (like a self-driving car's brake system), or a super-fast trading platform that processes millions of dollars in milliseconds, or a mobile app that needs to run smoothly on a phone. In the real world, experts would use languages like C++, Rust, or Swift for these jobs because they are faster and more efficient. Python, in these scenarios, is like trying to win a Formula 1 race with a bicycle—it's just not built for the job.
The researchers asked 25 different AI models to tackle these missions. They wanted to see three things:
- The Habit: How often would the AI stubbornly pick Python even when it was a bad idea?
- The Hypocrisy: If the AI said, "You should use Swift for this," would it actually write the code in Swift, or would it sneakily switch back to Python?
- The Variety: Could the AI use a mix of different tools, or would it just stick to the same old favorite?
The Results: A Heavy Preference for the Familiar
The findings were a bit like watching a chef who only knows how to make pasta, even when you order a steak, a salad, or a soup.
- Python Overload: The AI models were incredibly stubborn. On average, they used Python for 35.3% of all the projects they built, even though the tasks were designed to be bad fits for it. In some cases, smaller AI models used Python for as much as 66.5% of their work. It was as if they were so used to using Python that they couldn't imagine using anything else.
- The "Say One Thing, Do Another" Problem: This was the most surprising part. When the researchers asked the AI, "What language should we use?", the models were actually pretty smart. They correctly suggested that Python was a bad idea for these specific tasks. They recommended better languages like C++ or Rust. But then, when it came time to actually write the code, they ignored their own advice!
- Only 48.8% of the time did the AI actually use one of the top three languages it had just recommended.
- For some models, this consistency was terrible. One model, for instance, recommended Swift and Kotlin for a mobile app but then built the whole thing in Python. It's like a travel agent telling you, "You need a heavy-duty suitcase for this trip," and then packing your clothes in a paper bag.
- The "Phantom Evidence" Glitch: The researchers also peeked inside the AI's "brain" (its reasoning process) to see why it made these choices. They looked at nearly 10,000 reasoning traces. They found that in 7.8% of the cases, the AI was making things up. It would invent a fake rule, like "The user asked for Python," or "The previous example used Python," even though no such rule existed. The researchers called this "phantom evidence." It's like a student who didn't study for a test, so they make up a story about how the teacher said the answer was "C," even though the teacher never said that.
Why Does This Happen?
The study suggests that for most AI models, picking a programming language isn't a deep, thoughtful decision. It's more like a reflex.
- The "Automatic" Switch: In 69.8% of the cases where the AI chose Python, it didn't even think about it. It just assumed Python was the answer and started coding.
- The "Easy Way Out": In another 20.5% of cases, the AI admitted it chose Python simply because it was "easier" or "faster to write," ignoring the fact that the project needed something tougher.
The researchers found that the bigger, more advanced "reasoning" models (the ones that are supposed to be better at thinking) weren't immune to this. They still fell into the trap of picking Python by default, or worse, they would think about using a different language, decide on it, and then switch to Python at the last second.
The Takeaway
The paper concludes that while these AI models are getting better at writing code, they are still unreliable decision-makers when it comes to choosing which language to use. They often ignore the specific needs of the project, lie about why they made a choice, and contradict their own advice.
The researchers aren't saying AI is broken; they are saying that we can't just let the AI pick the tools for us without checking. If you ask an AI to build a project, you can't just say "Make it work." You have to be the boss and say, "Use Rust for this part, and C++ for that part," because the AI might otherwise try to build a rocket ship out of cardboard just because it's the material it knows best. Until the AI learns to think more carefully about its choices, humans need to keep a close eye on the blueprint.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.