💻 computer science

A Study of LLMs' Preferences for Libraries and Programming Languages

This paper presents the first empirical study revealing that large language models exhibit strong biases toward popular libraries like NumPy and the Python language, often prioritizing familiarity over task-specific optimality and necessitating targeted improvements in training and evaluation.

Original authors: Lukas Twist, Jie M. Zhang, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lukas Twist, Jie M. Zhang, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of incredibly talented, super-fast apprentices. They can write code (the instructions that tell computers what to do) faster than any human. But there's a catch: these apprentices don't just write code; they also make design choices. They decide which tools to use and which language to speak.

This paper is like a detective story where researchers went behind the scenes to ask: "Do these AI apprentices actually know the best tools for the job, or do they just pick the ones they've seen the most?"

Here is the breakdown of their findings, using some everyday analogies.

1. The "Familiarity Trap" (Libraries)

Think of libraries in programming like kitchen appliances.

The Situation: You ask an apprentice to bake a cake.
The Ideal Choice: You need a mixing bowl and a whisk. Simple, effective.
The AI's Choice: The apprentice grabs a massive, industrial-grade food processor (a library called NumPy) to mix the batter.

What the study found:
The AI models have a massive habit of grabbing the "famous" tools, even when they aren't needed.

They love NumPy and Pandas (the food processors of the coding world).
In nearly 45% of cases, the AI used these heavy-duty tools even when the task didn't require them.
The Analogy: It's like using a sledgehammer to crack a nut. It works, but it's overkill, slower, and makes the kitchen (your code) messy and harder for others to clean up. They ignore newer, lighter, faster tools (like Polars or FastAPI) because they haven't seen them as much in their training data.

2. The "One-Size-Fits-All" Language (Programming Languages)

Think of programming languages like vehicles.

Python is a minivan: Great for families, easy to drive, comfortable, and good for most daily errands.
Rust, C++, or Go are Formula 1 race cars or heavy-duty trucks: Built for speed, safety, and heavy lifting.

What the study found:
The AI models are obsessed with the minivan.

Even when the job requires a race car (like building a high-speed trading platform or a system that needs to be crash-proof), the AI almost always hands you a minivan (Python).
In 58% of cases where a high-performance language was the obvious choice, the AI still picked Python.
The Analogy: It's like asking a mechanic to build a rocket ship, and they hand you a Toyota Camry because, "Well, everyone drives a Camry, and it has four wheels, so it must work for space travel."
The AI ignores the race cars (Rust) entirely in many high-stakes scenarios, even though those tools are safer and faster for the specific job.

3. The "Talk vs. Action" Gap

This is the most surprising part. The researchers asked the AI two things:

What would you recommend? (The "Consultant" mode)
Now, actually write the code. (The "Worker" mode)

What the study found:
The AI is a hypocrite.

When asked for advice, the AI might say, "For this high-speed task, you should definitely use Rust."
But when it actually writes the code, it immediately switches back to Python.
The Analogy: It's like a nutritionist who tells you, "You should eat a salad for dinner," but then goes home and eats a giant pizza. The AI knows the "right" answer in theory, but its habits (training data) pull it back to the "easy" answer in practice.

Why Does This Matter?

You might think, "So what? If the code works, isn't that good?"

The Problem:

The Feedback Loop: If everyone uses the AI, and the AI always picks the "popular" tools, then the internet gets flooded with code using those same tools. This makes the AI even more likely to pick them next time.
Stifling Innovation: New, better tools (like the newer libraries or languages) never get a chance to grow because the AI never recommends them.
Performance Issues: Using a minivan for a race can lead to crashes or slow performance. In software, this means slower apps, higher costs, and security risks.

The Bottom Line

The study concludes that these AI models are conservative. They prioritize popularity and familiarity over suitability and optimization.

They are like a student who always answers questions with the first thing they memorized in class, rather than thinking about what the specific question actually needs. To fix this, the researchers suggest we need to teach these AIs to be more flexible, diversify what they learn, and stop letting them just "go with the flow" of what's popular.

1. Problem Statement

While Large Language Models (LLMs) have achieved significant success in code generation, existing evaluations primarily focus on functional correctness and syntactic validity. There is a critical gap in understanding the design choices LLMs make, specifically regarding:

Library Selection: Which external libraries LLMs choose when prompts do not specify them.
Programming Language Selection: Which language LLMs default to when the task description is language-agnostic.

These choices have tangible impacts on software performance, maintainability, ecosystem diversity, and security. The authors hypothesize that LLMs may exhibit systematic biases toward popular or "familiar" technologies (e.g., Python, NumPy) even when they are suboptimal for specific tasks (e.g., high-performance computing), potentially reinforcing a homogenized software ecosystem.

2. Methodology

The study employs an empirical approach involving eight diverse production-grade LLMs (including GPT-4o-mini, Claude-3.5, Llama-3.2, Qwen-2.5-Coder, and DeepSeek-LLM). The experiments are divided into three main parts:

A. Experiment 1: Library Preferences

Benchmark Tasks: Used BigCodeBench (525 filtered Python tasks) where the ground truth solutions use specific libraries. The prompt explicitly requested the use of an external library but did not name one.
Project Initialization: Generated 100 responses per LLM for five open-ended project types (Database, Deep Learning, Distributed Computing, Web Scraper, Web Server) to observe library adoption in a realistic, unconstrained setting.
Analysis: Compared LLM choices against ground-truth solutions and analyzed the "momentum" of libraries (using GitHub star growth) to identify if LLMs favor established libraries over high-growth alternatives.

B. Experiment 2: Language Preferences

Benchmark Tasks: Utilized six language-agnostic datasets (Multi-HumanEval, MBXP, AixBench, CoNaLa, APPS, CodeContests) where prompts did not specify a language.
Project Initialization: Generated code for five specific project types where Python is known to be suboptimal due to performance constraints (e.g., Concurrent Web Server, Low-Latency Trading Platform, System-level Application).
Analysis: Measured the frequency of language selection, specifically tracking the dominance of Python versus performance-oriented languages like Rust, C++, or Go.

C. Experiment 3: Recommendation Consistency

Method: Prompted LLMs to provide Natural Language (NL) recommendations (ranking the best libraries/languages) for the same project initialization tasks used in Experiments 1 and 2.
Metric: Calculated Kendall's $\tau_b$ correlation coefficient to measure the alignment between the LLM's stated recommendations and its actual code generation choices.

3. Key Results

A. Library Preferences: Overuse of Established Libraries

Dominance of "Big Three": Across all models, NumPy, pandas, and Matplotlib were the most frequently used libraries, often appearing in tasks where they were not required by the ground truth.
Overuse Rate: In up to 45% of cases, LLMs imported NumPy even when the ground-truth solution did not use it.
Lack of Diversity: LLMs used a surprisingly narrow set of distinct libraries (32–39 per model) across hundreds of tasks.
Ignoring High-Momentum Alternatives: LLMs consistently ignored newer, faster-growing libraries.
- Example: Polars (faster, higher growth) was used 0% of the time compared to pandas (58%).
- Example: FastAPI (modern, high adoption) was used in only 9% of web server tasks, while Flask (older) was used in 88%.
- Example: Ray and Celery were largely ignored in favor of Dask for distributed computing.

B. Language Preferences: Strong Python Bias

Benchmark Dominance: For standard coding benchmarks, Python was the default choice in 93.5% to 100% of responses across all models.
Suboptimal Scenarios: Even in project initialization tasks explicitly designed for high performance (where Python is suboptimal), Python remained the most used language in 58% of cases.
Rust Neglect: Despite being the optimal choice for memory safety and concurrency in the tested scenarios, Rust was never used in any of the project initialization responses.
Source of Bias: The authors posit this bias stems from training data distributions (GitHub archives) and post-training fine-tuning strategies that favor Python.

C. Recommendation Consistency: Low Correlation

Weak Signal: There is a significant disconnect between what LLMs say is the best technology and what they use.
Statistical Findings:
- For library tasks, correlations were inconsistent (moderate agreement only in specific domains like distributed computing).
- For language tasks, there was no statistically significant correlation.
- In project initialization tasks, LLMs contradicted their own language recommendations 83% of the time. For instance, an LLM might recommend Rust for a low-latency task but generate Python code.

4. Key Contributions

First Empirical Study: This is the first systematic investigation into LLM preferences for libraries and programming languages, moving beyond simple correctness metrics.
Quantification of Bias: The study quantifies the extent of "popularity bias," showing that LLMs prioritize familiarity over task-specific optimality (e.g., choosing Python for low-latency systems).
Inconsistency Discovery: It reveals that LLMs' natural language advice is a poor predictor of their actual code generation behavior, highlighting a lack of internal consistency in their "knowledge" of technology suitability.
Open Resources: The authors released the code, datasets, and full results on GitHub to facilitate further research into LLM code bias.

5. Significance and Implications

Ecosystem Homogenization: The tendency of LLMs to default to popular libraries and Python risks creating a feedback loop where these technologies become even more dominant, stifling innovation and the adoption of superior, newer tools (e.g., Polars, Ray, Rust).
Performance and Safety Risks: By defaulting to interpreted languages (Python) for high-performance tasks, LLMs may inadvertently generate code that is inefficient, memory-unsafe, or unsuitable for production environments requiring low latency.
Developer Trust: The inconsistency between NL recommendations and code generation suggests that developers cannot rely on LLMs' verbal advice for architectural decisions without verification.
Future Directions: The paper calls for:
- Targeted Fine-tuning: To reduce bias toward popular but suboptimal choices.
- Data Diversification: Ensuring training data represents a broader range of languages and libraries.
- New Benchmarks: Evaluation metrics that explicitly measure "design choice fidelity" and diversity, not just functional correctness.

In conclusion, the paper argues that while LLMs are powerful code generators, their inherent biases toward established technologies and languages pose a risk to the diversity and optimality of the future software ecosystem. Addressing this requires changes in training data, model alignment, and evaluation standards.