AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents
This empirical study reveals that AI-generated code from state-of-the-art LLM agents currently lacks full reproducibility, with only 68.3% of projects executing successfully in clean environments due to significant gaps between declared and actual runtime dependencies.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you ask a super-smart, hyper-fast robot chef to cook you a complex meal. The robot hands you a beautifully plated dish and a recipe card that says, "You just need flour, eggs, and sugar."
You take the recipe home, mix those three ingredients, and try to bake the cake. It explodes.
Why? Because the robot forgot to tell you that the flour needed a specific type of yeast, the eggs needed to be from free-range chickens (a specific brand), and the sugar needed to be dissolved in hot water first. The robot saw the idea of the cake, but it didn't understand the reality of the kitchen.
This is exactly what a new study found about AI coding agents (like Claude, OpenAI's Codex, and Google's Gemini). These AIs are great at writing computer code, but they are terrible at telling you what else you need to make that code actually run.
Here is the breakdown of the study using simple analogies:
1. The "Out-of-the-Box" Test
The researchers asked three different AI chefs to build 300 different software projects (like a weather app, a data analyzer, or a website). They gave the AIs a strict rule: "Give me everything I need to run this on a brand-new, empty computer."
The Result:
- 68% of the time, the code worked perfectly right away.
- 32% of the time, the code failed immediately. It was like trying to start a car with no gas.
The scary part? Even when the code did work, it was often a fluke. The AI had hidden secrets it didn't tell you about.
2. The "Iceberg" of Dependencies
In the world of coding, "dependencies" are just other programs or libraries your code needs to borrow to do its job. Think of them like ingredients.
The study found a massive gap between what the AI said you needed and what you actually needed.
- The Claim: The AI says, "You only need 3 ingredients."
- The Reality: When you actually run the code, it turns out you needed 37 ingredients.
The Metaphor:
Imagine the AI tells you to build a house and says, "You just need a hammer and some nails." You show up with a hammer and nails, but you can't build a house because you also need wood, concrete, electricity, plumbing, and a permit.
- For Python (a popular coding language), the AI was pretty good, only missing about 2 extra ingredients on average.
- For Java (used for big enterprise systems), the AI was terrible. It claimed you needed 2 ingredients, but you actually needed 21. That's an 18x gap!
3. The "Specialist" Chefs
The study tested three different AI models, and they turned out to have very different personalities and skills, even though the companies don't advertise this.
- Gemini: The Python Wizard. It was perfect at Python (100% success rate) but completely failed at Java (only 28% success). It's like a chef who is a master at making sushi but burns every steak you ask for.
- Claude: The Enterprise Pro. It was surprisingly good at Java (80% success), which is usually the hardest language for AIs to handle. It's the chef who can handle the complex, high-end restaurant kitchen.
- Codex: The Scripting Specialist. It was okay at Python and JavaScript but struggled with the big, complex systems.
The Lesson: You can't just pick an AI because it's "the best." You have to pick the one that specializes in the specific language you are using.
4. The Real Problem: It's Not Just Missing Ingredients
The title of the paper says "Dependency Gaps," but the study found something even more worrying.
When the code failed, missing ingredients (dependencies) were only 10% of the problem.
The other 50%+ of the time, the code itself was broken.
- The AI wrote code that looked correct but had logic errors (like a recipe that says "bake for 1000 hours").
- The AI put files in the wrong folders.
- The AI forgot to initialize variables (like trying to pour water into an empty bucket that doesn't exist yet).
It's like the robot chef not only forgot the ingredients but also wrote the recipe in a language you can't read, or told you to mix the eggs after the cake was already baked.
5. The Hidden Cost
If you are a developer using these tools, here is the reality:
- You ask the AI to build a tool.
- It gives you the code.
- You try to run it, and it crashes.
- You spend 15 minutes (on average) debugging, adding missing packages, and fixing broken logic.
If a company uses AI to generate 100 projects, they will spend roughly 8 hours just fixing the mistakes the AI made. That's a lot of "hidden tax" on productivity.
The Bottom Line
AI coding agents are like sophisticated autocomplete tools, not yet true development partners. They can write the logic of the code, but they haven't learned to understand the environment the code lives in.
Until AI learns to say, "Here is the code, and here is the exact list of 37 packages, the specific version of the operating system, and the configuration file you need to make this run," we are still in a "Reproducibility Crisis."
In short: The AI is great at writing the story, but it keeps forgetting to tell you which library you need to read it. Until it fixes that, we can't fully trust it to build our software alone.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.