Usage, Effects and Requirements for AI Coding Assistants in the Enterprise: An Empirical Study
This paper presents an empirical study surveying 57 developers and analyzing 35 existing surveys to evaluate the readiness, impact, and requirements of AI coding assistants and CodeLLMs for enterprise software engineering.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef in a massive, high-stakes kitchen (the enterprise). For years, you've had to chop every vegetable, measure every spice, and write every recipe from scratch. Then, a new kitchen assistant arrives: an AI Coding Assistant. It's like a super-smart sous-chef that can instantly suggest recipes, chop vegetables at lightning speed, and even write the menu for you.
This paper is a report card on how well this new sous-chef is actually working in real-world corporate kitchens. The authors (researchers from IBM) didn't just watch the assistant work; they asked 57 real chefs (developers) from different departments (finance, research, sales) how they feel about it. They also looked at 35 other reports from around the world to see the bigger picture.
Here is the breakdown of their findings, using simple analogies:
1. The "Speed vs. Quality" Trade-off
The Claim: Most chefs feel faster. About 88% of the developers said the AI made them more productive. Some said they were twice as fast.
The Analogy: It's like having a power drill instead of a hand screwdriver. You can put up shelves in half the time. However, just because you can drill a hole fast doesn't mean the shelf is perfectly level. The paper notes that while speed is up, there are worries about whether the code (the shelf) is secure and easy to fix later. Some chefs are so confident in the power drill that they forget to check if the wall is solid.
2. One Size Does Not Fit All
The Claim: The paper found that different departments use the AI for totally different reasons.
The Analogy:
- The Researchers use the AI like a library card catalog. They ask it to find obscure information or generate complex data sets to test new theories.
- The Sales Team uses it like a marketing copywriter. They want it to help "sell a product" or generate synthetic data to show clients.
- The Software Engineers use it like a spell-checker and auto-complete. They want it to write boring, repetitive code so they can focus on the creative design of the app.
- The Takeaway: You can't just give everyone the same tool settings. A "one-size-fits-all" approach doesn't work; the AI needs to be tailored to the specific job.
3. The "Trust but Verify" Problem
The Claim: Developers keep about 25% to 50% of the code the AI writes. They don't just copy-paste everything.
The Analogy: Think of the AI as a junior intern. The intern is great at drafting a report, but they might make up facts or use the wrong tone. The senior developer (the human) has to read every word, fix the errors, and make sure it sounds right.
- If the AI writes code that looks perfect but has a hidden security flaw (like a lock that looks nice but doesn't actually lock), the human has to catch it.
- The paper found that if the AI makes too many mistakes, or if the human has to spend more time fixing the AI's work than writing it themselves, the tool becomes a burden, not a help.
4. What the Chefs Want Next (The Wishlist)
The developers told the researchers exactly what they need the "super sous-chef" to do next. They split these requests into two categories:
Short-Term Requests (The "Fix the Kinks" list):
- Better Memory: The AI needs to know the whole kitchen, not just the counter it's standing on. It needs to understand the entire codebase (the whole restaurant's history) without getting confused.
- Seamless Integration: Stop making us switch windows. The AI should be right inside the tool we use, like a built-in oven timer, not a separate app we have to open.
- Honesty: If the AI isn't sure, it should say, "I'm not 100% on this," instead of confidently giving a wrong answer.
Long-Term Requests (The "Sci-Fi Dream" list):
- The Architect: Instead of just chopping vegetables, the AI should be able to design the whole kitchen layout from scratch based on a vague idea.
- The Autonomous Agent: The AI should be able to run the whole shift. It should be able to find a bug, fix it, test it, and submit the work without the human touching it.
- The Expert: It needs to know the specific rules of the building (compliance, security laws) so it doesn't accidentally break the law while cooking.
5. The "Homework" Gap
The Claim: Most existing studies only looked at students or very specific tasks. This study looked at real, experienced professionals in big companies.
The Analogy: Previous studies were like testing a new car on a quiet, empty racetrack with test drivers. This study drove that car in rush-hour traffic with real commuters. They found that while the car is fast, the traffic (complex enterprise rules, security, different languages) makes it harder to use than the racetrack tests suggested.
Summary
The paper concludes that AI coding assistants are a powerful productivity booster, but they aren't magic. They are like a very fast, very knowledgeable, but occasionally hallucinating junior partner.
- Good: They save time on boring tasks and help people learn new languages.
- Bad: They can introduce security holes, and they don't always understand the specific context of a company's unique rules.
- Future: To be truly useful, they need to stop being just "autocomplete" and start becoming "autonomous architects" that understand the whole project, not just the line of code they are currently typing.
The authors emphasize that we need to stop treating these tools as a "set it and forget it" solution and start treating them as a collaborative partner that requires human oversight, customization, and trust-building.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.