Developers' Experience with Generative AI Beyond Productivity Assessment -- Insights from an Empirical Mixed-Methods Field Study
This mixed-methods field study reveals that while professional developers generally find Generative AI tools beneficial for repetitive tasks and report productivity gains, the effectiveness of specific interaction modes (in-code suggestions vs. chat) varies by task complexity, with combining them often diminishing benefits and introducing cognitive load during development-heavy work.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine software developers as master chefs in a busy kitchen. For years, they've been chopping, sautéing, and plating dishes entirely by hand. Recently, a new "smart sous-chef" (Generative AI) has been introduced to help them. This paper isn't just about how much faster the kitchen is working; it's about how the chefs feel working with this new assistant, what kind of help they actually want, and whether the assistant sometimes gets in the way.
Here is a breakdown of the study's findings in plain English:
The Setup: A Four-Day "Kitchen" Experiment
Researchers invited 22 professional software developers (the chefs) from SAP to spend four days working with a specific AI tool called GitHub Copilot.
- The "Controlled" Days: Two days were like a cooking competition. The chefs were given specific, pre-set recipes (coding tasks) to solve in a lab-like setting while researchers watched their every move, measured their stress, and recorded their screen activity.
- The "Real Life" Days: Two days were just normal work. The chefs went back to their usual kitchen, doing whatever they normally do, but they kept a diary of what they were doing and how helpful the AI was.
What the Chefs Thought Before the Experiment
Before the study, the chefs generally liked the idea of the smart sous-chef.
- The Good: They felt the AI was a great helper for boring, repetitive tasks like chopping vegetables (writing boilerplate code) or cleaning up the station (writing unit tests). It made them feel more productive and less tired.
- The Bad: They weren't sure if the AI made the work more fulfilling. In fact, some felt that because the AI was doing the "fun" creative parts, they felt less satisfied with their job.
- The Fear: They worried the AI might make mistakes (like serving a burnt dish) or that they might forget how to cook from scratch if they relied on it too much.
The Big Discovery: Don't Switch Tools Mid-Task
The most surprising finding was about how the chefs used the AI. The AI offered two main ways to interact:
- The "Autocomplete" Mode: The AI whispers suggestions right next to the code as you type (like a sous-chef handing you the next ingredient before you ask).
- The "Chat" Mode: You stop typing and have a conversation with the AI in a separate window to ask for a whole recipe or explain a complex problem.
The Rule of Thumb:
- If you are doing a simple, repetitive task (like writing a standard function), use the Autocomplete mode. It's fast and keeps you in the flow.
- If you are doing a complex task (like debugging a broken machine or brainstorming a new menu), use the Chat mode. It gives you the big picture and explanations.
The Trap: The study found that if a chef tried to use both modes for the same task—switching back and forth between whispering suggestions and having a chat conversation—they actually got slower and felt more stressed than if they hadn't used the AI at all. It was like trying to listen to a radio and talk to a friend at the same time while cooking; you just get confused and drop the spoon.
The "Mental Load" Surprise
You might think that having a helper would make a chef's brain feel lighter. Surprisingly, the study found that when developers used AI on their hardest tasks, they actually felt more mentally taxed (higher cognitive load), even though they finished the work at the same speed.
Why? Because using the AI adds a new layer of work: you have to read what the AI wrote, check if it's right, and fix it if it's wrong. It's like having a sous-chef who hands you a pre-chopped onion, but you still have to inspect it to make sure it's not rotten. If the AI's "help" is good, the chef feels productive. If the help is bad, the chef just feels like they are doing extra work.
The Future of the Kitchen
The chefs are open to using this new technology, but they have a clear vision of their future roles:
- The AI will take over the boring, repetitive, and time-consuming chores (chopping, cleaning, basic prep).
- The Human Chef will focus on the high-level stuff: designing the menu, understanding the customer's taste, and making sure the final dish is safe and delicious.
However, they are worried that if they rely too much on the AI, they might lose their own "muscle memory" for cooking. They also noted that while they ask the AI for quick tips faster than they ask their human colleagues, they still trust their human teammates more for solving really hard problems.
The Takeaway
The study concludes that AI is a powerful tool, but it's not a magic wand. To get the most out of it, developers need to be intentional:
- Pick the right "mode" for the job (Autocomplete for simple things, Chat for complex things).
- Don't switch back and forth between modes, or you'll just get tired.
- Remember that the AI is a helper, not a replacement for human judgment.
By treating the AI like a specific tool rather than a constant companion, developers can keep their stress down and their productivity up.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.