The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study
This longitudinal mixed-methods study reveals that while AI coding assistants are perceived to improve productivity and reduce time spent on coding, they simultaneously shift engineers' focus from creation to verification, introduce a new category of "supervisory engineering work," and create a paradox where stable productivity gains coincide with a significant decline in developer experience, flow state, and cognitive well-being.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine software engineering as a chef cooking a massive, complex banquet. For years, the chef's job was to chop every vegetable, measure every spice, and stir every pot from scratch.
Now, imagine a magical sous-chef (the AI coding assistant) has been hired. This sous-chef can chop vegetables and mix ingredients incredibly fast. But what happens to the head chef's day? Does the chef just relax and enjoy the view?
According to this study, which tracked professional software engineers over six months, the answer is more complicated. The work isn't just getting faster; the nature of the work is changing, and the chef's experience of the kitchen is shifting in surprising ways.
Here is what the study found, broken down into simple concepts:
1. The Shift: From "Cooking" to "Tasting"
The Finding: Engineers are spending significantly less time actually writing code (the "cooking"). However, they aren't spending that saved time doing nothing. Instead, their focus is shifting toward checking the work.
- The Metaphor: The sous-chef (AI) is chopping the onions in seconds. But the head chef now spends more time tasting the soup to make sure the sous-chef didn't add salt instead of sugar, or checking if the vegetables are cut the right way.
- The Result: The study calls this a shift from Creation (making new things) to Verification (checking what was made). Interestingly, the time saved on writing code wasn't fully replaced by time spent checking; the total time on these tasks just went down.
2. The New Job Title: "The Supervisor"
The Finding: The study identified a new type of work that didn't really exist before. It's not just "writing" or "testing." It's a mix of directing, evaluating, and fixing the AI's mistakes.
- The Metaphor: The head chef is no longer just a cook; they are now a Supervisor. Their job is to tell the sous-chef what to chop, watch the chopping happen, and then step in immediately if the sous-chef starts chopping the wrong vegetable.
- The Term: The researchers call this "Supervisory Engineering Work." It involves guiding the AI, deciding if its output is good, and fixing it when it's "almost right but not quite."
3. The Paradox: "I'm Faster, But I'm Tired"
The Finding: This is the most surprising part of the study.
- Productivity: 84% of engineers felt they were getting more productive. They were finishing tasks faster.
- Experience: However, the feeling of doing the work got worse. Over six months, the number of engineers who felt their work experience had gotten worse nearly doubled (from 14% to 27%).
- The Metaphor: Imagine driving a car that goes 200 mph (Productivity is up!). But, the steering wheel is loose, the road is full of potholes, and you have to constantly swerve to avoid hitting things (Experience is down). You are getting to the destination faster, but the ride is more stressful and less enjoyable.
- Specifics: The study found that engineers felt less "in the zone" (a state called Flow) and had higher Cognitive Load (mental stress). The constant cycle of asking the AI for help, checking its answer, and fixing it broke their concentration.
4. The Trust Game
The Finding: Engineers are learning to be skeptical. They don't blindly trust the AI.
- The Metaphor: At first, everyone was excited to let the sous-chef cook the whole meal. Six months later, the chefs have learned that the sous-chef sometimes hallucinates (makes up ingredients that don't exist) or uses the wrong spice. Now, every single dish the sous-chef prepares is tasted and inspected by the head chef before it goes to the customer.
- The Shift: The study noted that as engineers got more comfortable with the tools, they became more concerned about the long-term quality and safety of the code, not less.
5. The Changing Toolkit
The Finding: The tools themselves are changing fast.
- The Metaphor: It's like the kitchen equipment is being upgraded every few weeks. In the six months of the study, most engineers switched which "sous-chefs" they were using or started using a combination of different ones. They aren't sticking to just one tool; they are building a toolkit.
Summary
The paper concludes that AI isn't just a "speed button" that makes software engineers work faster in the same way they always have. Instead, it is reorganizing the job.
- Old Job: Create code, test code.
- New Job: Direct the AI, verify its output, fix its errors, and constantly manage the trust relationship.
While engineers feel they are getting more done (Productivity), the mental effort required to manage this new relationship is making the work feel more fragmented and stressful (Experience). The study suggests that the "magic" of AI is real, but it comes with a hidden cost: the constant need to supervise and correct, which changes the rhythm and feel of the work.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.