💬 NLP

(Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding

This study reveals that while coding agents significantly boost developer productivity by accelerating task completion, they simultaneously impair users' code comprehension and ability to extend their work, particularly when interactions involve low-effort prompting, highlighting a critical trade-off between efficiency and understanding that developers must address.

Original authors: Nishant Balepur, Connor Baumler, Valerie Chen, Eunsol Choi, Rachel Rudinger, Jordan Lee Boyd-Graber

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nishant Balepur, Connor Baumler, Valerie Chen, Eunsol Choi, Rachel Rudinger, Jordan Lee Boyd-Graber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are learning to cook. For years, the only way to get good at it was to chop the vegetables, season the sauce, and stir the pot yourself. You might make a few mistakes, but by the time you served the meal, you knew exactly why it tasted the way it did. Now, imagine a magical sous-chef robot that can chop, season, and stir instantly. You just tell it, "Make me a lasagna," and it does. You get your dinner faster than ever before. But here is the catch: because you didn't do the chopping or the stirring, you have no idea how the layers of cheese and sauce fit together. If the robot breaks down tomorrow, or if you need to tweak the recipe for a friend who hates tomatoes, you might be stuck. You have the meal, but you've lost the recipe. This is the heart of a new study in the world of computer science, specifically looking at how "coding agents"—smart AI tools that write software for us—are changing the way people learn to build things. The big question isn't just "Can the AI do the job?" but "Does the human still understand the job they just finished?"

This paper, titled "(Im)Paired Programming," dives into that exact dilemma. The researchers wanted to see if using these AI coding assistants helps people get work done faster, but at the cost of actually understanding the code they are building. They set up a game for 54 computer science students: build a website for a unique version of tic-tac-toe called "Zic-Zac-Zoe." The students were split into two groups. One group used a "Chatbot" that gave them hints and snippets of code, forcing them to type the code themselves. The other group used a powerful "Agent" that listened to their instructions and actually edited the code files for them, doing the heavy lifting.

The results were a bit of a shocker, like finding out the magic robot chef is great at feeding you, but terrible at teaching you how to cook. The students using the AI Agent finished the initial website task much faster and with fewer errors than the students who wrote the code themselves. If you only looked at who finished first, the AI Agent was the clear winner. However, when the researchers tested how well the students actually understood what they had built, the story flipped completely. The students who let the AI do the work scored significantly lower on questions about how their code worked. They couldn't explain why certain parts of the website looked the way they did, or what would happen if they changed a specific line. It was as if they had been handed a finished puzzle but didn't know how the pieces fit together.

Even more interesting, when the students were asked to add a new feature to their website—like a "Reset" button—without any help from the AI, the advantage of the AI group disappeared. The students who had written their own code from the start were just as good at adding the new feature as the ones who had used the AI. The AI had given them a head start, but it hadn't given them the skills to keep going once the robot stopped helping. The study suggests that while these agents are amazing at getting tasks done quickly, they might be creating a "productivity trap" where users feel productive but are actually losing their ability to think critically about their own creations.

The researchers also looked at how the students used the tools. They found that students who took the "lazy" route—copying and pasting instructions without thinking, or just clicking "Accept All" on every change the AI made—understood the least. Those who took the time to read the code, ask for explanations, or manually type out the syntax did better, though still not as well as the group that wrote everything from scratch. Surprisingly, even though the students knew they understood less when using the AI, they still preferred it. They found it easier, faster, and less mentally exhausting. It's a bit like preferring a GPS that drives the car for you, even if you know you're forgetting how to read a map.

Ultimately, the paper argues that we can't just measure success by how fast a task gets done. If we want people to be able to fix, explain, and improve their code later, we need to design AI tools that force them to stay engaged. The authors suggest that future AI shouldn't just be a "do-it-for-me" button, but a partner that encourages users to think, read, and understand the code it generates. Until then, we might be getting a lot of finished websites, but we risk losing the people who truly know how they work.

Technical Summary: (Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding

Problem Statement

While coding agents (e.g., Cursor, Claude Code) are increasingly evaluated based on task completion metrics, there is a critical gap in understanding how these tools affect a developer's comprehension of the code they produce. Current research often equates agent success with the ability to finish tasks quickly, potentially overlooking the shift in user workflow from writing code to prompting and reviewing. This shift risks impairing user oversight, learning, and communication, particularly when AI is unavailable or when users must extend or debug their own work. The paper investigates whether users who rely on agents optimized for task completion retain the ability to understand and modify the code they generate.

Methodology

The authors conducted a between-subjects user study with 54 computer science students (17 B.S., 31 M.S., 11 Ph.D.) to compare two AI interaction modes during a web development task.

Experimental Design

Task: Participants were tasked with creating a web-based variant of tic-tac-toe ("Zic-Zac-Zoe") involving HTML structure, CSS styling, and JavaScript logic. The task required implementing specific game rules (e.g., alternating turns, win conditions) within a 50-minute limit.
Conditions:
1. Agent Group ( $n=27$ ): Users interacted with an AI agent (based on AIDER) that directly edited their code files in response to prompts. Users could accept or reject diffs but did not write the core logic themselves.
2. Chatbot Group ( $n=27$ ): Users interacted with a chatbot that provided generic syntax snippets or high-level guidance but did not edit the code. Users were required to write the code from scratch or manually adapt the snippets.
Metrics:
1. Productivity: Measured by initial task accuracy (based on a 15-rule rubric) and time-to-submission.
2. Comprehension: Assessed via 12 tailored multiple-choice questions (6 recall, 6 reasoning) generated by an LLM based on the specific code each user produced. Questions tested knowledge of code elements and the ability to reason about code behavior.
3. Extension Ability: A follow-up task where users had to extend their initial code to pass new test cases (e.g., adding a reset button, changing win conditions) without the assistance of the agent (using only the chatbot).
4. Background Control: A pre-study quiz on HTML/CSS/JS skills controlled for prior ability.

Data Analysis

The study utilized linear regression to analyze the relationship between background ability, AI group, and outcomes. Qualitative coding was applied to user prompts and interaction logs to identify strategies (e.g., copy-paste vs. syntax adaptation) and review behaviors (e.g., auto-accept vs. manual review).

Key Results

1. Productivity vs. Understanding Trade-off

Initial Task: Agent users achieved significantly higher accuracy and faster completion times compared to chatbot users ( $p=0.00$ , Cohen's $d=1.4$ ).
Comprehension: Despite higher productivity, agent users scored significantly lower on comprehension questions ( $p=0.00$ , Cohen's $d=0.9$ ). They struggled to recall code elements and reason about code behavior compared to those who wrote the code themselves.
Extension Task: When extending code without agents, the initial productivity advantage of the agent group disappeared. While agent users performed slightly better initially, this difference vanished when controlling for initial task accuracy ( $p=0.18$ ). The regression analysis revealed that agents provide higher-quality code scaffolds but sacrifice user comprehension, which ultimately hinders the ability to extend the code when the agent is removed.

2. The Role of Background Skill

Masking Effect: In the chatbot group, higher background ability correlated with higher initial task accuracy. In the agent group, this correlation disappeared; agents "equalized" performance, masking differences in user skill levels.
Comprehension Driver: Background skill remained a strong predictor of comprehension across both groups, suggesting that traditional coding knowledge is still vital for understanding, even when using agents.

3. Interaction Patterns and Code Attributes

Prompting Strategies: Users who employed "lazy" strategies (e.g., directly copying task criteria) had the lowest comprehension. Users who added specific code syntax to their prompts had the highest comprehension.
Review Behaviors: Users who auto-accepted agent changes had lower comprehension than those who reviewed changes file-by-file. However, even the most engaged agent users did not reach the comprehension levels of the chatbot group.
Code Readability: Agent users with fewer lines of code, fewer comments, and lower token volume (more concise code) demonstrated better comprehension. This suggests that concise, less "noisy" code is easier for users to digest.

4. User Perception

Users self-reported that agents were more helpful and required less mental effort than chatbots.
Crucially, users recognized that their understanding was weaker with agents but still preferred using them for new tasks due to speed and ease of use.
Many users felt a lack of "ownership" over the code generated by agents.

Key Contributions

Empirical Evidence: The study provides quantitative evidence that coding agents optimized for task completion improve initial productivity but significantly harm code comprehension and the ability to extend code without AI assistance.
Methodological Framework: It introduces a robust framework for measuring code understanding in AI-assisted workflows, combining tailored comprehension questions, extension tasks, and interaction analysis.
Interaction Insights: It identifies specific user behaviors (lazy prompting, auto-accepting) and code attributes (readability, conciseness) that correlate with lower understanding.
Dataset Release: The authors released a dataset containing 54 user websites, prompts, AI traces, background ability scores, and comprehension data.

Significance and Claims

The paper argues that the current trajectory of coding agent development, which prioritizes task completion metrics, is insufficient for real-world software engineering where understanding, maintenance, and extension are critical.

The "Im" in (Im)Paired: The authors claim that while agents keep users "in the loop" for the workflow, they effectively remove users from the loop of understanding.
Design Implications: The study suggests that future agent design must move beyond simple task completion. Proposed directions include:
- Dissuading low-effort prompting: Agents should refuse or nudge users away from copy-paste strategies.
- Generating readable code: Optimizing for code readability (conciseness, clarity) rather than just functional correctness.
- Promoting active engagement: Designing review mechanisms that force users to actively engage with code changes rather than passively accepting them.
Limitations: The authors modestly note that the study is limited to CS students and web development tasks. They acknowledge that for short-term, low-stakes projects (e.g., "vibe coding"), productivity might be the sole desired metric, but for learning and professional workflows, understanding cannot be neglected.

The paper concludes that for coding agents to be truly beneficial in the long term, they must be designed to jointly optimize for both productivity and user understanding.