💻 computer science

Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants

This mixed-method study at BNY Mellon, involving 2,989 survey responses and 11 interviews, argues that evaluating AI coding assistants requires a holistic, multifaceted approach that incorporates long-term human-centered factors like technical expertise and work ownership, rather than relying solely on traditional short-term productivity metrics.

Original authors: Valerie Chen, Jasmyn He, Behnjamin Williams, Jason Valentino, Ameet Talwalkar

Published 2026-02-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Valerie Chen, Jasmyn He, Behnjamin Williams, Jason Valentino, Ameet Talwalkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a massive construction company. For years, you've measured your builders' productivity by counting how many bricks they lay per hour or how many walls they finish in a day. But recently, you've handed every builder a magical assistant robot that can instantly suggest the next brick, write the blueprints, and even fix mistakes.

Now, you're confused. The robots are popular, but are they actually making the company faster? And if they are, does "faster" mean the same thing it used to?

This paper is like a deep-dive investigation by a team of researchers (from Carnegie Mellon University and BNY Mellon) who went to talk to nearly 3,000 of these "builders" (software developers) to figure out how to measure success in this new age of AI robots.

Here is the story of what they found, broken down into simple parts:

1. The Big Confusion: Happy vs. Fast

The researchers first asked a simple question: "Are you happy with your robot assistant?" and "How much time does it save you?"

The Result: Most developers said, "Yes, I love the robot! It makes my day easier." (86% were satisfied).
The Twist: But when asked, "How much time did you save?" most said, "Not much. Maybe 30 minutes a week."

The Analogy: Imagine you have a super-fast car that gets you to work in 5 minutes, but you spend 45 minutes stuck in traffic. You might love the car because it's fun and reliable, but you aren't actually getting to work any faster than before. The study found that developers love the feeling of using the AI, but it doesn't always translate to huge chunks of saved time. This proves you can't just use one number (like "time saved") to judge if the tool is working.

2. The Six New Rules of the Game

Since the old way of counting (bricks per hour) doesn't work anymore, the researchers interviewed 11 developers to find new ways to measure success. They found six distinct factors that matter, which they grouped into three stages of a project:

Stage A: While Building (The "In the Moment" Feel)

Self-Sufficiency (The "Do-It-Yourself" Superpower):
- Before: If a builder didn't know how to fix a leaky pipe, they had to stop, call a senior expert, or search a giant library of manuals.
- Now: The robot whispers the answer right in their ear. They feel like a superhero who can solve problems without leaving their desk.
Frustration & Brain Load (The "Mental Tug-of-War"):
- The Catch: The robot isn't perfect. Sometimes it suggests a solution that looks right but is actually wrong. The developer has to stop, think hard, and double-check everything. This can actually make them more tired and frustrated, even if they are typing faster.

Stage B: Handing Over the Keys (The Team Check)

Task Completion Speed (The "Throughput" Check):
- This is the old-school metric: How fast did we finish the job? The study found that while AI helps, it doesn't always mean the job is done faster. Sometimes it just means the job is done with less effort, but the time saved is small.
Peer Review (The "Safety Inspection"):
- Before, a senior builder would check a junior's work. Now, if the junior uses the robot, the senior has to ask: "Did you write this, or did the robot?" If the robot wrote it, the senior has to spend extra time understanding the code to make sure it's safe. Sometimes, the robot makes the code look "too perfect" or confusing, making the safety check harder.

Stage C: The Long Haul (The Future of the Builder)

Technical Expertise (The "Learning Curve"):
- The Risk: If a junior builder relies on the robot to do all the thinking, they might never learn how to fix a leaky pipe themselves. They might become great at pressing buttons but terrible at understanding the plumbing. The study warns that if we aren't careful, we might create a generation of developers who can't work without the robot.
Ownership (The "Pride of Creation"):
- The Feeling: Developers love to say, "I built this." If the robot wrote 90% of the code, do they still feel proud? Do they feel responsible if it breaks? The study found that developers worry that if they didn't write the code themselves, they won't feel a deep connection to it, and they might be slower to fix it when it breaks later.

3. It Depends on What You Are Doing

The researchers also found that the AI helps differently depending on the task:

Building something new: The robot is great at giving a head start, but you have to be careful not to just copy-paste blindly.
Fixing old code: The robot struggles here because it needs a lot of context. It's like trying to fix a 50-year-old house with a robot that only knows how to build new houses.
Writing manuals or tests: This is where the robot shines. It's like having a robot that can instantly write the instruction manual for the house you just built. This saves the most time.

The Bottom Line

The paper concludes that we need to stop looking for a single "magic number" to measure productivity.

The Analogy: Imagine trying to judge a chef's skill just by counting how many plates they serve. If they use a robot to chop vegetables, they might serve more plates, but if the robot makes the food taste bad or the chef forgets how to cook, the restaurant fails in the long run.

To truly understand if AI coding assistants are helping, we need to look at the whole picture:

Are developers happy?
Are they learning, or just copying?
Do they feel responsible for the code?
Is the team checking the work effectively?

The authors say we need a "holistic" view—a balanced scorecard that values the human experience and long-term growth, not just the speed of the output.

Technical Summary: Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants

Problem Statement
The integration of AI coding assistants (e.g., GitHub Copilot, Cursor) into software development workflows necessitates a re-evaluation of how developer productivity is measured. While traditional frameworks like SPACE and DORA offer established metrics for software engineering, they were designed for deterministic tools and workflows. AI assistants, by contrast, generate non-deterministic outputs and support a broader, evolving scope of tasks. Existing literature presents conflicting perspectives: some studies rely on self-reported satisfaction and flow, while others focus on objective metrics like commit frequency or lines of code. This divergence suggests that current metrics may be insufficient to capture the multifaceted impact of AI on developer productivity, particularly regarding long-term career trajectories and code ownership.

Methodology
The authors employed a sequential exploratory mixed-methods research design conducted at BNY Mellon, involving two distinct phases:

Large-Scale Survey: A quantitative survey was distributed to approximately 8,000 engineers, yielding 2,989 responses from developers with access to GitHub Copilot. The survey utilized the Developer Experience (DX) framework to measure two specific metrics:
- General Satisfaction: A consumer satisfaction score (ranging from "very dissatisfied" to "very satisfied").
- Perceived Time Savings: Self-reported weekly time saved (ranging from "no savings" to "2+ hours").
- The study analyzed the correlation between these two metrics to determine if they align.
Semi-Structured Interviews: Following the survey, 11 in-depth interviews were conducted with a purposively sampled group of developers. Participants varied by seniority (early career, mid-career, management), role (backend, frontend, full-stack), and department (Customer Products, Platform Engineering, Data). The interviews explored how developers define productivity, their use cases for AI tools, and the nuanced factors influencing their experience across development, deployment, and long-term maintenance.

Key Findings and Results

Misalignment of Metrics (RQ1): The survey results revealed a weak positive correlation ( $r=0.34$ ) between developer satisfaction and perceived time savings. While 86% of respondents were satisfied or very satisfied with GitHub Copilot, approximately 60% reported saving less than one hour per week. Notably, nearly 400 developers reported being "very satisfied" despite saving only 30 minutes weekly, while over 100 developers saving 2+ hours weekly reported being neutral or dissatisfied. This indicates that high satisfaction does not necessarily equate to significant time savings, and single metrics fail to capture the full picture.
Six Distinct Productivity Factors (RQ2): Through thematic analysis of the interviews, the authors identified six factors that developers consider relevant when evaluating AI productivity. These are categorized into three dimensions:
1. Development Process:
  - Self-sufficiency: The extent to which AI reduces the need for context-switching to external resources (e.g., Stack Overflow, documentation) or colleagues.
  - Frustration and Cognitive Load: The mental effort required to verify AI outputs, handle hallucinations, and refine prompts, which can offset efficiency gains.
2. Deployment Process:
  - Task Completion Rate: The speed of completing well-defined tasks, though participants noted that "lines of code" is an insufficient proxy for value or criticality.
  - Ease of Peer Review: The impact of AI-generated code on the code review process, including concerns about junior developers optimizing for the wrong metrics or reviewers struggling to trust AI-assisted code.
3. Long-Term Developer Impact:
  - Technical Expertise: Concerns that early or excessive reliance on AI may hinder the development of fundamental debugging and architectural skills, particularly for junior engineers.
  - Ownership of Work: The psychological sense of authorship and accountability; developers expressed that deep understanding of code is crucial for maintenance and incident resolution, which may be compromised if code is not written "by hand."
Context-Dependent Impact (RQ3): The study found that the impact of these factors varies significantly by use case:
- Implementing New Features: AI often accelerates task completion and reduces cognitive load but raises concerns about over-reliance and the erosion of technical expertise.
- Improving Existing Code (Refactoring): AI is less effective here due to the high context requirements; the bulk of the work involves understanding legacy code rather than writing new lines, limiting the tool's utility for task acceleration.
- Generating Tests and Documentation: AI is highly effective for these "boilerplate" tasks, significantly reducing time and frustration, though developers emphasize the need to manually verify test scenarios to ensure expertise is maintained.

Significance and Contributions
The paper argues that measuring AI productivity requires a shift from single-metric evaluations to a holistic, multi-dimensional framework. The authors' primary contribution is the identification of six factors that extend beyond immediate efficiency gains to include long-term considerations such as technical expertise and ownership of work, which are largely absent in prior frameworks like SPACE and DORA.

The study highlights that while AI tools may not always yield dramatic time savings, they significantly alter the developer experience by changing how work is approached, reviewed, and maintained. The authors propose that future research and industry evaluations must incorporate these human-centered, long-term factors to avoid misleading conclusions about AI's value. They provide a set of operationalized questions (Table 2) to help researchers and organizations better evaluate these dimensions in empirical studies.

Limitations
The authors note that the study is limited to a single organization (BNY Mellon) and primarily focuses on GitHub Copilot. While the participant pool was diverse in terms of role and seniority, the findings may not fully generalize to all organizations or independent developers using different AI tools. Additionally, the reliance on self-reported time savings in the survey is acknowledged as a potential limitation compared to objective telemetry.