How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study
Through a mixed-methods study of 100 practitioners, this paper reveals that building Software Engineering agents shifts development bottlenecks from coding to non-coding activities like requirements and coordination, fostering an evaluation-driven workflow characterized by a seven-stage process and challenges such as unreliable evaluation signals and comprehension debt.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine software development used to be like building a house with a hammer and a saw. You had to cut every single board, nail every single shingle, and sand every surface by hand. It was slow, tiring, and the "hard part" was just swinging the hammer.
Now, imagine someone hands you a magical, hyper-fast robot that can cut, nail, and sand a whole house in the blink of an eye. Suddenly, the hammer isn't the problem anymore. The problem is that the robot is so fast it builds a whole mansion before you've even finished drawing the blueprints.
This is exactly what happened when developers started using SE agents (AI robots that write code). A new study by researchers who talked to 20 builders from 12 different companies and surveyed 80 more found that while the robots made writing code cheap and fast, they didn't make the job disappear. Instead, the "bottleneck" (the traffic jam in the process) just moved to a different part of the road.
Here is what the researchers discovered, using the language of a curious explorer.
The New Seven-Step Dance
The paper suggests that building these AI agents isn't a straight line anymore; it's a looping dance with seven steps. It's less like a factory assembly line and more like a video game where you keep replaying levels to get a higher score.
- The Blueprint (Requirements): You tell the robot what to do. But now, you have to write the instructions so clearly that both humans and the robot can read them.
- The Scorecard (Evaluation): This is the most important new step. You don't just check the work at the end; you use a scorecard to steer the robot while it works.
- The Fuel (Data): You feed the robot examples of good work to learn from.
- The Build (System Construction): You pick a brain for the robot (a model) and build a "harness" (a suit of armor with tools and memory) around it.
- The Test Drive (Testing & Deployment): You let the robot run and see if it crashes.
- The Feedback Loop (Human Feedback): You watch what it does and tell it, "No, do it this way," or "Yes, that was great."
- The Tune-Up (Adaptive Maintenance): The robot's brain might get an update from its creator, changing how it thinks. You have to constantly tweak your harness to keep up.
The Great Shift: From Hammer-Wielder to Robot-Manager
The study found that because the robots can write code so fast, the old idea that "coding is the hard part" is ruled out. The researchers argue that coding was never the hardest part; it was just the loudest part.
Now that the robot does the heavy lifting, the real work has shifted to reviewing and evaluating.
- The "Vibe Coding" Effect: Because the robot can build things so fast, the lines between "researcher," "engineer," and "manager" are blurring. One person might now do the whole job, from dreaming up the idea to fixing the final bug.
- The "Black Box" Problem: The researchers suggest that because the robot is a "black box" (you can't see exactly how it thinks), you can't just trust it. You need a strict Evaluation-Driven Development style. This means you define the rules for success before you start, and you constantly check if the robot is actually getting better, not just faster.
The Six Traps (Challenges)
Even with super-fast robots, the builders hit six major snags. The paper suggests these are real, tough problems, not just minor glitches.
- The Broken Scorecard: How do you know the robot did a good job? The researchers found that the "tests" used to grade the robot are often broken. Sometimes the robot finds a better solution than the test expected, but the test says "Fail" because it's looking for the old answer. Other times, the test is just too expensive to run every time.
- The "Change Nothing, Change Everything" Curse: This is a spooky one. The researchers found that if the company that made the robot's brain updates it (even if you didn't touch your code), your robot might suddenly start behaving differently. A tool that worked yesterday might break today, even though you changed zero lines of code.
- Safety vs. Speed: Builders often admit they are scared of the robots but let them run anyway to get things done faster. The paper suggests this is dangerous. One team let a robot loose, and it accidentally deleted a user's home directory because it forgot its instructions.
- The "Unwritten Rules" Gap: Robots can only read what is written down. But in real life, a lot of knowledge is just "in people's heads" (like why a specific wall was built crooked). The researchers found that robots can't access this "unsaid" knowledge, leading to confusion.
- The Comprehension Debt: This is the biggest surprise. The robots are writing code faster than humans can understand it. It's like the robot is building a skyscraper in a day, but you are still trying to figure out the blueprint. The builders are accumulating a "debt" of code they don't understand. To fix this, some teams are starting to save the instructions to rebuild the code, rather than saving the code itself.
- The Fake Productivity: If you just count how many lines of code the robot writes, it looks like everyone is super productive. But the researchers suggest this is a trap. Writing 10,000 lines of code that no one understands or needs is not "productive." It's just "noise."
How Sure Are We?
The researchers are pretty confident in these findings because they didn't just guess; they measured them.
- They interviewed 20 experts and then surveyed 80 more.
- When they asked the survey group if they agreed with the findings, 91% agreed with the new workflow, and between 71% and 95% agreed with the specific challenges.
- They even went back to the original interviewees to check if they agreed with the summary (a process called "member checking"), and the experts said, "Yes, that's exactly what we do."
The Bottom Line
The paper suggests that building AI agents hasn't made software engineering easy; it has just made it different. The "hard part" has moved from writing the code to managing the robot, checking its work, and making sure it doesn't accidentally delete the internet.
The researchers conclude that as implementation becomes cheap, the bottlenecks don't disappear—they just move. The future of building software isn't about typing faster; it's about being a better manager, a stricter judge, and a smarter architect for the robots that do the heavy lifting.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.