AI-Conducted Interviews in Empirical Software Engineering: An Experience Report
This experience report demonstrates that AI-conducted, self-administered interviews are operationally viable and well-received by participants for short, low-risk empirical software engineering studies, though they currently serve best as a complementary tool to human-led interviews due to limitations in depth, sensitivity, and privacy.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the world of software engineering as a massive, bustling construction site where teams build digital skyscrapers. To understand how these builders work, researchers often stop by with clipboards and ask questions, a method called "interviews." It's like a detective chatting with a witness to figure out how a crime happened, but instead of crimes, they're investigating how programmers fix bugs, organize their days, or use new tools. Traditionally, this requires a human researcher to sit down, ask questions, and take notes, which is time-consuming and tricky when everyone is in different time zones or speaks different languages.
Recently, a new kind of "digital detective" has arrived: Artificial Intelligence (AI). Think of AI as a super-smart robot that can read millions of books and learn to talk like a human. Because these robots are getting so good at conversation, researchers started wondering: "Can we let the robot do the interviewing instead of a human?" This is the big question this paper tackles. It asks if a robot can successfully chat with software developers, ask them about their work, and gather useful stories without a human ever picking up a phone. The researchers aren't trying to prove the robot is better than a human, but rather if it's a possible tool that works well enough to be useful.
The Experiment: Letting the Robot Take the Mic
In this study, the researchers from Brazil decided to test-drive this idea. They created two custom "AI interviewers" (which they called MyGPTs) and sent them out to talk to software professionals. One robot was designed to ask about "refactoring"—a fancy word for cleaning up messy code to make it run better. The other robot asked about using AI tools within "Scrum," a popular way teams organize their work.
Instead of scheduling a video call, the researchers sent a link. The participants clicked it, opened the chat on their own phones or computers, and started talking. They could even use their voices to speak to the robot, just like talking to a smart assistant. The robot asked a series of questions, listened to the answers, and at the end, it automatically wrote a neat summary of the conversation. The participant then had to click a button to send that summary back to the researchers. It was a self-service interview: no human researcher was in the room, no scheduling conflicts, and the whole thing happened whenever the participant felt like it.
What They Found: The Good, The Weird, and The Glitchy
The researchers looked at 66 of these completed interviews. Here is what the data told them, broken down into the good news and the things that need fixing.
The Good News: People Liked It
Most of the people who finished the interview thought it was a pretty cool experience.
- Comfort: About 91% of them felt comfortable answering the questions.
- Clarity: Nearly 96% said the robot's questions were easy to understand.
- Speed: Over 97% felt the conversation moved at a good pace.
- Willingness: Almost 90% said they would be willing to do it again.
It turns out, many people actually liked the idea of talking to a robot. Some felt less pressure than they would with a human, and they appreciated being able to answer at their own speed without worrying about someone checking their watch.
The "Wait, What?" Moments
However, it wasn't perfect. When asked to compare the robot to a human interviewer, the results were a bit mixed. About 42% of people felt they were right in the middle—neither better nor worse. About 38% felt they preferred talking to a human, while only about 20% preferred the robot.
The biggest complaints were that the robot sometimes asked questions that felt too generic or didn't dig deep enough into the answers. About 29% of people felt the robot wasn't sensitive enough to their specific stories, and another 27% felt the conversation didn't go deep enough. Some people also missed the human connection, feeling a bit strange talking to a machine instead of a person.
The Technical Glitches
There were also some practical hiccups.
- The Wrong Summary: In about 9% of cases (6 out of 66), the participant sent back the wrong thing. Instead of the neat summary the robot generated, they pasted random parts of the chat or unstructured text.
- The Wrong Topic: In 2 cases, the participant answered questions about "Scrum" but sent back a summary about "refactoring," showing that the link between the question and the answer can get mixed up when a human is doing the work alone.
- Voice Issues: Since people used their voices, sometimes the robot misunderstood sounds. One person mentioned that a cough was accidentally recorded as an answer, confusing the flow of the interview.
The Big Takeaway: A Helpful Sidekick, Not a Replacement
The most important thing this paper tells us is what the AI interviewers can and cannot do.
The researchers are very clear: They did not prove that robots can replace human interviewers. They didn't check if the robot's summaries were as accurate, deep, or rich as a human's notes would be. They only checked if the process worked and if people were okay with it.
The conclusion is that AI interviews are a great complementary tool. They are perfect for quick, short, low-risk chats where you just need a general idea of what people think. They are fantastic for saving time on scheduling and letting people talk whenever they want. But, if you need to dig deep into complex emotions, handle very sensitive secrets, or need a super-deep understanding of a tricky problem, a human interviewer is still the best choice.
Think of the AI interviewer like a self-checkout kiosk at a grocery store. It's fast, convenient, and works great for a few items. But if you have a weird coupon, a heavy bag of potatoes, or a question about the produce, you still want a human cashier. This study shows that for software researchers, the "self-checkout" is ready to use, but they still need to keep a human in the loop to make sure the data is good.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.