A survey of generative AI adoption and perceived productivity among scientists who program
A survey of 868 scientific programmers reveals that while generative AI adoption is highest among less experienced users who prefer general-purpose tools, perceived productivity is driven more by the volume of accepted generated code and lack of formal development practices than by actual validation, highlighting a potential risk of over-reliance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a massive library where scientists are trying to build complex machines (their research) using a very specific, difficult language: code. For years, many of these scientists have been told, "You need to learn to build these machines perfectly," but they were never given a proper workshop or a master builder to teach them. They are often learning by trial and error, which is slow and frustrating.
Now, a new kind of "magic assistant" has arrived: Generative AI. It's like a super-fast robot that can write the blueprints for these machines instantly.
This paper is a report card on how scientists are using this robot, how much they think it's helping them, and whether they are actually building better machines or just building faster ones that might fall apart.
Here is the breakdown of their findings, using some everyday analogies:
1. Who is using the Magic Robot?
The New Kids on the Block.
The study found that the people using this AI the most are the students and the less experienced programmers. Think of it like a new video game controller. The people who have been playing for 20 years (the senior scientists) are a bit skeptical or prefer their old, reliable controllers. But the new players? They love the new gadget immediately.
- The Tool Choice: Most scientists aren't using the "Pro" tools designed specifically for building machines (like GitHub Copilot). Instead, they are using the "General Chat" tools (like ChatGPT).
- Analogy: It's like trying to fix a car engine by talking to a very smart, friendly librarian in a chat window, rather than using a specialized mechanic's diagnostic computer. They prefer the chat because it feels more natural, even if it's not the most precise tool for the job.
2. The "Productivity" Illusion
The Speed vs. Quality Trap.
The researchers asked: "Do you feel more productive?"
The answer was a resounding "Yes!" from the beginners and those who don't follow strict building rules.
The Catch: The study found a strange pattern. The scientists who felt most productive were the ones who:
- Had the least experience.
- Didn't use "safety nets" like code testing or peer reviews.
- Accepted the biggest chunks of code at once.
The Analogy: Imagine you are baking a cake.
- The Old Way: You write the recipe, check the ingredients, ask a friend to taste it, and then bake it. It takes a long time.
- The AI Way: You ask the robot for a recipe. It spits out a 50-page instruction manual.
- The "Productive" Scientist: The scientist who says, "Wow, I got a 50-page manual in 2 seconds! I am so productive!" and immediately starts baking without reading it.
- The Reality: They might have just accepted a recipe that says "add 500 eggs." They feel fast, but the cake might be a disaster. The study suggests scientists are measuring their success by how much code they generated, not by how well the code works.
3. The Safety Nets (Or Lack Thereof)
The "No-Net" Trapeze Act.
In professional software building, you have safety nets:
- Version Control: A "Save Point" so you can go back if you mess up.
- Code Review: A friend checking your work.
- Testing: Running a simulation to see if the machine breaks.
The study found that many scientists don't use these nets. They are doing trapeze without a safety harness.
- The Interaction: When scientists do use these safety nets, the "productivity boost" from AI goes down. Why? Because they are actually checking the work! They are slowing down to make sure the robot didn't make a mistake.
- The Danger: The scientists who feel the most productive are often the ones flying without a net, accepting the robot's output blindly.
4. Why Do Some People Say "No" to the Robot?
About 25% of the scientists refused to use the AI. Why?
- "I want to learn": They felt that using the robot would make them lazy and they wouldn't learn how to build the machines themselves. (Like a student refusing a calculator so they learn math).
- "It's unethical": They worried about the environmental cost of the AI or that the companies making it were stealing data.
- "It's not worth it": For small, simple tasks, the robot was actually slower because it made mistakes that took longer to fix than just doing it yourself.
- "It lies too much": The robot sometimes made up facts or code that didn't work (hallucinations), which was annoying and dangerous.
The Big Takeaway
The paper warns us: Just because the robot writes code fast, doesn't mean the science is good.
We are currently in a phase where scientists feel incredibly productive because they are generating more code. But if they don't start checking that code (testing, reviewing, understanding it), they might be building a house of cards. The "magic assistant" is great, but if you don't know how to drive the car, you might just crash it faster.
In short: The beginners are loving the speed, but the experts are worried that without a safety net, we might be generating a lot of beautiful, fast, but broken science.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.