Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications
This paper presents a four-year analysis of undergraduate programming submissions to introduce a three-dimensional taxonomy for student comments used as AI code-generation specifications, revealing that students predominantly write "What" comments, shift to "How" comments for procedural tasks, and prioritize verifying generated code over repeatedly rewriting specifications.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef, but instead of chopping vegetables and stirring pots, you are the "Commander" of a super-fast, hyper-creative robot sous-chef. Your job isn't to cook the meal yourself; it's to write the recipe instructions in plain English so the robot does the cooking. This is exactly what a group of researchers watched 1,161 university students do for four years in a computer class. They didn't write code line-by-line; they wrote comments (little notes) to tell an AI tool called GitHub Copilot how to build a program.
Here is what the researchers discovered about how these student-commanders talked to their robot helpers.
The Main Discovery: "What" vs. "How"
The biggest surprise? Most students didn't talk like programmers. They talked like people describing a goal.
- The "What" Crowd (77%): The vast majority of notes students wrote were about what they wanted the result to be. For example, instead of saying "Loop through this list and check the numbers," they wrote, "If the number is negative, return false." They described the destination, not the driving directions.
- The "How" Shift: However, when the task got tricky—like needing to repeat a step many times or do a complex sequence of actions—students switched gears. They started writing "How" comments, giving step-by-step marching orders. It's like when you tell a friend, "Make me a sandwich" (the what), but if you need a specific, complicated club sandwich, you suddenly have to say, "First toast the bread, then layer the turkey, then add the pickles..." (the how).
The "Don't Tweak the Recipe" Rule
You might think that if the robot makes a mistake, the student would frantically rewrite the instructions over and over. The paper argues against this.
The data shows that students rarely rewrote their comments. In fact, once they wrote a comment, they mostly left it alone.
- The Reality: Out of every 100 comments, about 48 were left completely untouched, and another 41 were brand new. Only a tiny sliver (about 4%) were heavily rewritten or completely scrapped.
- The Real Work: So, if they weren't rewriting the notes, what were they doing? They were acting like strict quality inspectors. The students spent their energy checking the robot's work, running tests, and fixing the code the robot produced, rather than constantly changing their own instructions. The paper suggests that the hard part wasn't writing the prompt; it was verifying the output.
What the Students Said (The Reflections)
After the tasks, the students wrote reflections, and the researchers found 13 main themes. Here are the big ones:
- Speed is King: Students loved the tool because it was fast. It felt like a "scaffold" that helped them get started quickly without getting stuck on boring details.
- The "Familiarity" Trap: The robot was great at simple, repetitive tasks (like boilerplate code), but it struggled when the task was weird, very specific, or required a unique solution.
- The Trust Issue: Even though the robot was fast, students knew they couldn't just trust it blindly. They felt a heavy "verification burden." They had to double-check everything because the robot sometimes gave answers that looked right but were actually wrong.
- Learning vs. Crutches: Some students worried that if they relied too much on the robot, they wouldn't learn to cook for themselves. They felt that using the tool as a "crutch" might hurt their own ability to solve problems later.
What the Paper Rules Out
It is important to know what this study did not find.
- It is NOT a "Magic Button": The paper explicitly rejects the idea that students just asked the AI for a full solution and walked away. That didn't work well. The students who succeeded were the ones who treated the AI as a partner they had to guide and check.
- It is NOT about "Perfect" Code: The study did not find that the AI always produced perfect code. In fact, the students noted that the AI often added unwanted code or missed tiny technical details.
- It is NOT a "Solved" Problem: The paper does not claim that teaching students to prompt AI is easy or that everyone is now an expert. It suggests that this is a new skill set that requires practice, especially in knowing when to be specific and when to check the work.
How Sure Are We?
The researchers are very confident in these numbers because they looked at a massive dataset: 10,257 submission attempts from four years of classes. They used automated tools to count over 136,000 comments and analyzed student reflections with high accuracy.
- They found that 99.3% of the students eventually got their code to pass the tests.
- They measured the "tweaking" of comments with mathematical precision, finding that heavy rewrites were rare.
- However, the paper admits some limits: This was just one university course using Java. We don't know if these exact same patterns happen in a different language or a different type of class. Also, the study only looked at the comments students submitted, not the ones they might have typed and deleted in the middle of working.
The Takeaway
The paper suggests that in the age of AI, the most important skill isn't just typing code; it's specifying what you want and verifying that you got it. Students learned that writing a clear "What" is the first step, but the real magic happens when you act as the boss, checking the robot's work and only stepping in to give "How" instructions when the robot gets lost. It's not about letting the AI do the thinking for you; it's about learning how to talk to it so it does the thinking with you.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.