Prompt Quality and Pull Request Outcomes: A Stage-Based Empirical Study of LLM-Assisted Development
This study empirically analyzes 265 developer-ChatGPT interactions in open-source pull requests to demonstrate that specific prompt characteristics—Context, Specificity, and Verification—exert distinct, stage-dependent effects on AI-assisted software development outcomes, where Specificity and Context drive actionable code generation, Verification predicts code adoption, and Context determines integration depth.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef (the developer) trying to cook a complex dish, but you have a very talented, yet sometimes literal-minded, sous-chef (the AI). You don't just shout "Make dinner!" and expect a Michelin-star meal. You have to give specific instructions.
This paper is a study of how the way you ask the sous-chef for help affects the entire cooking process, from the first ingredient to the final plate being served to the customer.
The researchers looked at 265 real-world examples where developers shared their conversations with an AI (ChatGPT) inside software projects. They wanted to see if the quality of the "recipe request" (the prompt) determined whether the AI's code was useful, accepted, and actually used in the final product.
Here is the breakdown of their findings, using simple analogies:
1. The Three Ingredients of a Good Request
The researchers broke down every request into three parts, like checking a recipe card:
- Context (The Setting): Did the developer explain where this code fits? (e.g., "This is for the login screen of our banking app.")
- Specificity (The Details): Did the developer say exactly what they wanted? (e.g., "Make the button blue and round," not just "Make it look nice.")
- Verification (The Taste Test): Did the developer say how to check if it's right? (e.g., "If I click this, it should log me in without an error.")
2. The Three Stages of the Kitchen Workflow
The study found that the "perfect" request changes depending on which stage of the cooking process you are in. It's not one-size-fits-all.
Stage 1: Getting the First Draft (Code Generation)
- What matters most: Context and Specificity.
- The Analogy: If you want the AI to actually write a recipe, you need to tell it what kind of food you are making and give it clear steps.
- The Finding: If the request was vague or lacked background info, the AI often just gave a generic explanation instead of actual code. But if the developer gave a clear goal and some background, the AI almost always produced usable code.
- What didn't matter yet: Whether the request included a "taste test" (Verification) didn't stop the AI from writing the code.
Stage 2: Deciding to Use the Draft (Code Adoption)
- What matters most: Verification.
- The Analogy: Now the AI has written a recipe. The developer has to decide: "Can I trust this?" If the developer included a way to check the result (like "It must pass this specific test"), the developer was much more likely to say, "Yes, use this!"
- The Finding: Even if the code was well-written, if the developer didn't provide a way to verify it was correct, they often rejected it. The "taste test" was the key to getting the code accepted.
- What didn't matter as much: By this stage, having a super-specific goal or extra background info wasn't the deciding factor anymore; the ability to prove it worked was.
Stage 3: Mixing it into the Final Dish (Integration Depth)
- What matters most: Context again.
- The Analogy: The code is accepted, but now it needs to be mixed into the big pot of the existing software. If the AI's code was written without knowing the "flavor profile" of the rest of the dish, it might taste weird and need to be heavily rewritten.
- The Finding: The more background information (Context) the developer gave the AI, the more the AI's code fit perfectly into the existing project. If the AI had to guess how it fit, the developers had to spend a lot of time fixing it later.
- The Lesson: To get code that fits seamlessly, you must tell the AI exactly where it belongs in the bigger picture.
3. The "Robot vs. Human" Checker
The researchers also tried to see if they could use an AI to grade these requests automatically.
- The Result: It was a mixed bag. The AI was okay at spotting "Specificity" (clear instructions) but was bad at spotting "Context" (it often missed the background info) and "Verification" (it struggled to see if a test was included).
- The Takeaway: You can't just let a robot grade all these requests. You need a human to double-check the tricky parts, especially the background context.
4. The Final Outcome (Did the PR get merged?)
Interestingly, the quality of the prompt didn't really change how fast the project managers decided to merge or close the request. That seemed to depend more on how big the request was and the general rules of the project, not just the prompt itself.
Summary
Think of working with AI in software development like a relay race:
- Start: You need Context and Specificity to get the baton (code) moving.
- Middle: You need Verification to prove the runner is on the right track so the next person takes the baton.
- Finish: You need Context again to make sure the runner crosses the finish line in the right lane without crashing into the other runners.
The paper concludes that writing a good prompt isn't just about getting code; it's about setting up the entire workflow for success. If you skip the "Context" or "Verification" steps, the code might get written, but it might get rejected or require a lot of extra work to fix later.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.