How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses
This study empirically analyzes how five AI coding agents differ in their pull request description styles and how these variations influence human reviewer engagement, response timing, sentiment, and merge outcomes, highlighting the critical role of presentation in human-AI software collaboration.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a bustling construction site where a new type of worker has just arrived: AI Coding Agents. These aren't just tools that help you lay a single brick; they are autonomous robots that can build entire rooms, write blueprints, and hand you a finished project to inspect.
This paper is like a forensic investigation into how these robot workers communicate with their human supervisors (the code reviewers) when they submit their work.
Here is the story of the study, broken down into simple concepts:
🏗️ The Setup: The "Handover" Problem
In the old days, when a human worker finished a job, they would write a note explaining what they did. Sometimes the note was messy, sometimes it was perfect. Now, we have five different robot workers (let's call them Copilot, Claude, Cursor, Devin, and Codex).
The researchers wanted to know:
- How do these robots write their notes? (Do they use bullet points? Do they say "please"? Do they use emojis?)
- How do the human supervisors react? (Do they get angry? Do they ignore the note? Do they approve the work quickly or send it back?)
🤖 The Five Robot Personalities
The study found that each robot has a very distinct "personality" when writing their pull request descriptions (the notes attached to their work):
- OpenAI Codex (The Organized Architect): This robot is the most structured. It uses clear headings, bullet points, and lists. It's like a student who highlights the most important parts of their essay.
- Result: Humans loved this. They reviewed it fast, and it got approved the most often.
- Claude Code (The Chatty Diplomat): This robot writes long, detailed notes and uses a lot of emojis to make things friendly. It's like a colleague who sends a very polite, enthusiastic email.
- Result: Humans engaged with it a lot, wrote long comments, and it got approved quickly, though it sparked a lot of discussion.
- GitHub Copilot (The Over-Worker): This robot changes a lot of code and writes very long descriptions, but it doesn't organize them well. It's like a worker who dumps a huge pile of bricks on your desk and says, "Here, build a wall."
- Result: Humans were confused. They wrote a lot of comments trying to figure it out, but the work got approved the least often. It took the longest to finish.
- Cursor (The Polite Minimalist): This robot keeps its notes short and plain text, but it is incredibly polite (using words like "please" and "thank you"). It's like a quiet intern who is very respectful but doesn't explain much.
- Result: Surprisingly, it got approved very quickly, even though humans sometimes gave it negative feedback (perhaps because the notes were too vague).
- Devin (The Strict Splitter): This robot follows the rules perfectly but breaks its work into tiny, separate pieces (many small commits). It's like a worker who submits a new brick every 5 minutes.
- Result: Humans got tired of checking so many small pieces. It took a long time to finish, and it often got rejected without a clear resolution.
🧐 The Human Reaction: The "Review"
The researchers looked at how humans responded to these different styles. They found that how the robot presents its work changes how the human feels and acts.
- Structure is King: When the robot organized its notes clearly (like Codex), humans felt less stressed, reviewed it faster, and said "Yes" more often.
- Politeness vs. Clarity: Being polite (like Cursor) didn't necessarily mean the work was easier to understand. Sometimes, a polite but vague note made humans frustrated.
- The "Wall of Text" Effect: When the robot dumped too much unorganized information (like Copilot), humans spent hours arguing in the comments, and the project got stuck.
🏁 The Big Takeaway
The main lesson of this paper is simple: In a team of humans and robots, how you say something is just as important as what you say.
Even if the robot writes perfect code (the bricks are good), if the note explaining it is messy, confusing, or rude, the human supervisor will struggle.
- Good Communication = Fast Approval.
- Bad Communication = Confusion, Anger, and Delays.
The Analogy:
Think of the AI agents as delivery drivers.
- Codex is the driver who leaves a neatly labeled package with a clear note: "Fragile, handle with care." You open it immediately.
- Copilot is the driver who dumps a giant, unmarked box on your porch and runs away. You have to spend hours digging through it to find what you need.
- The Study proves that if you want your deliveries (code) to be accepted, you need to teach your drivers (AI) how to write better notes.
💡 Why This Matters
As we move into an era where AI does more of the heavy lifting, we can't just expect the AI to "do the work." We need to design these AI agents to be better communicators. If we teach them to write clear, organized, and polite notes, human workers will be happier, and software will get built faster.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.