To Copilot and Beyond: 22 AI Systems Developers Want Built
Through a survey of 860 Microsoft developers, this paper identifies 22 desired AI systems that prioritize "bounded delegation"—absorbing assembly tasks while preserving professional craft—alongside critical requirements for quality signals, authority scoping, and uncertainty management to address the growing right-shift burden in AI-assisted development.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are the head chef of a bustling, high-end restaurant. For years, your job has been chopping vegetables, searing steaks, and plating dishes. But recently, you hired a super-fast robot arm that can chop vegetables and sear stears ten times faster than you ever could.
At first, it seems like a miracle. You're cooking more food in less time! But then, a problem emerges. The robot is so fast that the kitchen is suddenly flooded with perfectly cooked steaks and chopped onions, but nobody knows how they were made.
- Did the robot use the right spices?
- Is the meat actually safe to eat?
- Where did the ingredients come from?
- Who is responsible if a customer gets sick?
Now, instead of spending your day cooking, you are stuck in a panic, trying to inspect every single steak the robot made, re-reading the recipe books, and answering questions from health inspectors. You are working harder than before, even though the robot is doing the "cooking."
This is exactly what this paper is about.
The Big Problem: The "Robot Chef" Paradox
The researchers (from Oregon State University and Microsoft) asked 860 software developers: "You know those AI coding tools (like GitHub Copilot) that write code for you? What else do you need them to do?"
They found a funny mismatch. Most AI tools focus on writing the code (the chopping and searing). But developers only spend about 10% of their day actually writing code. The other 90% is spent on the messy, human stuff:
- Debugging why things broke.
- Writing documentation so others understand the code.
- Checking if the code is secure.
- Explaining technical decisions to non-tech bosses.
- Onboarding new team members.
The AI is speeding up the 10% part, but it's leaving the other 90% (the "messy stuff") untouched. This is creating a bottleneck. The code is being generated faster than humans can verify, understand, or trust it.
The Solution: 22 New "Kitchen Assistants"
The developers didn't just want a faster robot chef. They wanted 22 specific types of AI assistants to handle the rest of the work. They grouped these into five categories:
1. The "Tech Debt Janitor" (Development)
- The Problem: Old codebases are like old houses with creaky floors and outdated wiring. Everyone knows it needs fixing, but no one wants to do it because it's boring and risky.
- The AI Wish: A robot that can safely clean up old code, fix dependencies, and update libraries, but only if it asks for permission first and doesn't accidentally knock down a wall.
- The Catch: It must stop if it gets confused. It can't just "guess" how to fix a complex problem.
2. The "Architect's Assistant" (Design & Planning)
- The Problem: Designing a new system is like planning a skyscraper. You need to think about traffic flow, safety, and future expansions.
- The AI Wish: A tool that can brainstorm different building designs, point out potential weak spots, and organize the to-do list.
- The Catch: The AI can suggest ideas, but it cannot make the final decision. The human architect must remain the boss. The AI shouldn't just copy-paste old designs; it needs to understand the specific context of this building.
3. The "Safety Inspector" (Quality & Risk)
- The Problem: When you ship code, you need to make sure it doesn't have bugs or security holes. Currently, this happens after the code is written, which is too late.
- The AI Wish: An inspector that checks the code while you are writing it. It should say, "Hey, you forgot a test for this part," or "This looks like a security risk."
- The Catch: The AI can flag problems, but it cannot approve the code for release. A human must always sign off.
4. The "Infrastructure Butler" (Operations)
- The Problem: Keeping servers running is like managing a complex electrical grid. When an alarm goes off at 3 AM, you need to know why immediately.
- The AI Wish: A tool that gathers all the logs, traces, and history to tell you, "The server crashed because of this specific change," so you can fix it fast.
- The Catch: The AI can look at the data, but it cannot touch the live servers to fix things. It can't restart the engine without a human pressing the button.
5. The "Knowledge Keeper" (Meta-Work)
- The Problem: Documentation is always outdated. New hires spend weeks trying to figure out how things work because the manuals are wrong.
- The AI Wish: A tool that automatically updates the manuals when code changes, creates personalized training plans for new hires, and helps write emails to bosses that sound professional but not robotic.
- The Catch: The AI can draft the email or the manual, but a human must read and approve it before sending. It can't talk to customers on its own.
The Golden Rule: "Bounded Delegation"
The most important finding of the paper is a concept the authors call "Bounded Delegation."
Think of it like hiring a very smart intern.
- You want the intern to do: The filing, the data entry, the research, the formatting, the drafting. (The "assembly work").
- You NEVER want the intern to do: The final decision, the signing of the check, the ethical judgment, or the creative spark. (The "craft").
Developers are saying: "We want AI to do the heavy lifting, but we want to keep the steering wheel."
They don't want the AI to replace their identity as engineers. They want the AI to handle the boring, repetitive parts so they can focus on the creative, high-level problem-solving.
The Four "Guardrails"
For any of these 22 AI systems to be acceptable, developers said they must follow four strict rules:
- Know Your Limits: If the AI doesn't know the answer, it must say "I don't know" instead of making things up.
- Show Your Work: The AI must show where it got its information (provenance). No magic answers.
- Ask First: It can't just change things. It must ask for permission before making a move.
- Least Privilege: It should only have access to the specific data it needs, nothing more. No opening the safe just to get a pen.
The Bottom Line
This paper is a wake-up call. We've been obsessed with making AI write code faster. But the real value of AI in software engineering isn't about how much code it can generate. It's about where it stops.
The best AI tools won't be the ones that replace developers. They will be the ones that protect developers from burnout, help them verify their work, and let them focus on the parts of the job that make them feel like skilled professionals.
In short: We don't need a robot that cooks the meal for us. We need a robot that washes the dishes, so we can enjoy the dinner.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.