From Prompt to Process: a Process Taxonomy and Comparative Assessment of Frameworks Supporting AI Software Development Agents
This paper introduces a six-dimensional process taxonomy and scoring rubric to comparatively assess six emerging AI software development frameworks, revealing a convergence toward persistent artifacts and human review while highlighting a structural trade-off between process depth and portability alongside critical risks like specification drift and platform dependence.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you've hired a brilliant, hyper-fast apprentice to help you build a house. This apprentice is an AI coding agent. In the past, you might have just shouted, "Build me a wall!" and hoped for the best. Sometimes the wall would be great; other times, it would be crooked, made of the wrong bricks, or built in the wrong spot.
This paper argues that simply shouting instructions (prompts) isn't enough anymore. We need frameworks—which are like detailed rulebooks, blueprints, and management systems—to guide these AI apprentices. The author, Sanderson Oliveira de Macedo, looked at six popular "rulebooks" currently used in the industry to see how they organize the work.
Here is a breakdown of the paper's findings using simple analogies:
1. The Problem: From "Shouting" to "Managing"
In the old days, you talked to the AI one sentence at a time. It was like playing a game of "telephone" where the message gets lost. The AI would forget what you said five minutes ago, or it would make up facts (hallucinate).
The paper says we are moving to a new era where the AI doesn't just chat; it works. It plans, edits files, runs tests, and fixes its own mistakes. But without a manager, this autonomous worker can get chaotic. The "frameworks" the paper studies are the managers that tell the AI:
- What to build (Specification).
- What it already knows about the project (Context).
- Who is doing what (Roles).
- How to build it (Execution).
- How to check if it's right (Validation).
- Whether it can work on different construction sites (Portability).
2. The Six Rulebooks (The Frameworks)
The author picked six specific "rulebooks" to compare. Think of them as different management styles:
- GitHub Spec Kit & OpenSpec: These are like architects. They insist you write a perfect blueprint (a specification) before the AI lays a single brick. They focus heavily on the plan and can work with many different AI tools.
- BMAD Method: This is like a corporate HR department. It breaks the work down into specific roles (Product Manager, Architect, Developer, QA) and assigns the AI to act as these different people. It's very structured but can be heavy.
- Get Shit Done (GSD): This is like a personal assistant who only works for one specific boss (a specific AI tool). It's great at organizing the AI's memory and focus, but it's not very flexible if you want to switch bosses.
- Spec Kitty: This is like a construction site with safety fences. It isolates the AI's work in a separate area (a "worktree") so it can't accidentally break the main building. It forces a human to inspect the work before it gets merged.
- Reversa: This is the reverse engineer. Instead of building a new house from scratch, it looks at an old, crumbling building (legacy code) and tries to figure out the original blueprints so the AI can fix it.
3. The Big Discovery: The "No Free Lunch" Trade-off
The most important finding of the paper is that no single rulebook is perfect.
The author created a scoring system (a rubric) to grade these frameworks. Here is the analogy:
- Some frameworks are like Swiss Army Knives: They are light, portable, and work everywhere, but they don't have a deep, specialized tool for every job. They are great at planning but weak at checking the work.
- Other frameworks are like Heavy-Duty Construction Cranes: They are incredibly powerful, have strict safety checks, and deep processes, but they are hard to move and only work in specific places.
The Trade-off: The more deeply a framework manages the process (checking every step, assigning roles), the harder it is to move that framework to a different AI tool. You can't have the deepest process and the easiest portability at the same time with current tools.
4. The Hidden Dangers (Risks)
The paper also warns about "construction site hazards" that these frameworks haven't fully solved yet:
- Drift: The blueprint (specification) might say "red brick," but the AI builds a "blue wall" anyway, and nobody notices until it's too late.
- Blind Trust: We might trust the AI's "finished" work too much, even if it looks good but is actually broken underneath.
- Fragile Extensions: These frameworks often rely on community-made add-ons. If the person who made the add-on stops updating it, the whole system could break.
- Lock-in: Some frameworks are so tied to one specific AI tool that if that tool changes its rules, your whole process falls apart.
5. What's Next? (The Research Agenda)
The paper concludes that we are currently in the "wild west" phase. We have cool tools, but we don't have enough data to know which one actually works best in the long run.
The author suggests we need to stop just showing off cool demos and start doing real science:
- Measure the middle steps: Don't just check if the final code works; check if the AI's plan and the blueprint were good.
- Test the memory: Does the AI actually read the right files, or is it guessing?
- Watch the teams: See how real human teams perform over months, not just days.
Summary
The paper is a map of the current landscape of AI software tools. It tells us that while we have moved from "chatting with AI" to "managing AI teams," we haven't found the perfect manager yet. We have to choose between tools that are flexible and easy to move, or tools that are deep and rigorous but hard to switch. The future lies in building better ways to measure if these tools are actually making software better, not just faster.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.