💻 computer science

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

CodeTeam is an LLM-powered multi-agent framework that addresses the challenges of natural-language-to-repository generation by separating planning, decision-making, and implementation into coordinated stages, achieving state-of-the-art performance in both design quality and functional correctness on benchmark tests.

Original authors: Yifei Wang, Ruiyin Li, Peng Liang, Qiong Feng, Zengyang Li, Mojtaba Shahin, Arif Ali Khan

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifei Wang, Ruiyin Li, Peng Liang, Qiong Feng, Zengyang Li, Mojtaba Shahin, Arif Ali Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Building a Whole City from a Sketch

Imagine you ask a very smart, but slightly scattered, architect (an AI) to build an entire city from a single sentence: "I need a place where people can buy shoes."

If you just ask the AI to "write the code," it might build a beautiful shoe store, but it forgets to build the roads, the power plant, or the sewage system. Or, it might build the shoe store with a door that leads to a brick wall because it didn't talk to the "road builder" AI.

This is the challenge of NL2Repo (Natural Language to Repository). It's not just about writing a single function (like a single room); it's about generating a whole software project (a whole city) with many files that must talk to each other perfectly. Current AI models often get lost in the details, forgetting the big picture or creating "cross-file" messes where File A expects something File B didn't build.

The Solution: CodeTeam (The Construction Crew)

The authors propose CodeTeam, a system that doesn't rely on one lone AI genius. Instead, it acts like a well-organized construction crew with specialized roles. They break the job down into three distinct phases: Planning, Decision Making, and Building.

Here is how the team works, step-by-step:

1. The Architects (The Dreamers)

Instead of one person drawing the blueprints, the system hires four different Architect agents.

What they do: Each one sketches a different design for the software. One might say, "Let's use a modular design!" Another says, "No, let's keep it simple and flat!"
The Secret Weapon: Sometimes, these architects are allowed to peek at a library of successful past projects (Retrieval-Augmented Generation) to see how other people solved similar problems. This helps them avoid reinventing the wheel.
The Goal: Create a variety of "Software Design Sketches" (SDS). Think of these as detailed blueprints that list every room, every pipe, and who is responsible for building what.

2. The CTO (The Chief Decision Maker)

Once the four architects present their blueprints, a Chief Technical Officer (CTO) agent steps in.

What they do: The CTO reviews all the sketches, picks the best one, and turns it into a Machine-Checkable Contract.
The Contract: This isn't just a drawing; it's a strict legal document for the computer. It says: "File A must have this specific function. File B must depend on File A. Developer 1 is in charge of the kitchen; Developer 2 is in charge of the bedroom."
Why it matters: This contract prevents the builders from wandering off and building a garage where the kitchen should be. It sets the rules before a single brick is laid.

3. The Developers (The Builders)

Now, the actual coding begins. The system hires a specific number of Developer agents based on the CTO's contract.

Specialization: Unlike a generic AI that tries to do everything, these developers are assigned specific files. Developer 1 only builds the login page. Developer 2 only builds the database.
The Git Coordination (The Foreman): As they build, they use a lightweight version of Git (a tool developers use to track changes). When Developer 1 changes the login page, they leave a "commit message" (a note) saying, "I changed the password button." Developer 2 reads this note and updates their own code to match.
Dependency Awareness: The system knows that you can't paint the walls before you build the frame. It schedules the work so that files are built in the right order.

4. The QA Agent (The Inspector)

While the team builds, a Quality Assurance (QA) agent acts like a building inspector.

What they do: They run tests to see if the building stands up. If a door doesn't open or a pipe leaks, the QA agent doesn't just say "Error." They figure out who broke it and send a repair ticket back to that specific developer.
The Loop: The developer fixes the issue, the inspector checks again, and they repeat until the building is perfect.

What Did They Find? (The Results)

The researchers tested CodeTeam against other methods (like a single AI trying to do it all, or other multi-agent teams) using two main "exams":

The Blueprint Exam (SketchEval): They checked if the generated code looked structurally correct compared to real-world examples.
- Result: CodeTeam won. It built structures that looked much more like real software. The "Architects" and "CTO" steps helped them get the layout right, while the "QA" steps fixed the small cracks.
- Key Insight: The "Dynamic Developer Allocation" (hiring the right number of builders for the specific job) was the biggest factor in success. If you hire too few or too many, or assign the wrong people to the wrong rooms, the building fails.
The Live Test (NL2Repo-Bench): They actually tried to run the generated software to see if it worked.
- Result: CodeTeam had the highest success rate. It didn't just look good on paper; it actually functioned.
- Key Insight: By fixing structural errors early (like missing files or broken connections), the final product was much more likely to pass the "live" tests.

The Takeaway

The paper argues that building software from scratch isn't just a "writing" task; it's a management task.

Old Way: Ask one AI to write a whole book. It often forgets plot points or writes characters that don't match.
CodeTeam Way: Hire a team. Have one person plan the plot, one person edit the chapters, and one person check for typos.

By separating planning (Architects/CTO) from doing (Developers) and adding checking (QA), CodeTeam creates software that is not only smarter but also more reliable. It proves that for complex tasks, a coordinated team of AI agents is far better than a single, super-smart AI working alone.

Technical Summary: CodeTeam

Problem Statement

The paper addresses the challenge of Natural Language to Repository (NL2Repo) generation: constructing an entire software repository from a natural-language requirements document. Unlike function-level code generation (e.g., HumanEval), NL2Repo requires long planning horizons, stable cross-file interfaces, and iterative debugging of inconsistencies that only manifest during system-level integration.

Existing approaches face three critical gaps in this domain:

Lack of Machine-Checkable Contracts: Prior methods often lack a formal, machine-readable specification of the repository structure before coding begins.
Cross-File Inconsistency: Agents struggle to maintain interface consistency across multiple files during implementation.
Coordination Deficits: Current workflows often fail to coordinate bounded-context implementation against a global plan with dependency-aware scheduling and iterative repair.

While benchmarks like RepoBench and FEA-Bench focus on completion or incremental changes within existing codebases, the NL2Repo setting (from-scratch generation) demands architectural design, dependency management, and packaging from an empty workspace.

Methodology: The CodeTeam Framework

CodeTeam is an LLM-powered multi-agent framework that separates planning, decision-making, and implementation into distinct, coordinated stages. It utilizes a Software Design Sketch (SDS) as a central, machine-checkable contract.

1. Planning Stage: Architect Competition & CTO Selection

Architect Agents: Multiple Architect agents ( $N=4$ $N = 4$ ) independently draft competing SDSs. These sketches define the technology stack, file structure, public interfaces, and a developer allocation plan (specifying team size and file ownership).
- RAG Grounding: Optionally, architects retrieve design references from a curated corpus of public repositories to ground their sketches, preventing design convergence and missing modules.
- Diversity Mechanisms: Architects use randomized design preferences and receive only high-level summaries of previous candidates to ensure diverse solutions.
CTO Agent: A Chief Technical Officer agent evaluates candidates based on structural validity, interface consistency, implementability, and team appropriateness. It selects the best SDS and normalizes it into a canonical, machine-actionable contract (JSON format) that serves as the "single source of truth."

2. Implementation Stage: Dependency-Aware Execution

Developer Agents: Based on the CTO's selected plan, specific Developer agents are instantiated. Each agent owns a specific subset of files.
Context & Scheduling: Developers operate under a dependency-aware scheduler. They receive a compact context containing the target file, relevant SDS fragments, and interface briefs for dependencies (signatures, invariants), rather than full file contents.
Git-Based Coordination: Agents work on isolated branches. Upon completing a file, they commit changes with a structured update_reason message. This allows other agents to update their interface briefs without accessing full file contents, maintaining cross-file consistency.

3. Repair Stage: QA-Driven Iteration

QA Agent: A Quality Assurance agent runs lightweight, generated tests against completed dependency layers.
Feedback Loop: If failures occur, the QA agent diagnoses the root cause (e.g., interface mismatch, missing symbol) and routes the repair request to the specific owner of the source file. Dependent files are requeued only if the interface change propagates.
Termination: The loop continues until no failures are detected, a limit is reached, or resources are exhausted.

Key Contributions

CodeTeam Framework: A novel multi-agent system for NL2Repo that integrates SDS competition, optional RAG grounding, CTO-guided contract normalization, project-specific developer allocation, dependency-aware execution, Git-based coordination, and QA-driven repair.
Comprehensive Evaluation: A rigorous evaluation on the SketchEval benchmark (comparing Prompt-Engineering and Supervised Fine-Tuning variants against CodeS and other baselines) and an external validation protocol on NL2Repo-Bench (execution-based testing).
Ablation Insights: Systematic analysis demonstrating that retrieval-augmented planning and dynamic developer allocation are the primary drivers of performance, while Git-based coordination provides complementary stability.

Experimental Results

Performance on SketchEval (Structural Similarity)

Prompt-Engineering (PE): CodeTeam achieved an overall SketchBLEU of 51.7%, outperforming the strongest baseline, CodeS (PE), by 4.1 absolute points (8.6% relative).
Supervised Fine-Tuning (SFT): CodeTeam achieved 60.9%, surpassing CodeS (SFT) by 2.9 points.
Sub-score Analysis: Improvements were most pronounced in Structural Matching and Dataflow Matching, indicating better repository-level integrity and cross-file consistency compared to surface-level token overlap.
Difficulty Scaling: The performance gap widened significantly on Hard repositories (dense dependencies), where CodeTeam improved by 4.7 points over CodeS (PE).

Performance on NL2Repo-Bench (Execution-Based)

Pass Rate: CodeTeam achieved the highest average test pass rate in both settings: 34.6% (PE) and 42.3% (SFT).
Error Analysis: CodeTeam significantly reduced packaging errors (from 34.2% to 22.1%) and import errors (from 28.7% to 19.4%) compared to CodeS. This confirms that the SDS contract effectively prevents structural omissions (e.g., missing __init__.py or incorrect package names).
Shift in Failure Modes: As structural errors decreased, the proportion of logic errors increased, suggesting that CodeTeam successfully addresses organizational failures, shifting the bottleneck to semantic correctness.

Ablation Study Findings

RAG (Retrieval-Augmented Generation): Removing RAG caused an 8.1% relative drop in SketchBLEU. It primarily improved the planning stage (SDS parse success and structural validity) and plan diversity.
Dynamic Developer Allocation: Removing this mechanism caused the largest drop (9.9% relative). It is critical for aligning team size and file ownership with the repository's modular structure, reducing QA rounds and interface mismatches.
Git Coordination: Removing this caused a smaller but consistent drop (2.9% relative), acting as a reactive mechanism to stabilize interface drift during repairs.

Significance and Claims

The paper claims that NL2Repo is not merely a long-context code generation task, but a coordinated software engineering process.

Workflow over Model Capacity: The results suggest that structured multi-agent workflows (planning, role specialization, iterative repair) provide measurable benefits even when the backbone model is fine-tuned. The workflow compensates for cross-file consistency limitations that pure prompting or SFT cannot fully resolve.
Explicit Contracts: The SDS serves as a machine-checkable contract that binds file boundaries and dependencies, enabling precise repair routing and preventing the "cascading rework" common in single-agent or loosely coordinated systems.
Practical Deployment: The study demonstrates that investing tokens in upfront architectural planning (SDS generation) and dynamic allocation reduces downstream rework. It also suggests that a limited number of QA iterations (2–3 rounds) captures the majority of quality improvements, making the framework viable for resource-constrained scenarios.

The authors conclude that CodeTeam demonstrates how LLM-based repository generation can be made more structured, inspectable, and extensible by treating agents as a coordinated development team rather than isolated code generators.