Technical Summary: CodePori – Large-Scale System for Autonomous Software Development Using Multi-Agent Technology

Problem Statement

Existing Large Language Model (LLM)-based multi-agent systems have demonstrated capability in executing tasks and providing recommendations for software engineering (SE). However, current research predominantly relies on benchmark datasets (e.g., HumanEval, MBPP) that yield binary pass-or-fail results. These benchmarks often focus on isolated, small-scale tasks or single-function generation, failing to capture the complexities of real-world, large-scale software development. Consequently, there is a significant lack of empirical research examining the practical applicability, strengths, and limitations of LLM-based agents when tasked with generating code for large-scale systems characterized by multi-file modular structures, intricate inter-component communication, and advanced architectural designs.

Methodology

The study employed a two-phase empirical approach to investigate the potential of LLM-based agents in autonomous software development:

1. System Development: CodePori

The authors designed and developed CodePori, a multi-agent system capable of autonomously generating code for large-scale systems from high-level project descriptions. The system utilizes a collaborative, six-agent architecture, each with a specific persona and instruction set, orchestrated to mimic a human development pipeline:

• Manager Agent: Analyzes high-level descriptions and segments tasks into structured, JSON-formatted modules (defining objectives, inputs/outputs, dependencies).
• Architecture Agent: Transforms modular specifications into a comprehensive architectural overview, defining component interactions and data flow based on the specified programming language.
• Flow Structure Agent: Generates a structured file and folder hierarchy aligned with the tech stack.
• Dev Agent: Generates initial code for each module, ensuring adherence to coding guidelines, Google-style docstrings, and iterative refinement.
• Verification Agent: Conducts detailed code analysis to identify flaws, ensuring alignment with project descriptions and inter-module correctness. It does not modify code directly but instructs the Dev agent on necessary improvements.
• Finalization Agent: Reviews the implementation, iteratively collaborates with other agents to enhance quality, and ensures the code is production-ready.

The system uses the Claude-3.5 Sonnet model as the underlying LLM. A centralized state manager maintains context (project description, architecture, code files, verification reports) to facilitate communication and minimize redundancy. The workflow involves iterative loops where agents refine code based on feedback from previous steps.

2. Participant-Based Evaluation

To assess the system's practical applicability, the authors conducted a survey involving 118 participants (students from a "Fine-Tuning Large Language Models" course at Tampere University).

• Task: Participants provided five different project descriptions each (totaling 590 inputs) covering diverse categories (e.g., Games, AI/ML, Finance, Chatbots).
• Evaluation: Participants evaluated the system's performance regarding speed, accuracy, requirement interpretation, and code quality. They provided feedback on strengths, challenges (e.g., hallucinations, memory limits), and areas for improvement.
• Data Analysis: Quantitative analysis was performed on closed-ended questions, while open-ended responses were analyzed using open-coding and selective coding methods to identify core categories of feedback.

Key Contributions

The paper makes the following specific contributions:

1. System Design: The development of CodePori, a multi-agent system designed to autonomously generate code for large-scale systems from high-level descriptions, moving beyond single-function generation.
2. Empirical Insights: An evaluation based on feedback from 118 participants and 590 project descriptions, providing real-world data on the effectiveness and limitations of multi-agent systems in large-scale code generation.
3. Comprehensive Analysis: A detailed identification of the system's strengths (e.g., speed, structure) and specific challenges (e.g., hallucinations, short-term memory, code smells) that are often missed by standard benchmarks.
4. Open Resources: Public release of the survey dataset and the CodePori system source code on GitHub to support replication and further research.

Results

The study yielded mixed but informative results regarding the practical applicability of CodePori:

Performance Metrics

• Speed: The system generated code for projects in an average of approximately 9 minutes and 14 seconds.
• Participant Ratings:
  • Poor: 30.51%
  • Good: 20.33%
  • Average/Fair: 49.16% (26.28% Average, 22.88% Fair)

Identified Strengths

Participants highlighted several positive aspects:

• Fast Code Generation: Significant reduction in development time for setup, dependencies, and initial structure.
• Structured Output: Generation of well-organized file/folder architectures and multi-layer designs.
• Requirement Interpretation: Ability to translate user requirements into functional code structures.
• Documentation: Generation of concise code comments and documentation.
• ML Suitability: Effective generation of functional machine learning code requiring only minor modifications.

Identified Challenges

Participants reported several critical limitations preventing fully autonomous deployment:

• Hallucinations (11.86%): Generation of unrequested components, unnecessary dependencies, or misinterpretation of requirements.
• Short-Term Memory Limitations (5.93%): Agents forgetting previously generated classes or functions in other modules, leading to calls to non-existent methods.
• Code Smell & Quality (16.95%): Issues included hardcoded constants, code duplication, dead code, and poor overall maintainability.
• Code Inconsistency: Lack of uniform patterns in function calls and variable definitions across generated files.
• Dependency Management: Conflicts in library versions and improper handling of imports (e.g., relative vs. absolute imports in Python).
• Orchestration Failures: Issues in coordinating agents across different files, leading to missing modules or circular dependencies.
• Syntax Errors & Configuration: Frequent syntax errors and difficulties in getting generated files to work cohesively without manual intervention.

Areas for Improvement

Participants suggested specific enhancements:

• Integration of a progress tracking system for real-time monitoring.
• Enhanced debugging features allowing agents to analyze error logs and runtime environments.
• Mechanisms for human intervention when agents crash or encounter complex errors.
• Improved framework selection logic (e.g., defaulting to vanilla JavaScript unless a framework is explicitly requested).
• Automated library import management (e.g., enforcing absolute imports in Python).

Significance and Claims

The paper claims that while LLM-based multi-agent systems show potential for large-scale software development, their successful integration requires addressing specific technical challenges and adopting a practitioner-centric perspective.

• Beyond Benchmarks: The study argues that standard benchmarks (HumanEval, MBPP) are insufficient for evaluating large-scale systems as they focus on isolated tasks and fail to capture repository-level challenges like multi-file reasoning and inter-component communication.
• Practitioner-Centric Evaluation: The research highlights the necessity of moving beyond automated pass/fail metrics to include human feedback on code quality, maintainability, and real-world applicability.
• Future Directions: The authors conclude that future research must focus on mitigating hallucinations, improving agent memory and coordination, and developing benchmarks that reflect complex, real-world development scenarios. They emphasize that human oversight remains crucial for verifying outputs and ensuring reliability in production systems.

The study does not claim that CodePori is a fully autonomous solution ready for immediate industrial deployment without human intervention. Instead, it positions the system as a significant step toward automating SE processes, identifying the specific gaps that must be bridged to achieve broader adoption in both industry and academia.