CIAO - Code In Architecture Out - Automated Software Architecture Documentation with Large Language Models

1. Problem Statement

Software architecture documentation is critical for system comprehension, maintenance, and evolution. However, in both industrial and open-source projects, such documentation is frequently missing, outdated, or inconsistent with the actual code implementation. This leads to architectural drift, erosion, and technical debt.

While recent Large Language Models (LLMs) have shown promise in generating local documentation artifacts (e.g., API descriptions, code comments, or test summaries), existing approaches generally fail to produce coherent, system-level architectural descriptions that adhere to established standards. There is a gap in using LLMs to automatically reconstruct a holistic architectural view directly from a GitHub repository that satisfies professional standards like ISO/IEC/IEEE 42010.

2. Methodology: The CIAO Process

The authors propose CIAO (Code In Architecture Out), a structured, automated workflow that transforms a GitHub repository into a system-level architectural document. The process consists of four main stages:

A. The Standards-Oriented Template

CIAO utilizes a custom template derived from three major frameworks:

• ISO/IEC/IEEE 42010: For defining system scope, stakeholders, and rationale.
• SEI Views & Beyond: For separating structural, behavioral, and deployment perspectives.
• The C4 Model: For a four-level hierarchy (Context, Container, Component, Code).

The template comprises eight specific sections:

1. System Overview: Purpose, scope, and responsibilities.
2. Architectural Context: External actors, systems, APIs, and data sources (C4 Level 1).
3. Containers: Runtime logical organization (applications and data stores) (C4 Level 2).
4. Components: Internal logical structure, modules, and relationships (C4 Level 3).
5. Code-Level: Mapping components to specific source files, directories, and patterns (C4 Level 4).
6. Cross-Cutting Concerns: Security, logging, configuration, etc.
7. Quality Attributes & Rationale: Performance, scalability, and design decisions.
8. Deployment: Infrastructure, nodes, and execution environments.

B. The Automated Workflow

1. Repository Flattening: The tool uses REPOMIX to convert the entire repository into a single, AI-friendly text file. It filters out binaries and large datasets while retaining source code, configuration files (e.g., Dockerfile, package.json), and the folder structure.
2. Prompt Engineering: CIAO employs a task-decomposition strategy. Instead of generating the whole document at once, it creates parallel prompts for each of the eight sections.
   • Global Prompt: Defines the role ("Meticulous Software Architect"), audience, and constraints (e.g., "do not hallucinate").
   • Section-Specific Prompt: Tailored to the specific goals of the section (e.g., generating PlantUML diagrams for the Container view).
3. LLM Generation: The system submits these prompts to an LLM (specifically GPT-5 was selected based on a pilot study for its accuracy and stability).
4. Assembly & Rendering: Generated text sections are assembled into a single Markdown document. Textual PlantUML diagrams are automatically rendered into images and embedded into the final output.

3. Key Contributions

• A Standards-Based Template: A novel, lightweight template for system-level documentation that aligns with ISO 42010, SEI, and C4, specifically designed to be consumable by LLMs.
• The CIAO Workflow: A fully automated, end-to-end pipeline that takes a GitHub repository URL as input and outputs a ready-to-use architectural document (suitable for a README.md).
• Empirical Validation: A comprehensive study involving 22 developers who evaluated the documentation generated for their own repositories.
• Open-Source Prototype: The release of the implementation, generated documentation, and survey data for reproducibility.

4. Experimental Results

The study evaluated the generated documentation based on five Research Questions (RQs):

• RQ1 (Value): Developers found the documentation valuable (15/22 agreed or strongly agreed). The most appreciated sections were Architectural Diagrams and the Components view, which helped in understanding dependencies and modification points.
• RQ2 (Comprehensibility): The documentation was rated as clear and well-structured. Terminology was consistent with standard architectural practices. High-level sections (Context, Containers) were particularly easy to follow.
• RQ3 (Accuracy): The documentation was generally accurate regarding the implemented architecture.
  • High Accuracy: Sections grounded in explicit code evidence (Components, Containers, Code-Level) received the highest agreement.
  • Lower Accuracy: High-level interpretive sections (System Overview, Architectural Context) and Use Case Diagrams showed more inconsistencies and omissions.
• RQ4 (Limitations): The primary limitations identified were:
  • Diagram Errors: 32 occurrences of issues, including truncated class diagrams, misleading relationships, and "artificial" looking visuals.
  • Deployment Issues: Confusing relationships in deployment diagrams and missing infrastructure details.
  • Missing Information: Occasional omissions of specific components or use case tables.
• RQ5 (Cost & Time): The process is highly efficient.
  • Time: Average generation time was ~3 minutes per repository.
  • Cost: Average API cost was ~$1.19 per document.

5. Significance and Conclusion

The paper demonstrates that a structured, standards-oriented approach can effectively guide LLMs to produce usable, system-level architectural documentation.

• Practical Impact: The tool offers a cost-effective solution for "architectural recovery," helping teams quickly understand legacy systems, onboard new developers, or address architectural drift.
• Feasibility: The low time and monetary cost make it viable for integration into CI/CD pipelines or regular maintenance cycles.
• Future Directions: The authors acknowledge that diagram generation is the weakest link. Future work will focus on integrating static/dynamic analysis to improve diagram accuracy and extending the evaluation to industrial-scale, closed-source projects.

In summary, CIAO bridges the gap between raw code and high-level architectural understanding, proving that LLMs can be guided to produce documentation that is not just a code summary, but a structured architectural artifact aligned with industry standards.