Technical Summary: AgenticAKM

Problem Statement

Architecture Knowledge Management (AKM) is essential for maintaining accurate, comprehensive software architecture documentation, yet it remains a laborious process often neglected by developers and architects. This neglect leads to "knowledge vaporization," where critical design rationales, component structures, and decision histories become disconnected from the implementation.

While Large Language Models (LLMs) offer a potential pathway to automate the generation of Architecture Decision Records (ADRs) and other artifacts, naive, single-prompt approaches face significant limitations:

1. Context Constraints: LLMs cannot process entire codebases due to context window limits.
2. Distributed Knowledge: Architectural knowledge is scattered across code, commit histories, issue trackers, and documentation, which a single prompt fails to synthesize effectively.
3. Quality Issues: Direct LLM outputs often lack depth, omit design rationales, and fail to provide verifiable reasoning, resulting in incomplete or misleading documentation.

Methodology: AgenticAKM

To address these limitations, the authors propose AgenticAKM, a multi-agent system that decomposes the complex task of architecture recovery and documentation into manageable, specialized sub-tasks. The system is coordinated by a central Orchestrator and supervised by a human architect who configures the workflow and monitors execution.

The architecture consists of four distinct agent groups, each equipped with specific tools and roles:

1. Architecture Extractor Agents: Responsible for parsing code repositories and documentation to create foundational summaries.
   • Repo Summarizer: Analyzes the codebase to identify components and functionalities.
   • Issue Summarizer: Extracts context from issue trackers (e.g., Jira) regarding pain points and feature requests.
   • Document Summarizer: Condenses existing requirements and technical documents.
2. Architecture Retriever Agents: Locate and retrieve relevant information from internal knowledge bases or external sources.
   • ADR Retriever: Performs semantic searches over vector databases of past decisions.
   • Diagram & Requirement Retrievers: Fetch existing architectural diagrams and requirement specifications using file system and vector search.
3. Architecture Generator Agents: Create new artifacts based on the context provided by extractors and retrievers.
   • ADR Generator: Drafts structured ADRs (Title, Context, Decision, Consequences).
   • Diagram & Specification Generators: Create visual models (e.g., UML, C4) and technical specifications.
4. Architecture Validator Agents: Act as quality control checkpoints to ensure accuracy and compliance.
   • Summary & Document Validators: Cross-reference generated summaries against source code and facts.
   • ADR Validator: Scrutinizes logical consistency, format, and redundancy.
   • Policy Validator: Ensures compliance with organizational standards.

Workflow: The system operates iteratively. For example, in the ADR generation instantiation, a Repo Summarizer creates a summary which is validated. If rejected, it loops back for refinement (up to three times). Once approved, the ADR Generator drafts records, which are then validated. If rejected, the generator refines the draft. This iterative loop ensures each stage builds upon a validated foundation.

Key Contributions

• AgenticAKM Framework: A novel multi-agent paradigm for AKM that moves beyond monolithic LLM calls to a structured, collaborative workflow.
• Decomposition Strategy: The division of architecture recovery into specialized roles (Extraction, Retrieval, Generation, Validation) to handle distributed knowledge and context limits.
• Instantiation and Validation: An initial implementation focused on generating ADRs from code repositories, validated through a comparative user study.

Experimental Results

The authors conducted a user study involving 29 unique code repositories (ranging from 1,000 to 350,000 lines of code, primarily Python and JavaScript) and 13 participants (students and professionals).

Experimental Setup:

• Comparators: A "Baseline Approach" (single prompt to an LLM) vs. the "Agentic Approach" (iterative multi-agent workflow).
• Models: Experiments were run using GPT-5 and Gemini-2.5-pro (top-ranked models at the time of the study).
• Metrics: Participants rated ADRs on Relevance, Coherence, Completeness, Conciseness, and Overall Quality (1–5 scale).

Findings:

• Quantitative: The Agentic approach consistently outperformed the Baseline LLM-only approach across all metrics.
  • Overall Quality: Agentic (3.8–3.9) vs. Baseline (3.3).
  • Completeness: The Agentic approach showed significantly higher scores, addressing the "omissions" common in single-prompt outputs.
  • Conciseness & Relevance: Agentic outputs maintained high scores (3.9–4.3), balancing brevity with informational richness.
• Qualitative: Participants described Agentic outputs as "very structured and clear" and reflective of "actual architectural reasoning." In contrast, Baseline outputs were often criticized as "very wordy," "abstract," or "generic."
• Language Nuance: While repository language generally did not impact quality significantly, the authors noted that for Java repositories, LLM-only approaches occasionally generated higher-quality ADRs than the Agentic approach, suggesting room for further optimization.

Significance and Claims

The paper claims that AgenticAKM represents a promising and practical approach for automating AKM. By decomposing the problem and introducing iterative validation, the system generates ADRs that are more complete, concise, and contextually accurate than those produced by naive single-prompt LLM interactions.

The authors position this work as a shift from design synthesis (creating new architectures) to knowledge recovery and preservation (extracting and documenting existing knowledge). They emphasize that while the current instantiation focuses on ADRs, the framework is designed to be extensible to other artifacts (e.g., C4 diagrams) and multi-modal data, offering a robust methodology for reducing the manual burden of software architecture documentation.