LLM-based Automated Architecture View Generation: Where Are We Now?

1. Problem Statement

Software architecture views are critical for documenting system design and communicating with stakeholders, yet their manual creation is labor-intensive, and existing artifacts are frequently outdated (75% are never updated). While Large Language Models (LLMs) have shown promise in code generation and documentation, their ability to bridge the gap between implementation details (source code) and high-level architectural abstractions (views) remains largely unexplored.

The core challenges identified are:

• Granularity Mismatch: LLMs tend to operate at the code level rather than abstracting to architectural patterns.
• Semantic Flaws: Generated views may be syntactically valid but semantically incorrect or lack appropriate architectural concerns.
• Evaluation Gap: Existing metrics often fail to capture the nuance of architectural correctness versus visual similarity.

2. Methodology

The authors conducted a large-scale empirical study to evaluate the effectiveness of LLMs and agentic approaches in generating architecture views directly from source code.

Experimental Setup

• Dataset: 340 open-source GitHub repositories (filtered from a larger dataset of 15,000 views) with manually annotated ground-truth architecture views.
• Scale: 13 experimental configurations yielding 4,137 generated views.
• Models & Agents:
  • LLMs: DeepSeek-V2.5, GPT-4o, and Claude-3.5 Sonnet.
  • Agents: A general-purpose coding agent (Claude Code) and a custom-built specialized agent (ArchView).
• Prompting Strategies: Zero-shot, One-shot, and Few-shot prompting.
• Workflow:
  1. Code Summarization: Source code is summarized (using hierarchical summarization for large contexts) to extract architectural context.
  2. View Generation: Models generate PlantUML code based on the summary and specific architectural concerns (e.g., deployment, security).
  3. Compilation: PlantUML is compiled into visual diagrams.
  4. Error Correction: A retry mechanism fixes syntax errors iteratively.

Evaluation Metrics

A hybrid evaluation approach was used:

1. Automated Metrics (LLM-as-a-Judge): Evaluated Clarity, Completeness, and Consistency (3Cs).
2. Image Similarity: Structural Similarity Index (SSIM) to measure visual correspondence with ground truth.
3. Human Evaluation: Two independent experts evaluated Accuracy and Level of Detail (LoD) on a blinded subset of 110 views.
4. Statistical Analysis: Friedman tests, Wilcoxon signed-rank tests, and Kruskal–Wallis tests were used to determine statistical significance.

3. Key Contributions

1. Large-Scale Empirical Dataset: Creation of a dataset containing 4,137 architecture views generated across 340 repositories, serving as a benchmark for future research.
2. Comprehensive Comparative Analysis: Systematic comparison of prompting strategies (Zero/One/Few-shot) against general-purpose and custom agentic workflows.
3. ArchView Framework: Introduction of a custom multi-agent framework (ArchView) that integrates domain knowledge, concern specifications, and dynamic notation adaptation, outperforming general agents.
4. Replication Package: Full release of datasets, prompts, and scripts to ensure reproducibility.

4. Key Results

Performance Comparison (RQ1)

• Custom Agent (ArchView) is Superior: ArchView achieved the best overall performance with a 22.6% failure rate for clarity and 50% success in Level of Detail. It significantly outperformed general-purpose agents.
• General Agents Struggle: The general-purpose agent (Claude Code) performed the worst, with a 71.8% clarity failure rate. It tended to generate code-level granularity rather than architectural abstractions.
• Prompting Impact: Few-shot prompting offered marginal improvements over zero-shot (reducing clarity failures by ~9.2%), but diminishing returns were observed.
• Statistical Significance: ArchView showed statistically significant improvements in Clarity and Consistency compared to other approaches (Large effect sizes).

Notation and Granularity (RQ2)

• Notation Complexity: Counter-intuitively, richer notations (e.g., "boxes and arrows, icons") yielded higher SSIM scores (0.738) compared to simple "boxes" (0.534). Complex vocabularies provided better structural cues for LLMs.
• Granularity: While high-granularity views showed slightly better structural similarity (SSIM), the differences were not statistically significant. However, semantic performance (LLM quality) declined as granularity increased, indicating difficulty in abstracting fine-grained details.

Architectural Concerns (RQ3)

• Reliable Views: Control Flow and Data Flow views demonstrated the highest reliability and balanced performance across visual and semantic metrics.
• Challenging Views: General Documentation and Security views were the most difficult to generate, often failing both visually and semantically.
• Semantic vs. Visual: A critical finding was the divergence between SSIM (visual similarity) and LLM-as-a-Judge (semantic correctness). For example, the general agent achieved moderate SSIM but near-total semantic failure, proving that visual similarity does not guarantee architectural accuracy.

5. Significance and Implications

For Researchers

• Bottleneck Identification: Code summarization remains a critical bottleneck; if the abstraction from code to summary fails, the view generation fails regardless of the model.
• Evaluation Metrics: Current metrics (like SSIM) are insufficient. Future frameworks must include architecture-specific metrics for abstraction appropriateness and stakeholder comprehensibility.
• Domain Knowledge: General-purpose LLMs are insufficient for architectural tasks. Specialized agents with domain knowledge and structured workflows are essential.

For Practitioners

• Assistive, Not Autonomous: LLMs should be positioned as assistive tools to accelerate documentation workflows, not as autonomous architects. Human oversight is required to correct granularity mismatches and semantic errors.
• Targeted Application: Automated generation is most viable for structural views (Control/Data Flow) but less reliable for complex concerns like deployment or security without specialized tuning.
• Knowledge Extraction: Even imperfect generated views serve as valuable starting points for understanding legacy systems and identifying gaps in documentation.

Conclusion

The study concludes that while LLMs can generate syntactically valid architecture views, they currently lack the ability to consistently produce architecturally accurate abstractions without specialized intervention. ArchView, a custom agentic approach integrating domain knowledge, represents the current state-of-the-art, significantly reducing failure rates compared to general prompting or general-purpose agents. However, the persistent gap between code-level details and architectural abstractions suggests that human expertise remains indispensable in the architecture documentation process.