Using LLMs to Evaluate Architecture Documents: Results from a Digital Marketplace Environment

Technical Summary: Using LLMs to Evaluate Architecture Documents – Results from a Digital Marketplace Environment

Problem Statement
Software development is increasingly reliant on generative AI, yet the quality of AI output remains fundamentally dependent on the quality of its input. Early-stage artifacts, such as software architecture documents, are critical inputs for downstream implementation. Defects, ambiguities, or missing information in these documents can lead to costly correction cycles later in the development lifecycle. While traditional methods for evaluating architecture documentation exist (e.g., ATAM, SAAM, manual checklists), they are often time-consuming and difficult to scale. The authors investigate whether Large Language Models (LLMs) can effectively evaluate the quality of architecture documents and provide feedback comparable to human software architects, specifically within the context of a digital marketplace for smart city solutions.

Methodology
The research was conducted within the "Quasar" tool, an LLM-based system developed to support the evaluation of open-source solutions on the "Deutschland Digital" marketplace. The study focused on two specific open-source projects with varying levels of documentation completeness:

1. Project 1: A solution with comprehensive documentation (266 files, including Markdown, PDFs, and images).
2. Project 2: A solution with basic documentation (11 Markdown files and 25 images).

System Architecture and Implementation:

• Input Processing: The system accepts various documentation formats (PDF, DOCX, Markdown, images). A converter (using Docling) transforms these into readable formats.
• Evaluation Logic: The system employs a weighted quality model where specific criteria are evaluated against predefined prompts.
• LLM Integration: The "Quasar" backend utilizes an "LLM client" component to decouple the evaluation logic from specific model vendors. For the study, the authors used locally hosted Qwen2 (multimodal, capable of processing text and diagrams) and Llama3.1 (text-only) on NVIDIA Tesla V100 GPUs.
• Procedure: Two software architects manually evaluated the two projects based on 25 predefined statements (rated 0–4). Simultaneously, Quasar evaluated the same projects three times each using the same 25 prompts to assess consistency and agreement with human experts.

Key Contributions

1. Tool Development: The creation of "Quasar," a modular tool capable of ingesting diverse documentation formats and generating quality scores based on LLM analysis.
2. Empirical Comparison: A direct comparison of LLM-based evaluation against human expert evaluation across two distinct documentation scenarios (comprehensive vs. sparse).
3. Contextual Analysis: An investigation into how the quality and density of input artifacts influence the reliability of LLM evaluations.

Results
The study yielded significant findings regarding the relationship between input quality and LLM performance:

• Consistency (RQ 1.2):
  • Project 1 (High-quality input): The three automated runs showed largely consistent results.
  • Project 2 (Low-quality input): The runs showed weak consistency. In three instances, the model failed to produce a result for specific statements across all runs.
• Agreement with Humans (RQ 1.3):
  • Project 1: There was a strong correlation. Two statements matched exactly, and 36% of assessments differed by a maximum of 15%. The average deviation was 27%.
  • Project 2: No apparent correlation was established. The average deviation was 43%, indicating that the LLM could not reliably replicate human judgment on sparse documentation.
• Effort (RQ 1.1):
  • Processing times were comparable between humans and the machine for the smaller dataset (22 mins human vs. 14 mins Quasar for Project 2; 60 mins human vs. 68 mins Quasar for Project 1). The authors note that for larger datasets, the automated solution is expected to be faster.

Significance and Claims
The paper claims that LLMs are a promising support tool for software architects but emphasizes that their utility is highly conditional on the quality of the input artifacts.

• Input Dependency: The core finding is that the better the quality and structure of the architecture document, the more consistent and accurate the LLM's evaluation is relative to human experts. Conversely, low information density leads to significant deviations and hallucinations.
• Role of AI: The authors position LLMs not as a replacement for human experts, but as a mechanism for "initial, lightweight quality assurance" that can provide an early impression of documentation quality, thereby saving time for experts.
• Limitations: The authors explicitly state that the study is limited by a small sample size (two projects, two experts) and the specific models used. They caution that generalization is not possible based on current results and that the "inconsistencies" observed in low-quality inputs require further analysis before the technology can be broadly generalized.
• Future Outlook: The authors suggest that future iterations should utilize larger context windows (e.g., Google Gemini), Retrieval-Augmented Generation (RAG) to preserve context across files, and fine-tuning to improve domain-specific knowledge. They also envision integrating these capabilities directly into architects' daily tools (e.g., Enterprise Architect, draw.io) rather than as standalone services.