Large Language Models for Analyzing Enterprise Architecture Debt in Unstructured Documentation

1. Problem Statement

Enterprise Architecture (EA) Debt refers to the accumulation of suboptimal design decisions, outdated technologies, and misaligned components that degrade an organization's IT landscape. These issues often manifest as EA Smells (symptoms of potential debt).

• The Gap: Current detection methods rely heavily on manual expert assessment or the analysis of structured artifacts (e.g., ArchiMate models, knowledge graphs).
• The Challenge: A significant portion of architectural knowledge resides in unstructured documentation (process descriptions, strategy papers, meeting notes, policies). These sources contain critical semantic insights (e.g., ambiguous ownership, temporary solutions) that structured models miss.
• The Need: There is a lack of automated tools capable of analyzing unstructured text to detect EA Smells, particularly in environments with strict data privacy constraints that prevent the use of cloud-based APIs.

2. Methodology

The study employs a Design Science Research (DSR) approach to design, implement, and evaluate an artifact (a software prototype) for automated EA Smell detection.

A. Scope and Requirements

• Target Layer: The Business Layer of EA (processes, goals, roles), where documentation is predominantly unstructured and governance is often lighter.
• Target Smells: From a catalog of 63 known EA Smells, the authors selected 12 specific smells deemed feasible for automated detection via LLMs (e.g., "Contradiction in Input," "Temporary Solution," "Shiny Nickel," "Lack of Documentation").
• Constraints: The system must run on-premise on a resource-constrained environment (Windows, 16 GB RAM, CPU-only) to ensure data protection.

B. Artifact Design

The prototype pipeline consists of:

1. Ingestion: Processing .docx and .pdf files.
2. Preprocessing: Tokenization and chunking with overlap to retain context.
3. Detection: An LLM analyzes chunks to identify smells and extract rationales.
4. Reporting: Generation of a report with findings, severity, and traceable snippets.

C. Model Selection and Training

• Base Model: LLaMA-3.2-3B-Instruct. Selected for its instruction-following capabilities and compatibility with Parameter-Efficient Fine-Tuning (PEFT) on CPU hardware.
• Fine-Tuning Strategy: Used LoRA (Low-Rank Adaptation) combined with Few-Shot Learning.
  • Dataset: 960 synthetic instances (80 examples per smell across 8 business domains like Order-to-Cash, Procure-to-Pay).
  • Configuration: 50% positive (smell present) and 50% negative examples. Trained for 4 epochs.
• Benchmark: A Custom GPT (proprietary, cloud-based) was configured with the same system prompt and training data (via in-context learning/uploaded files) to serve as a performance baseline.

D. Evaluation Protocol

• Dataset: 30 synthetic but realistic business documents created using DeepSeek AI, containing embedded "ground truth" annotations of EA Smells.
• Protocols:
  • Run 1: Single document processing.
  • Run 2: Batches of 10 documents.
  • Run 3: Single batch of 30 documents (to test scalability; GPT could not handle this due to interface limits).
• Metrics: Accuracy, Precision, Recall, F1-Score, False Positive Rate (FPR), processing time, and adaptability.

3. Key Contributions

1. Problem Analysis & Requirements: Defined functional and non-functional requirements for LLM-based EA Smell detection, emphasizing data privacy and resource constraints.
2. Artifact Implementation: Developed and open-sourced a fine-tuned, on-premise LLM prototype capable of detecting 12 specific business-layer EA Smells from unstructured text.
3. Empirical Evaluation: Conducted a comparative study between a resource-constrained local model (LLaMA-3-3B) and a proprietary cloud baseline (Custom GPT), analyzing trade-offs in accuracy, speed, and scalability.

4. Results

A. Performance Metrics (Runs 1 & 2)

Metric	Model A (On-Prem LLaMA)	Model B (Custom GPT)
Precision	Low (0.26 / 0.19)	High (0.88 / 0.89)
Recall	Low (0.22 / 0.13)	Modest (0.22 / 0.25)
F1 Score	0.24 / 0.15	0.35 / 0.39
False Positive Rate	Very High (0.95 / 1.00)	Very Low (0.09)
Accuracy	0.15 / 0.08	0.40 / 0.42

• Model A (LLaMA): Exhibited high sensitivity but low precision. It flagged an average of 10.5 smells per document, often generating false positives (e.g., flagging "Shiny Nickel" on standard process descriptions) and generic rationales. It suffered from "over-detection."
• Model B (GPT): Exhibited high precision but modest recall. It was conservative, rarely flagging false positives, but missed some embedded smells, particularly in documents with multiple overlapping issues.

B. Processing Time & Scalability

• Speed: Model B was significantly faster (2 seconds per document) compared to Model A (120 seconds per document on CPU).
• Batching: Model A handled a batch of 30 documents in ~50 minutes. Model B was limited to 10 documents per batch due to interface constraints.
• Scalability: Increasing batch size for Model A did not improve precision; outputs became more generic and omitted specifics.

C. Error Analysis

Both models exhibited specific failure modes:

1. Omission: Missing embedded smells, especially in multi-smell documents.
2. Misclassification: Mapping a valid issue to the wrong smell (e.g., confusing "Temporary Solution" with "Big Bang").
3. Context Leakage: In batch processing, text from one document was sometimes attributed to another.
4. Fabricated Citations: Rationales citing sentences not present in the source text.

5. Significance and Implications

• Feasibility of On-Premise LLMs: The study proves that LLMs can detect EA Smells in unstructured text even on constrained hardware, making them viable for organizations with strict data sovereignty requirements. However, current small models require human-in-the-loop validation due to high false positive rates.
• Triage vs. Diagnosis: The authors conclude that LLMs should be used as triage tools to surface potential issues for architects to review, rather than as definitive diagnostic engines.
• Trade-off Management: There is a clear trade-off between data privacy/control (On-premise LLaMA) and accuracy/speed (Cloud GPT). Organizations must decide based on their sensitivity to data leakage versus their need for precision.
• Future Directions:
  • Improving ground truth via multi-annotator protocols.
  • Developing hybrid architectures that convert unstructured text into Knowledge Graphs before analysis.
  • Enhancing retrieval-augmented generation (RAG) to reduce context leakage and improve traceability.

In summary, while the on-premise prototype is not yet ready for fully automated production use due to accuracy issues, it successfully demonstrates a pathway for integrating LLMs into EA governance, particularly for organizations that cannot utilize public cloud APIs.