🤖 AI

Evaluating Large Language Models for Detecting Architectural Decision Violations

This study evaluates the effectiveness of using a multi-model Large Language Model pipeline to detect architectural decision violations in open-source software, finding that while LLMs are highly accurate at identifying code-related violations, they still struggle with implicit or deployment-oriented decisions that require human expertise.

Original authors: Ruoyu Su, Alexander Bakhtin, Noman Ahmad, Matteo Esposito, Valentina Lenarduzzi, Davide Taibi

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruoyu Su, Alexander Bakhtin, Noman Ahmad, Matteo Esposito, Valentina Lenarduzzi, Davide Taibi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, complex LEGO city. To make sure the city doesn't turn into a chaotic mess, you write a "Rule Book" (these are the Architectural Decision Records or ADRs). The rules might say, "All skyscrapers must be blue" or "No houses can be built next to the train tracks."

As the city grows, people start building things quickly. Eventually, someone builds a red skyscraper or a house right on the tracks. In software, this is called "architectural drift"—it’s when the code stops following the original plan, making the whole system messy and prone to breaking.

The Problem: The "Manual Inspector" Headache

Normally, to catch these mistakes, you’d need a human "Master Architect" to walk through the entire city, look at every single LEGO brick, and compare it to the Rule Book. This is exhausting, slow, and humans get tired and miss things.

The Experiment: Hiring "Robot Inspectors"

The researchers in this paper wanted to see if Large Language Models (LLMs)—the "brains" behind AI like ChatGPT—could act as automated inspectors.

They didn't just hire one robot; they set up a "Robot Jury."

The Lead Inspector (LRM): One AI looked at the rules and the code and said, "I think this is a violation!"
The Jury (Validators): Three other AIs looked at the Lead Inspector's work to see if they agreed.
The Human Judges: Finally, real human experts checked the robots' work to see who was actually right.

The Results: How smart are the robots?

1. They are great at "Visual" Rules (The High Scores)
If the rule is simple and clearly visible in the code—like "All functions must use this specific naming style"—the robots were incredible. They were right over 90% of the time. It’s like a robot looking at a blue LEGO brick and instantly confirming it matches the "Blue Rule."

2. They struggle with "Vibe" and "Hidden" Rules (The Low Scores)
The robots hit a wall when the rules weren't about the "bricks" themselves, but about how the city functions.

The "Invisible" Rules: If a rule is about how the city's electricity grid works (infrastructure) or how different neighborhoods talk to each other (system interaction), the robots got confused. They couldn't "see" the electricity; they could only see the bricks.
The "Missing Info" Problem: Sometimes the robots couldn't tell if a rule was broken or if they just didn't have enough information to decide. This is like a robot looking at a house and saying, "I don't know if this house is allowed to be here because I can't see the map of the train tracks."

The Verdict: A Helpful Assistant, Not a Replacement

The paper concludes that AI is like a highly efficient junior inspector.

What they can do: They can scan thousands of lines of code in seconds and flag obvious mistakes, saving the human architect a massive amount of boring work.
What they can't do: They lack the "big picture" wisdom. They don't understand the intent behind a rule or the complex, invisible connections that keep a system running.

The takeaway: Don't fire your human architects! Instead, give them these "Robot Inspectors" to handle the grunt work, so the humans can focus on the complex, high-level thinking that requires true wisdom.

Technical Summary: Evaluating Large Language Models for Detecting Architectural Decision Violations

1. Problem Statement

Software architecture integrity is maintained through Architectural Decision Records (ADRs), which document the reasoning behind design choices. However, software "architectural drift"—where the actual implementation deviates from the intended design—is a common cause of technical debt and complexity.

Detecting these violations is traditionally a manual, error-prone process because ADRs are often written in natural language, scattered across repositories, and lack machine-readable formats. While traditional static analysis tools exist, they struggle to interpret the high-level, implicit, or informal architectural knowledge captured in ADRs. This paper investigates whether Large Language Models (LLMs) can automate the detection of these violations by reasoning across both natural language documentation and source code.

2. Methodology

The researchers designed a large-scale empirical study using a multi-model pipeline to ensure robustness and minimize individual model bias.

Dataset: The study utilized 980 ADR files extracted from 109 GitHub repositories. The selection was filtered to ensure projects were substantial (top quartile of TLoC and commits) and that ADRs followed recognizable Markdown templates.
The Multi-Model Pipeline:
- Large Reasoning Model (LRM): Marco-o1 acted as the primary detector, responsible for extracting decisions and determining if the code was Compliant (C), Not Compliant (NC), or if the Code is Insufficient to Answer (CIA).
- Validation Models (V1–V3): Mistral-Nemo, Qwen3, and Llama-3.1 acted as independent "experts" to verify the LRM’s reasoning and judgment.
Technical Implementation:
- RAG (Retrieval-Augmented Generation): To handle large codebases that exceed context windows, the authors used a RAG approach. They embedded the source code using the all-MiniLM-L6-v2 transformer and used a FAISS vector database to retrieve the top-5 most relevant code snippets for each ADR.
- Prompting: They employed Chain-of-Thought (CoT) and few-shot prompting to guide the models through complex architectural reasoning.
Human Validation: To establish a "ground truth," three human experts manually reviewed a sample of 305 decisions using a majority-voting protocol to validate the LLMs' accuracy and reasoning.

3. Key Contributions

Large-Scale Empirical Evaluation: One of the first studies to evaluate multiple state-of-the-art LLMs on the specific task of ADR violation detection across over 100 open-source projects.
Multi-Model Validation Framework: A novel pipeline that combines a primary reasoning model with multiple independent validators and human experts.
Taxonomy of Failures: A systematic classification of why LLMs fail, categorized by both ADR types (e.g., infrastructure-specific) and error types (e.g., semantic misinterpretation).

4. Key Results

Agreement (RQ1): The models showed substantial agreement (Fleiss’ $\kappa = 0.724$ ). Consistency was highest when the code was clearly compliant (C) and lowest when determining if information was insufficient (CIA) or non-compliant (NC).
Accuracy (RQ2): LLMs demonstrated high performance. The LRM (Marco-o1) achieved the highest accuracy (91.1%), followed closely by Qwen3 (90.4%).
- Class-wise Performance: Models were most accurate at detecting Compliant decisions. They were moderately successful at detecting Not Compliant decisions but struggled significantly with the CIA label, indicating difficulty in distinguishing between "no violation" and "missing evidence."
Limitations (RQ3):
- Hardest ADR Types: Infrastructure and Deployment-specific decisions (42.39% of errors) were the most difficult, as they involve tools like Docker, Kubernetes, or CI/CD pipelines that require specialized environmental context.
- Primary Error Drivers: The most common error was Semantic and Logical Misinterpretation (44.57%), where models failed to grasp the nuanced logical structure of a decision.

5. Significance and Implications

The study concludes that LLMs are not yet ready to replace human architects but are highly effective decision-support tools.

For Practitioners: LLMs can automate routine compliance checks for "code-centric" decisions (e.g., design patterns, module boundaries), acting as an early-warning system for architectural drift.
For Researchers: The findings suggest that future improvements should focus on improving RAG retrieval quality and integrating static analysis with LLM reasoning to bridge the gap in infrastructure and deployment-level understanding.
Hybrid Workflows: The authors advocate for a "human-in-the-loop" approach where LLMs flag potential issues for human review, particularly for complex, context-dependent, or infrastructure-heavy architectural decisions.