Technical Summary: Towards Automated Identification of Violation Symptoms of Architecture Erosion

1. Problem Statement

Software architecture erosion occurs when the implemented system diverges from its intended architecture, leading to degraded maintainability, performance, and sustainability. While various symptoms of erosion exist (structural, quality, and evolution), violation symptoms—instances where the code violates design principles, patterns, or decisions—are the most immediate indicators requiring attention.

Traditionally, identifying these violations relies on static code analysis or manual review of textual artifacts. However, static analysis tools often suffer from limited violation type detection, narrow language support, and an inability to capture semantic nuances. Manual identification through code reviews is accurate but tedious, time-consuming, and prone to human error. Despite the richness of information in code review comments regarding architectural decisions, no prior study had investigated the automated identification of violation symptoms specifically within textual artifacts like code review comments.

2. Methodology

The study employs a multi-phase empirical approach to assess the feasibility of automating the detection of violation symptoms from code review comments.

2.1 Dataset Construction

• Source: Code review comments from four open-source projects: OpenStack (Nova, Neutron) and Qt (Base, Creator), collected via the Gerrit REST API between 2014 and 2020.
• Labeling: A dataset of 21,583 comments was mined. Four researchers manually labeled 606 comments as "violation symptoms" (positive class) and 606 as "non-violations" (negative class) to create a balanced training set. Inter-rater agreement (Cohen's Kappa) was 0.857.
• Imbalanced Test Set: A separate held-out test set of 150 comments (7 violations, 143 non-violations) was constructed to simulate real-world low-prevalence conditions (~4.7% positive rate).

2.2 Phase 1: ML/DL Classification

The authors trained and evaluated classifiers using three pre-trained word embedding techniques: word2vec, fastText, and GloVe.

• Machine Learning (ML): Five algorithms were tested: Support Vector Machine (SVM), Logistic Regression (LR), Decision Tree (DT), Bernoulli Naive Bayes (NB), and k-Nearest Neighbor (kNN).
• Deep Learning (DL): A TextCNN model was trained.
• Ensemble Learning: A majority voting strategy (Hard Voting) was applied to combine predictions from individual classifiers.
• Hyperparameter Tuning: Grid search was used for ML models. For DL, specific settings included 200-dimensional embeddings, a learning rate of 0.001, and dropout of 0.25.
• Imbalanced Handling: Cost-sensitive learning techniques (e.g., class weighting, SMOTE resampling, cost matrices) were applied to the training data to address the natural imbalance in the test set.

2.3 Phase 2: Validation

• Qualitative: An online survey (24 responses) and semi-structured interviews (6 participants) were conducted with practitioners involved in the original code reviews to assess the perceived usefulness of the automated approach.
• Quantitative: A controlled experiment was performed with 35 developers (Control vs. Experimental groups). The Experimental group received hints generated by the classifiers highlighting potential violation symptoms. The metric was the proportion of violations correctly identified and fixed.

2.4 Phase 3: LLM Comparison

Three state-of-the-art Large Language Models (LLMs) were evaluated using prompt engineering (Zero-shot and Few-shot learning with 2, 4, 6, and 10 examples):

• GPT-4o (Proprietary)
• Qwen-3 (Open-weight)
• DeepSeek-R1 (Open-source MoE)
  Cost-aware prompting strategies were used to mitigate bias toward the majority class in the imbalanced setting.

3. Key Results

3.1 ML/DL Performance (RQ1 & RQ2)

• Best Classifier: Among ML/DL approaches, the SVM classifier with word2vec embeddings achieved the highest performance on the balanced dataset (F1-score: 0.808, Precision: 0.789, Recall: 0.828).
• Embedding Impact: 200-dimensional word embeddings generally outperformed 100 and 300-dimensional models. fastText embeddings yielded relatively good results for DL models (TextCNN).
• Ensemble: Ensemble classifiers using majority voting generally improved performance over individual ML classifiers (e.g., F1-score improvements up to 13.10%). However, ensembling did not consistently improve DL classifiers (TextCNN).
• Imbalanced Robustness: Under natural low-prevalence conditions, cost-sensitive SVM models maintained high Macro-F1 scores (0.816), demonstrating effectiveness in realistic scenarios. Statistical significance analysis (Friedman test) confirmed that performance differences were not due to random variation.

3.2 LLM Performance (RQ4)

• Superiority: The best LLM-based classifiers outperformed traditional ML/DL approaches.
• Top Performer: GPT-4o with a 2-shot setting achieved the highest F1-score (0.851) and Accuracy (0.848) among all evaluated models.
• Ensemble Limitations: Unlike ML/DL, ensembling LLMs (across models or shot settings) did not consistently improve performance and sometimes degraded it, likely due to the high baseline performance of individual LLMs and potential amplification of hallucinations.

3.3 Practical Validation (RQ3)

• Practitioner Feedback: 66.7% of survey respondents agreed that the identified symptoms represented potential architectural violations. Most agreed the approach could help locate and prioritize issues faster than manual review.
• Controlled Experiment: Providing violation symptom hints significantly improved developers' detection rates. The rate of correctly identifying and fixing violations (Category 1) increased from 13.0% (Control) to 29.4% (Experimental). The overall detection rate (Category 1 + 2) rose from 25.9% to 64.7%, a 150% relative improvement.

4. Key Contributions

1. Feasibility Demonstration: The first study to demonstrate the feasibility of automatically identifying architecture violation symptoms from code review comments using ML, DL, and LLMs.
2. Comprehensive Comparison: A rigorous comparison of 15 ML classifiers, 4 DL classifiers, and 3 LLMs across different embedding dimensions and shot settings.
3. Practitioner-Validated Insights: Empirical evidence from surveys, interviews, and a controlled experiment confirming that automated hints significantly improve developer performance in detecting architectural issues.
4. Dataset and Tools: The release of a labeled dataset of architecture violations from code reviews and the replication package to support future research.

5. Significance and Claims

The paper claims that automated identification of violation symptoms serves as a complementary layer to modern code review, offering early warnings to prevent the integration of code that deviates from the intended architecture.

• For Practitioners: The approach offers a practical tool to reduce the cognitive load of identifying architectural issues, helping teams prioritize refactoring and maintain architectural integrity.
• For Research: The study highlights that while LLMs (specifically GPT-4o) currently offer superior performance, traditional ML approaches (SVM with word2vec) remain highly effective and more cost-efficient for specific text classification tasks, especially when computational resources are constrained.
• Methodological: The work bridges the gap between theoretical architecture conformance checking and practical developer workflows by leveraging the rich semantic context found in code review discussions.

The authors conclude that while challenges regarding false positives and integration costs remain, the quantitative and qualitative evidence strongly supports the integration of such automated techniques into software development lifecycles to enhance system sustainability.