💻 computer science

A Pilot Study on Detecting Software Design Patterns with Large Language Models: An Empirical Evaluation

This pilot study empirically evaluates the effectiveness of four Large Language Models and three ensemble approaches in detecting five software design patterns using source code, PlantUML diagrams, and text descriptions, finding that NextCoder and Gemma 3 achieve higher accuracy while ensemble methods improve overall detection efficiency.

Original authors: Oishik Chowdhury, Bastin Tony Roy Savarimuthu, Sherlock A. Licorish

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Oishik Chowdhury, Bastin Tony Roy Savarimuthu, Sherlock A. Licorish

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are handed a massive, tangled ball of yarn that represents a huge software program. To a new developer, this ball looks like a chaotic mess. To an experienced developer, however, they can spot specific, familiar knots and loops within the yarn. These knots are called Design Patterns. They are like standard recipes or blueprints (like "how to build a secure door" or "how to organize a library") that programmers use to solve common problems.

The problem is that finding these patterns manually is like trying to find a specific knot in that giant ball of yarn while blindfolded. It takes forever, and if you miss a knot, the whole structure might be weak or prone to breaking.

This paper is a pilot study (a small-scale test run) asking a big question: Can new "AI brains" (Large Language Models or LLMs) help us find these knots faster and better?

Here is the breakdown of their experiment, explained simply:

1. The "AI Detectives"

The researchers didn't just use one AI; they hired a team of four different "detectives" (AI models) to see who was best at spotting the patterns:

Three "Coder" AIs: These are AIs specifically trained to read and write computer code (like Qwen2.5, NextCoder, and Nxcode-CQ).
One "General" AI: This is a smart AI not specifically trained on code (Gemma 3), just to see if a general brain could figure it out.
The "Council": They also tried a strategy where they asked three detectives to vote on the answer. If two out of three said "Yes, this is a pattern," the Council said "Yes." This is called an Ensemble.

2. The Three Ways of Showing the Clues

The researchers tested the detectives using three different ways to present the "crime scene" (the software code):

The Raw Code: Showing the AI the actual computer code (like showing a detective the raw evidence).
The Blueprint (PlantUML): Showing a simplified diagram of how the code connects (like showing a detective a floor plan instead of the messy room).
The Story (Text Description): Asking another AI to write a plain English summary of what the code does (like a detective reading a witness's testimony).

3. The Five "Knots" They Looked For

They focused on five specific types of design patterns (the "knots"):

Singleton: Making sure there is only one "boss" in the room.
Adapter: Using a plug adapter to make two different things fit together.
Bridge: Building a bridge to connect two separate islands.
Composite: Building a tree where branches are made of smaller branches.
Decorator: Adding extra features to an object without changing its core (like putting a new coat of paint on a car).

4. What Did They Find? (The Results)

The Star Performers:

NextCoder and Gemma 3 were the top detectives. Surprisingly, the "General" AI (Gemma 3) was just as good, if not better, than the specialized "Coder" AIs in some cases.
The Council (Ensemble): When the detectives voted together, they were often more accurate than any single detective working alone. It's like having a team of experts double-checking each other's work.

The Input Mystery:

You might think showing the AI the Raw Code would be best. But the study found something weird: It didn't matter much!
Whether they showed the Raw Code, the Blueprint, or the Story, the AIs performed roughly the same.
The Twist: The "Story" (text description) actually helped the AI find more patterns (higher "recall"), while the "Blueprint" was very good at avoiding false alarms (higher "precision").

The Struggles:

The AIs were great at finding simple patterns (like the "Singleton" or "Decorator").
They struggled a bit with complex, messy patterns (like "Composite" or "Bridge"), often missing them or getting confused. This is like a detective missing a knot because the yarn was too tangled to see the whole picture.

5. Why Does This Matter?

Think of software as a city.

New developers are like tourists trying to navigate the city. They get lost easily.
Experienced developers are like locals who know the shortcuts.
Design Patterns are the street signs.

This study shows that AI can act as a super-powered GPS. It can scan a messy codebase and say, "Hey, look! Here is a 'Bridge' pattern here, and a 'Singleton' there." This helps new developers understand the city quickly and helps experienced developers fix broken streets before they cause traffic jams (bugs).

The Bottom Line

The paper concludes that AI is ready to help, but it's not perfect yet.

Good News: We don't necessarily need super-expensive, specialized "Coder" AIs; smart general AIs work well too.
Good News: Asking the AI to read a plain English summary of the code is just as effective as reading the code itself.
Future Work: The researchers want to test more patterns, look at bigger chunks of code (not just single files), and teach the AI to explain why it found a pattern, not just that it found one.

In short: AI is becoming a very helpful assistant for software architects, capable of spotting the hidden blueprints in our digital buildings.

1. Problem Statement

Software design patterns are reusable solutions to common architectural problems, but their incorrect implementation or misapplication can lead to code smells and anti-patterns. While automated detection is crucial for understanding system architecture, debugging, and refactoring, existing methods face significant challenges:

Static Analysis Limitations: Traditional graph-based and machine learning (ML) approaches often require explicit structural formalization, struggle with codebase complexity, and rely heavily on hand-crafted features that may miss semantic nuances.
Manual Detection: Identifying patterns manually is time-consuming and requires deep expertise, creating a barrier for new developers.
LLM Gaps: While Large Language Models (LLMs) show promise in semantic reasoning, prior studies have not systematically evaluated how different input modalities (raw code vs. visual representations vs. text descriptions) affect detection accuracy, nor have they extensively compared specialized "coder" models against general "non-coder" models for this specific task.

2. Methodology

The authors conducted a pilot study to evaluate the capabilities of LLMs in detecting five specific design patterns: Singleton, Adapter, Bridge, Composite, and Decorator.

Dataset

Source: The P-MART dataset, widely used in prior research.
Selection: 144 files containing implementations of the five target patterns were extracted.
Balancing: An equal number of files without the corresponding patterns were added to create a balanced binary classification dataset.
Pattern Distribution: Adapter (65), Composite (58), Singleton (12), Decorator (5), Bridge (4).

Models Evaluated

The study tested four distinct LLMs and two ensemble approaches:

Qwen2.5 Coder (32B): A specialized code model accessed via API.
NextCoder (7B): A specialized code model run locally (8-bit quantized).
NxCode-CQ (7B): A specialized code model run locally (8-bit quantized).
Gemma 3 (27B): A general-purpose (non-coder) model accessed via Google API.
Ensemble 1: Majority vote of the three coder models (Qwen, NxCode, NextCoder).
Ensemble 2: Majority vote of the three best-performing models (NxCode, NextCoder, Gemma 3).

Input Modalities (Prompting Strategies)

The models were evaluated using three distinct input formats for the same code instances:

Source Code: Raw code snippets.
PlantUML: Visual class diagrams generated from the source code.
Text Descriptions: Natural language descriptions of methods and variables generated by a separate LLM (Qwen-3-Coder).

Procedure

Models were prompted to answer "Yes" or "No" regarding the presence of a specific design pattern role, accompanied by an explanation.
Performance was measured using Accuracy, Precision, Recall, and F1 Score.
Statistical significance was analyzed using the Friedman test and Conover post-hoc pairwise tests.

3. Key Results

RQ1: Model Performance Comparison

Top Performers: NextCoder and Gemma 3 demonstrated the highest overall accuracy and F1 scores.
- Gemma 3 (a non-coder model) surprisingly outperformed two of the three specialized coder models, particularly in detecting patterns from text descriptions.
- NextCoder excelled in detecting Adapter and Bridge patterns.
- NxCode-CQ performed best for the Composite pattern.
Pattern Specifics:
- Singleton & Decorator: Detected with high accuracy (F1 > 0.88) across most models.
- Bridge & Composite: These proved more difficult, with lower recall scores generally observed.
Ensemble Effectiveness: Both ensemble approaches improved overall performance. Ensemble 2 (mixing coder and non-coder models) achieved the highest aggregate F1 score, suggesting that combining diverse model architectures mitigates individual weaknesses.
Statistical Significance: The Friedman test revealed significant differences among models ( $p=0.012$ ). Post-hoc tests confirmed that Qwen 2.5 Coder and NxCode-CQ performed significantly worse than Gemma 3 and the ensembles.

RQ2: Impact of Input Modalities

No Significant Difference: Statistical analysis (One-way repeated measures ANOVA) showed no statistically significant difference in F1 scores across the three input modalities (Code, PlantUML, Text Description) ( $p=0.50$ ).
Nuanced Trade-offs:
- Text Descriptions: Achieved the highest Recall (0.75), indicating better ability to identify true positive instances.
- PlantUML: Achieved the highest Precision, suggesting it reduces false positives but may miss some instances (lower recall).
- Source Code: Performed consistently but did not significantly outperform the other modalities.

4. Key Contributions

Empirical Evaluation of LLMs: Provides one of the first systematic comparisons of multiple state-of-the-art LLMs (both coder and non-coder) specifically for design pattern detection.
Modality Analysis: Introduces text-based descriptions as a novel input modality for pattern detection and demonstrates that it is as effective as raw code or visual diagrams, potentially offering a new avenue for non-coder models to excel.
Ensemble Validation: Demonstrates that majority-vote ensembles of heterogeneous models (mixing coder and non-coder) can outperform individual models, enhancing detection robustness.
Challenge to Specialization: The finding that a general-purpose model (Gemma 3) outperformed specialized coder models in certain contexts challenges the assumption that code-specialized LLMs are strictly necessary for architectural analysis.

5. Significance and Future Work

Practical Impact: The study suggests that automated tools for design pattern detection can be built using LLMs, potentially aiding developers in understanding legacy code, identifying refactoring opportunities, and onboarding new team members.
Cost-Efficiency: The success of smaller, quantized models (7B) and non-coder models implies that expensive, massive code-specific models may not always be required for this task.
Future Directions:
- Expanding the study to include all 23 GoF (Gang of Four) patterns.
- Moving from single-file analysis to multi-file, inter-connected context analysis.
- Investigating Chain-of-Thought (CoT) prompting and Few-Shot learning to improve reasoning.
- Exploring multi-modal inputs (combining code, PlantUML, and text simultaneously) to see if they yield synergistic improvements.

Conclusion: The paper concludes that LLMs are a promising tool for design pattern detection. While specialized coder models are effective, general models and ensemble strategies offer competitive or superior performance, and the format of the input (code vs. description) does not significantly hinder detection capabilities, offering flexibility in tool design.