🤖 AI

A Methodology for Investigating AI Patterns Prevalence in Software Repositories

This paper proposes and validates a methodology that combines literature mining to identify 14 AI pattern classes with active learning to empirically measure their prevalence and accuracy across real-world GitHub repositories, addressing a current gap in understanding how AI patterns are actually used in practice.

Original authors: Srinath Perera, Hasinthaka Piyumal, Frank Leymann, Rania Khalaf

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Srinath Perera, Hasinthaka Piyumal, Frank Leymann, Rania Khalaf

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a city planner trying to understand how people actually build houses in a bustling, modern city. You have a thick book of "Ideal House Designs" (the AI patterns proposed by experts in literature), but you have no idea if anyone is actually using those designs in the real neighborhoods, or if they are just making up their own weird structures as they go.

This paper is the team of researchers who decided to go out, tour 100 actual house neighborhoods (GitHub code repositories), and figure out which "Ideal Designs" are truly being used.

Here is how they did it, explained simply:

1. The Problem: Too Many Ideas, Not Enough Proof

Experts have written down hundreds of "AI Patterns" (recipes for building smart software). But nobody knew if these recipes were actually being cooked in real kitchens. The researchers wanted to move from "theory" to "reality."

2. Step One: Cleaning Up the Recipe Book

First, they gathered 769 different "recipes" from books, blogs, and documentation. That was too messy to work with.

The Analogy: Imagine having 769 different names for "making a sandwich." Some say "club," others say "BLT," others say "ham and cheese on wheat."
The Fix: They used a smart computer (an LLM) to group similar recipes together. Then, humans looked at the groups and organized them into 14 clear categories (like "RAG," "Agent Architecture," "Preprocessing," etc.). This created a clean, organized menu of 14 main dishes.

3. Step Two: Finding the "Neighborhoods" (Code Communities)

They couldn't just look at a single line of code; that's like judging a whole house by looking at a single brick. They needed to find "communities" of code that work together.

The Analogy: Instead of looking at individual bricks, they looked at the "rooms" of the house. They used a special map (a call graph) to find groups of code that are tightly connected, like a kitchen where the stove, fridge, and sink are all linked.
The Selection: They picked 100 real-world AI projects from GitHub (the "neighborhoods") that were popular but not the massive, foundational frameworks (like TensorFlow). They wanted to see what typical developers were building, not just the giant tools everyone else builds on top of.

4. Step Three: The "Guess and Check" Game (Active Learning)

This is the cleverest part. They didn't have a teacher with the answer key (labeled data) to train their computer model.

The Analogy: Imagine a student taking a test but not knowing the answers.
1. The student makes a few guesses based on a few examples they generated themselves.
2. The computer looks at the 100 neighborhoods and says, "I'm 50/50 on this one, but I'm sure about that one."
3. The researchers (the human teachers) only look at the ones the computer is unsure about. They label those, and then the computer learns from them.
4. They repeat this cycle. The computer gets smarter with every round, asking for help only when it's truly confused. This saved them from having to read and label thousands of files manually.

5. Step Four: Counting the Results (With a Safety Net)

Once the computer was trained, it scanned the 100 neighborhoods to count how often each pattern appeared.

The Twist: The computer isn't perfect. It makes mistakes. So, the researchers didn't just give a single number (e.g., "RAG is used 20% of the time").
The Safety Net: They used a statistical trick (like running a simulation 20,000 times) to say, "We are 95% sure the real number is between X and Y." This gives a "confidence interval," acknowledging that their estimate has a margin of error.

What Did They Find?

The Score: Their computer got about 56% accuracy in identifying which pattern was being used. That sounds low, but remember, there were 8 different categories to choose from. Random guessing would only get you 11%. So, they were 5 times better than chance.
The Winners: They found that patterns like "Forecasting with Classical Models" and "Multimodal Prompting" were very common.
The Losers: Some patterns, like "Using Tools with LLMs," were hard to spot because they didn't leave a strong "fingerprint" in the code, so the computer often confused them with "nothing special" (the "None" category).
The "None" Problem: A lot of code got labeled as "None" (no pattern detected). This happened because some patterns are subtle, or the code was too messy for the model to recognize.

The Big Takeaway

This paper didn't just say "AI patterns exist." It built a robust method to prove how often they are used in the real world, even when we don't have a perfect dataset to start with.

They showed that:

You can organize a messy list of ideas into a clean taxonomy.
You can teach a computer to find these patterns by only asking humans for help when the computer is confused (Active Learning).
You can estimate how common these patterns are, while honestly admitting the "error bars" on your numbers.

It's like finally having a reliable census of a city's architecture, complete with a note that says, "We are pretty sure about the houses, but the tiny sheds might be a bit fuzzy." This gives developers and researchers a real, data-backed foundation to understand how AI is actually being built today.

Technical Summary: A Methodology for Investigating AI Patterns Prevalence in Software Repositories

Problem Statement

As Artificial Intelligence (AI) applications, particularly those leveraging Large Language Models (LLMs), become ubiquitous, there is a growing need to understand the practical application of AI design patterns. While numerous AI patterns have been proposed in academic literature and industry documentation (e.g., RAG, ReAct, Agent frameworks), their actual prevalence and significance in real-world codebases remain unvalidated. Existing software engineering research often relies on curated benchmarks (like P-MART or DPB) focused on traditional design patterns (e.g., GOF), which lack the necessary data for emerging AI patterns. Furthermore, a significant bottleneck exists: the scarcity of labeled datasets required to train models for detecting niche or emerging patterns. This paper addresses the challenge of empirically validating AI patterns in practice without pre-existing annotated datasets.

Methodology

The authors propose a four-step methodology to identify, detect, and estimate the prevalence of AI patterns in open-source repositories.

1. Literature Mining and Taxonomy Construction

The process begins with extracting 769 AI pattern candidates from 44 published sources, including books, surveys, practitioner articles, and documentation.

Clustering: Pattern descriptions are converted into word embeddings and clustered using the DBSCAN algorithm to group similar candidates.
Refinement: An LLM summarizes each cluster to create refined candidates.
Categorization: Through manual inspection, these candidates are categorized into 14 distinct pattern classes, including RAG, Multi-Agent Architecture, LLM Fine-Tuning, and MLOps.

2. Real-World Code Collection and Chunking

To validate these patterns, the authors curated a dataset of 100 open-source GitHub repositories implementing real-world AI applications. The selection criteria targeted Python repositories with 250–1,000 stars to focus on the "application layer" rather than foundational frameworks.

Code Communities: Instead of analyzing individual files or functions, the authors generated procedure call graphs for each repository.
Louvain Method: They applied the Louvain method to detect "code communities" (tightly coupled groups of methods).
Chunking Strategy: Each code community was treated as a single chunk for embedding generation. This approach was chosen because code within a community is likely to contain complete patterns, addressing the sensitivity of embeddings to chunking boundaries.

3. Active Learning for Model Training

Given the lack of labeled data, the authors employed an active learning approach to build a classifier for the most common pattern classes.

Bootstrapping: An initial model was trained on synthetic data generated by an LLM ( $D_{init}$ ).
Iterative Labeling: The model iteratively selected data points from the unlabeled pool ( $D_{unlabeled}$ ) that exhibited the highest prediction uncertainty (using margin sampling).
Human-in-the-Loop: These uncertain samples were manually labeled by humans. The model was then retrained using 5-fold cross-validation.
Ensemble Modeling: The final model utilized an ensemble of Logistic Regression, SVC, and KNN, weighted by their F1-scores.
Class Handling: To ensure stability, pattern classes with fewer than 5% of the data or fewer than 20 instances were merged into a "None" class during the training iterations.

4. Prevalence Estimation with Error Bounds

To estimate the true frequency of patterns while accounting for model errors and sampling noise, the authors applied:

Matrix Inversion: Using the normalized confusion matrix ( $C$ ) derived from cross-validation, they inverted the relationship $O_{samp} = C \cdot T_D$ to estimate the true distribution ( $T_D$ ) from the observed distribution ( $O_{samp}$ ).
Monte Carlo Simulation: To quantify uncertainty, they simulated sampling errors by drawing from a multinomial distribution and performed 20,000 iterations to calculate 95% confidence intervals for the prevalence estimates.

Key Contributions

Robust Methodology for Data-Scarce Domains: The paper proposes a framework combining active learning and human-in-the-loop annotations to detect patterns without pre-existing annotated datasets, addressing a critical bottleneck in software engineering research.
Novel Chunking Technique: The authors introduce a method using the Louvain algorithm on call graphs to identify "code communities" as chunks, optimizing embedding generation for pattern detection in complex, real-world code.
Prevalence Estimation with Bounds: The application of matrix inversion and Monte Carlo simulations provides usable error bounds for pattern frequency estimates, allowing researchers to assess the reliability of their findings.
Empirical Validation: The study synthesizes 769 candidates into a taxonomy of 14 classes and provides one of the first empirical validations of AI pattern prevalence by analyzing 100 open-source repositories.

Results

The study analyzed 2,442 code communities across 100 repositories.

Classification Performance: The final model achieved 56% accuracy and 55% recall in an 8-way classification task (including a "None" class). This significantly outperforms the 11% random-chance baseline.
Pattern Prevalence: The analysis confirmed the presence of several pattern classes. For example, "Forecasting with Classical Models" and "Multimodal Generative Prompting" were among the most frequent.
Error Bounds: Prevalence estimation provided usable confidence intervals for 5 out of 7 analyzed classes. Classes with fewer training samples (e.g., "Tool Use with LLMs") exhibited wider confidence intervals or zero bounds, indicating higher uncertainty due to misclassification (often confused with the "None" class).
Comparison to Prior Work: While Pandey et al. [23] achieved an F1-score of 0.91 on curated datasets (P-MART), this study's lower performance is attributed to the complexity of real-life code versus clean, curated segments.

Significance and Claims

The paper claims to provide a robust foundation for understanding how AI patterns are used in practice, a field currently lacking empirical data.

Practical Utility: The methodology allows researchers to confirm the wider availability of specific pattern classes and understand their relative frequency. For instance, the results suggest that "Forecasting with Classical Models" and "Multimodal Prompting" are significantly more common than "Agent Architecture" in the analyzed repositories.
Reliability Assessment: The proposed error bounds allow users to identify when estimates are unreliable (e.g., for rare classes) and determine when further iterations of active learning are necessary to improve precision.
Scalability: The authors assert that their approach is highly independent of specific AI patterns and can be extended to other domains (e.g., microservices, quantum computing) or used to detect multiple patterns within a single code community via probability distributions.

The paper concludes that while current accuracy levels are insufficient for identifying specific patterns in arbitrary code segments, the methodology successfully bridges the gap between theoretical pattern proposals and empirical reality, offering a path toward data-driven software engineering for AI.