🤖 machine learning

Multi-stage Dynamic Selection for Cross-Project Defect Prediction

This paper proposes a novel multi-stage dynamic selection framework for Cross-Project Defect Prediction that utilizes project-level and module-level classifier selection to mitigate distribution shifts and outperform state-of-the-art methods across 82 projects.

Original authors: Juscimara G. Avelino, Juscelino S. A. Junior, George D. C. Cavalcanti, Rafael M. O. Cruz

Published 2026-07-23

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Juscimara G. Avelino, Juscelino S. A. Junior, George D. C. Cavalcanti, Rafael M. O. Cruz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you've never seen the crime scene before. You only have a stack of old case files from completely different cities, written by different police forces with different slang and habits. Your job is to predict where the next criminal will strike in your new town. This is the daily struggle of software engineers trying to keep their programs bug-free. They want to find "defects" (bugs) in their code before the software crashes, but many new projects are like blank slates: they haven't been running long enough to build a history of mistakes. So, engineers try to learn from other projects, hoping that what worked in a video game company will help them fix a banking app. This is called Cross-Project Defect Prediction. The problem is, every project is unique; a model that works perfectly on one might fail miserably on another because the "crime scenes" (the code) look so different.

Enter a new team of digital detectives called Multi-DES. Instead of hiring a single detective to solve every case in the new town, or forcing one "super-detective" to try to understand every neighborhood at once, this team uses a clever two-step strategy. First, they audition a massive crowd of different detectives, each with their own unique style and toolkit, to see which group of experts works best together when looking at old case files. Then, when a new case arrives, they don't just pick one detective. Instead, they look at the specific details of that new case and instantly call in the best expert for that specific situation. It's like having a team of specialists where a traffic cop handles a car accident, a forensic accountant handles a fraud case, and a negotiator handles a hostage situation, all chosen on the fly. The researchers found that this "right expert for the right moment" approach is much better at finding bugs in new, unknown projects than the old methods that tried to use the same single solution for everything.

The Detective Agency: How Multi-DES Works

In the world of software, a "defect" is a bug—a mistake in the code that could cause the program to crash or act weirdly. Predicting these bugs is crucial because finding them early saves time and money. But here's the catch: to teach a computer how to spot a bug, you usually need a lot of past examples of bugs. New projects don't have these examples yet. So, engineers try to borrow knowledge from other, older projects. This is the "Cross-Project" part.

However, there's a big hurdle: Distribution Shift. Think of this like trying to learn how to drive in a country where everyone drives on the left, using only a manual written for a country where everyone drives on the right. The rules are similar, but the details are flipped. In software, one project might use a specific coding style, while another uses a completely different one. Traditional methods try to build one giant model that tries to understand all these differences at once. The authors of this paper argue that this is like trying to use a single, generic map for every city in the world; it's too broad and misses the local streets.

The paper proposes Multi-DES (Multi-stage Dynamic Ensemble Selection), which is a bit like a smart, adaptive hiring agency for software detectives. It operates in two main stages:

Stage 1: The Big Audition (Project Level)
Before the system ever sees the new project, it goes through a massive "overproduction" phase. Imagine a casting call where they try out every possible combination of:

Base Classifiers: Different types of algorithms (like Decision Trees, Random Forests, etc.). Think of these as different types of detectives (the observant one, the logical one, the pattern-spotter).
Dynamic Selection Techniques: Different ways to decide which detective to trust.
Pool Sizes: How many detectives are in the room.

They test all these combinations (4 base algorithms × 8 selection techniques × 10 pool sizes = 320 different configurations) on a set of "training" projects. But they don't just pick the one that got the highest score on a single test. Instead, they use a strategy called Aggregate Rank Minimization (ARM).

The ARM Strategy: The "All-Rounder" Judge
Imagine a talent show where you have to pick a winner based on singing, dancing, and acting. If you only pick the person with the best singing voice, they might be terrible at acting. ARM is like a judge who ranks every contestant on all three skills, then adds up their ranks to find the person who is the most consistent all-rounder. The paper suggests that by looking at multiple performance metrics (like F1-score, AUC, and False Alarm) together, the system finds a configuration that is robust and won't fail when the new project looks different from the old ones.

Stage 2: The On-the-Fly Selection (Module Level)
Once the best "audition" configuration is chosen, the system is ready for the new project. But here is the magic: it doesn't just apply one model to the whole project. Software is made of many "modules" (like individual rooms in a house or chapters in a book).

When the system looks at a specific module in the new project, it asks: "Which of our trained detectives is best at spotting bugs in this specific kind of code?" It dynamically selects the most competent classifiers for that specific piece of code. If a module looks like a banking app, it picks the "finance expert" from its pool. If another module looks like a game engine, it picks the "graphics expert." This happens in real-time, for every single piece of code.

What They Found

The researchers tested this idea on 82 software projects from four different public datasets (PROMISE, RELINK, NASA, and AEEEM). They used a strict testing method called "leave-one-project-out," meaning they trained on 81 projects and tried to predict the bugs in the 82nd, then repeated this for every project.

The results were quite promising:

Better than the Best: Multi-DES outperformed or matched the top existing methods in most scenarios. Specifically, it achieved the best results for the AUC (a measure of how well the model distinguishes between buggy and clean code) and False Alarm (how often it cries wolf) metrics on most datasets.
The Numbers: On the AEEEM dataset, Multi-DES scored an AUC of 0.755, beating the next best method (EASC-NB) which scored 0.692. On the NASA dataset, it scored 0.737 compared to 0.666.
Robustness: The system was particularly good at keeping "False Alarms" low, meaning it didn't waste engineers' time checking code that was actually fine.
No Cheating: Crucially, the system did this without looking at any data from the new target project during the training phase. It relied entirely on the old projects, proving that you don't need to peek at the new project's secrets to build a good predictor.

What They Ruled Out

The paper explicitly argues against the idea that a single, static model (one fixed set of rules applied to the whole project) is the best solution. They show that because different parts of a software project have different characteristics, a "one-size-fits-all" approach fails to generalize well when the new project is different from the training data. They also ruled out the idea that you need to know the target project's data distribution beforehand to make good predictions; their method works even when the target project is a complete mystery.

How Sure Are They?

The authors are confident in their findings based on the data they collected. They didn't just simulate a few scenarios; they ran extensive experiments on 82 real-world projects using standard, widely accepted metrics. They used statistical tests (the Wilcoxon Signed-Rank test) to confirm that their results weren't just luck. The paper states that Multi-DES is "statistically superior" in most pairwise comparisons, particularly for AUC and False Alarm. However, they do note a small exception: on the PROMISE dataset, their method wasn't the absolute best for the "False Alarm" metric, showing that while the method is strong, it isn't a magic bullet that wins every single time in every single scenario.

In short, Multi-DES suggests that the best way to predict bugs in a new, unknown project is to have a diverse team of experts ready to be called upon, and to pick the right expert for the specific job at hand, rather than trying to force one generalist to do it all.

Technical Summary: Multi-stage Dynamic Selection for Cross-Project Defect Prediction

Problem Statement
Software Defect Prediction (SDP) aims to identify defect-prone modules to support resource allocation. However, accurate models typically require sufficient historical defect data, which is often unavailable for new projects or those lacking detailed defect histories. Cross-Project Defect Prediction (CPDP) addresses this by training models on external "source" projects and applying them to a "target" project. The primary challenge in CPDP is the distribution shift between source and target projects, caused by differences in coding standards, development practices, and data collection procedures. Traditional CPDP methods often rely on global modeling assumptions or require information from the target project to estimate shifts, which limits their robustness, particularly when dealing with entirely new projects or scenarios with very limited data. Furthermore, existing approaches often apply a single static model or ensemble to the entire target project, failing to account for the fact that classifier competence varies across different regions of the feature space.

Methodology: Multi-DES Framework
The paper proposes Multi-DES (Multi-stage Dynamic Ensemble Selection), a framework designed to improve robustness under distribution shifts by performing model selection at two distinct levels: the project level and the module (instance) level. Crucially, Multi-DES operates as a "strict" cross-project predictor, relying exclusively on source project data and requiring no information from the target project during training.

The framework consists of three phases:

Overproduction:
- A Cartesian product of configurations is generated using a set of base learning algorithms ( $C_{list}$ ), dynamic selection techniques ( $\lambda_{list}$ ), and pool sizes ( $M$ ).
- Classifier pools are created using Bagging to promote diversity.
- These configurations are evaluated on a validation set ( $V$ ) composed of source projects to assess their performance across multiple metrics (e.g., F1-score, AUC, False Alarm).
Best Configuration Selection:
- Instead of selecting a configuration based on a single metric, the authors propose an Aggregate Rank Minimization (ARM) strategy.
- For each performance metric, configurations are ranked (1 being the best). In cases of ties, a mid-rank strategy is used.
- The average rank across all metrics is computed for each configuration.
- The configuration with the lowest average rank is selected, as it represents the best overall trade-off and highest expected generalization ability.
Prediction (Instance-Level Selection):
- The selected configuration (a specific dynamic selection strategy $\lambda^*$ and a pool of classifiers $P^*$ ) is applied to the target project.
- For each new software module (instance) in the target project, a Region of Competence (RoC) is defined based on the validation set.
- The system dynamically selects the most competent subset of classifiers from $P^*$ relevant to that specific instance and combines their predictions. This allows the system to adapt to local characteristics of the target project without assuming a global distribution.

Key Contributions

Multi-DES Framework: A novel CPDP framework that utilizes a two-stage selection scheme (project-level configuration selection and instance-level dynamic selection) to handle distribution shifts without target project information.
Aggregate Rank Minimization (ARM): A configuration selection strategy that aggregates rankings across multiple performance metrics to improve generalization, avoiding the overfitting risks associated with single-metric optimization.
Comprehensive Empirical Evaluation: An extensive evaluation using 82 software projects across four benchmark datasets (PROMISE, RELINK, NASA, AEEEM), comparing Multi-DES against state-of-the-art CPDP methods and various single-metric selection strategies.

Experimental Results

Configuration Diversity: The study found that no single base classifier, dynamic selection technique, or pool size was universally optimal. Random Forest was selected most frequently (69/82 times), but the best choice varied by project, validating the need for a search procedure.
ARM Effectiveness: The ARM strategy demonstrated superior or comparable performance to single-metric selection strategies (optimizing only F1, AUC, or False Alarm) across most datasets. It was particularly effective in identifying configurations that generalized well to unseen projects.
Performance vs. State-of-the-Art: Multi-DES achieved the best or second-best performance in most scenarios across the four datasets.
- It outperformed competing methods (including CamargoCruz09-DT, Turhan09-DT, Menzies11-RF, Watanabe08-DT, and EASC-NB) significantly in terms of AUC and False Alarm metrics.
- On the AEEEM dataset, Multi-DES achieved the best results for all metrics.
- Statistical significance tests (Wilcoxon Signed-Rank) confirmed that Multi-DES is statistically superior in most pairwise comparisons, particularly for AUC and False Alarm.
- Notably, Multi-DES achieved these results without using any information from the target project distribution, avoiding potential data leakage issues present in other approaches.

Significance and Claims
The paper claims that Multi-DES addresses the limitations of global modeling in CPDP by treating the problem as a dynamic system composed of multiple local regions with distinct characteristics. By selecting models at the instance level and using a multi-criteria configuration selection strategy (ARM), the framework is more robust to distribution shifts. The authors emphasize that Multi-DES is particularly suitable for predicting defects in new projects with limited data, as it relies solely on source data and does not make assumptions about the target project's distribution. The work suggests that future research could focus on reducing computational costs, extending the framework to online scenarios, and exploring other defect prediction settings.

The Detective Agency: How Multi-DES Works

What They Found

What They Ruled Out

How Sure Are They?

Technical Summary: Multi-stage Dynamic Selection for Cross-Project Defect Prediction

More like this