💻 computer science

Improving Test Case Prioritization in Continuous Integration Environments Using Multilayer Perceptron: An Empirical Comparative Study

This empirical study demonstrates that a Multilayer Perceptron (MLP) significantly outperforms five classical machine learning algorithms in prioritizing test cases within continuous integration environments, achieving superior fault detection efficiency and statistical significance across multiple industrial datasets.

Original authors: MUSA ISYAKU, ABDULSALAM YA'U GITAL, HARUNA CHIROMA

Published 2026-09-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: MUSA ISYAKU, ABDULSALAM YA'U GITAL, HARUNA CHIROMA

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, fast-moving world of modern software development, programs are updated constantly. To keep these systems reliable, engineers run automated checks called regression tests every time a change is made. Imagine a massive library where every book must be checked for damage every time a new page is added. If a librarian had to check every single book in order, the process would take forever, and the library would fall behind. In software, this means waiting too long to find a broken feature, which can be costly and frustrating. To solve this, researchers use a strategy called test case prioritization. Instead of running tests in a random order, they try to figure out which specific tests are most likely to fail first. By running the most suspicious tests at the very beginning, teams can find and fix problems much faster, saving time and resources.

For years, computer scientists have tried to use machine learning to help with this sorting task. These are computer programs that learn from past data to make predictions. However, a recent study from Abubakar Tafawa Balewa University in Nigeria suggests that while many methods exist, one specific type of learning model might be the most effective for this job. The researchers, Musa Isyaku, Abdulsalam Ya'u Gital, and Haruna Chiroma, set out to see if a model called a Multilayer Perceptron could outperform other popular methods when sorting through thousands of test cases in a continuous testing environment. They did not just guess; they put the model to the test against five other established algorithms using real-world data from industrial robotics and network technology companies.

The team gathered a massive collection of historical test records from ABB Robotics, a company that builds industrial robots. This dataset contained over 64,000 individual test executions from 106 different testing cycles. In this real-world scenario, most tests pass successfully, while only a small fraction fail. This creates a tricky situation for computer models: if a program simply guesses that every test will pass, it will be right most of the time, but it will completely miss the few tests that actually fail. The researchers needed a model that could spot those rare failures without getting distracted by the overwhelming number of successes. To ensure their findings were not just a fluke, they also tested their approach on three other industrial datasets from different companies, checking to see if the results held up in different environments.

They compared their chosen model, the Multilayer Perceptron, against five other common machine learning techniques: Random Forest, XGBoost, Logistic Regression, Support Vector Machine, and k-Nearest Neighbors. To make the comparison fair, they ran all six models under the exact same conditions, using the same data splits and the same methods for tuning their settings. The goal was to see which model could rank the failing tests highest on the list, ensuring they would be run first. The researchers measured success not just by how often the model was right overall, but by how quickly it could identify the failures. They used a specific score called the Average Percentage of Faults Detected, which essentially measures how early in the list the bad tests appear. A higher score means the model is better at catching problems early.

The results were clear and consistent. On the primary dataset from ABB Robotics, the Multilayer Perceptron model achieved an accuracy of 99.77 percent and a score of 86.35 percent on the fault detection metric. While other models performed well, the Multilayer Perceptron consistently ranked the failing tests slightly higher than the competition. The researchers used rigorous statistical tests to confirm that this difference was not due to chance. The analysis showed that the Multilayer Perceptron was statistically superior to all five other models, with a level of confidence that left no room for doubt. Even when they adjusted the settings of the models to see if they could improve them further, the Multilayer Perceptron remained the top performer, while some of the other models actually became slightly worse after tuning.

One of the most important lessons from this study is about how we measure success. The researchers found that looking only at overall accuracy can be misleading in this field. Because most tests pass, a model could have a very high accuracy score simply by ignoring the failures. The study demonstrated that to truly understand how well a prioritization system works, one must look at how well it finds the minority of failing tests and how early it finds them. The Multilayer Perceptron excelled at this, proving that deep learning models can handle the complex, uneven nature of real-world testing data better than traditional methods.

The study also looked at whether this model could work on data from different companies. When tested on datasets from IOFROL, PAINTCONTROL, and Westermo Network Technologies, the Multilayer Perceptron maintained strong performance, showing that it is not just a lucky fit for one specific set of data but a robust tool that can adapt to different industrial contexts. The researchers concluded that for organizations running large-scale automated testing, adopting this specific type of machine learning model could significantly speed up the process of finding bugs. By using a model that learns complex patterns from history, software teams can ensure that their most critical tests are always run first, keeping their systems reliable and their development cycles moving forward without unnecessary delays.

Technical Summary: Improving Test Case Prioritization in Continuous Integration Environments Using Multilayer Perceptron

Problem Statement
Test Case Prioritization (TCP) is a critical optimization technique in Continuous Integration (CI) regression testing, aimed at reordering test cases to detect faults as early as possible. While machine learning (ML) has been increasingly applied to TCP, existing literature exhibits several limitations: a lack of systematic empirical comparisons between deep learning and classical ML models under identical experimental conditions, an over-reliance on classification accuracy (which is misleading in class-imbalanced datasets), and insufficient cross-dataset generalization studies. Industrial regression testing environments often suffer from class imbalance (approximately 80% passing vs. 20% failing test cases), rendering standard accuracy metrics inadequate for evaluating fault detection efficiency.

Methodology
This study employs a supervised binary classification approach to predict the probability of a test case failure ( $y=1$ ) based on historical execution attributes. The research utilizes a unified experimental framework involving six ML models: a Multilayer Perceptron (MLP) and five classical baselines (Random Forest, XGBoost, Logistic Regression, Support Vector Machine, and k-Nearest Neighbors).

Datasets: The primary evaluation uses the ABB Robotics industrial dataset (64,024 test case executions across 106 CI cycles). To assess generalization, three additional industrial datasets (IOFROL, PAINTCONTROL, and Westermo Network Technologies) were included.
Preprocessing: The workflow involved handling missing values, removing non-informative attributes (e.g., Id, Name), log-transforming skewed Duration attributes, and applying StandardScaler. Class imbalance was addressed using balanced class weighting during training.
Model Configuration: All models were implemented within a scikit-learn Pipeline to prevent data leakage. Hyperparameter tuning was conducted using 5-fold GridSearchCV on the training set, optimizing for the F1-score of the minority (fail) class. The optimal MLP configuration identified was two hidden layers (64, 32 neurons), ReLU activation, and a learning rate of 0.001.
Evaluation Metrics: Performance was measured using Accuracy, Precision, Recall, F1-score, and the Average Percentage of Faults Detected (APFD). APFD was prioritized as the primary metric for TCP as it quantifies the speed of fault detection.
Statistical Validation: To address the limitations of a single train-test split, the study generated 1,000 bootstrap resamples of the held-out test set. Pairwise Wilcoxon signed-rank tests were applied to determine statistical significance ( $\alpha=0.05$ ) and effect sizes ( $|r|$ ) between the MLP and baseline models.

Key Results
On the primary ABB dataset, the MLP model demonstrated top-tier performance, though with specific nuances compared to baselines:

Performance: MLP attained 99.77% Accuracy, 100% Precision, 99.15% Recall, and a 99.57% F1-score. While XGBoost achieved a marginally higher default F1-score (0.9960 vs. 0.9957), MLP matched or exceeded baselines in the final tuned configuration. MLP achieved an APFD of 86.35%, tying with Logistic Regression and KNN, and slightly outperforming XGBoost (86.25%) and SVM (86.33%) in the tuned scenario.
Statistical Significance: The Wilcoxon signed-rank tests confirmed that MLP's superiority over all five baselines was statistically significant ( $p < 0.001$ ) with large effect sizes ( $|r| \geq 0.98$ ). Notably, despite MLP and Random Forest sharing an identical mean APFD of 0.8635 in the bootstrap analysis, the statistical test yielded a significant result favoring MLP, as did the comparison against KNN which also shared the 0.8635 APFD score.
Hyperparameter Sensitivity: While XGBoost showed a marginally higher F1-score with default parameters, its performance degraded after tuning (APFD dropped to 86.25%), whereas MLP remained stable, suggesting MLP is less sensitive to hyperparameter variations on this specific dataset.
Generalization: The MLP demonstrated robustness across the IOFROL and PAINTCONTROL datasets, maintaining high accuracy (approx. 89–94%) and APFD scores (approx. 0.81–0.83), though performance varied on the Westermo dataset due to extreme class imbalance (low recall).

Key Contributions

Empirical Comparison: Provides a rigorous, side-by-side comparison of a deep learning model (MLP) against five classical ML algorithms under strictly identical experimental conditions across four industrial datasets.
Metric Validation: Empirically demonstrates that classification accuracy is an unreliable metric for TCP in imbalanced CI environments, advocating for APFD and F1-score as primary evaluation criteria.
Statistical Rigor: Introduces a robust validation framework using 1,000-iteration bootstrap resampling combined with Wilcoxon signed-rank tests to establish statistical significance in single-split experimental designs, even when mean scores are identical.
Hyperparameter Analysis: Offers empirical evidence on the impact of hyperparameter tuning, showing that while tuning is beneficial, the MLP's optimal configuration (two hidden layers, ReLU) is relatively stable and less complex than the search space maximum.

Significance and Claims
The paper claims that Multilayer Perceptrons can effectively model the complex, non-linear relationships between test case attributes and failure probabilities, outperforming traditional ML techniques in CI-based TCP tasks when evaluated with rigorous statistical methods. The study posits that deep learning models offer a viable path to improving fault detection rates in industrial settings.

Crucially, the authors emphasize that their findings support the deployment of MLP-based TCP as a "drop-in enhancement" to existing CI pipelines, requiring only historical execution data. The study concludes that while accuracy alone is insufficient for evaluating TCP, the combination of high APFD scores and rigorous statistical validation provides strong evidence for the adoption of MLP in regression testing. The authors note that future work should explore online learning, cost-aware prioritization (APFDc), and explainability, but the current study establishes the foundational efficacy of MLP over classical baselines in this domain.

Technical Summary: Improving Test Case Prioritization in Continuous Integration Environments Using Multilayer Perceptron

More like this