Bug Severity Prediction in Software Projects Using Supervised Machine Learning Models

This study evaluates various supervised machine learning models, including ensemble trees and transformer-based approaches, on Eclipse Bugzilla data to predict bug severity, finding that while ensemble and DistilBERT models achieve the highest overall accuracy, linear classifiers excel at recalling critical bugs, thereby offering actionable insights for automated bug triage and software quality improvement.

The Gist

Imagine you are the captain of a massive, busy ship (your software project). Every day, hundreds of crew members shout out problems: "The engine is making a weird noise!" "The lifeboat latch is stuck!" "The coffee machine is leaking!"

Some of these problems are critical (the ship might sink if we don't fix the engine now). Others are minor (the coffee machine can wait until lunch).

In the real world of software, these "shouts" are called Bug Reports. In big projects like Eclipse (a huge software tool), there are tens of thousands of these reports piling up every day. Trying to read them all and decide which ones are emergencies is like trying to find a needle in a haystack while wearing blindfolded gloves. It's slow, tiring, and humans often make mistakes because they are tired or biased.

This thesis is about building a smart robot assistant (Machine Learning) that can read these reports instantly and tell the captain: "Captain, ignore the coffee machine. The engine is on fire! Fix that first!"

Here is a simple breakdown of how the author, Nafisha, built this robot and what she found.

---

1. The Goal: Sorting the Noise from the Fire

The main problem is Bug Severity Prediction.

• The Old Way: A human manager reads a report and guesses, "Hmm, this sounds serious," or "This sounds like a typo."
• The New Way: A computer program reads the report, learns from thousands of past examples, and instantly assigns a severity level: Critical, Major, Minor, or Trivial.

2. The Ingredients: The "School of Fish"

To teach the robot, the author needed a giant library of past stories.

• The Dataset: She used the Eclipse Bugzilla database. Think of this as a massive archive of 88,682 past "shouts" from the ship's crew.
• The Problem: The library was unbalanced. There were thousands of "minor" complaints (like a loose screw) but very few "critical" ones (like a sinking hull). If you teach a robot with mostly minor complaints, it will become lazy and think everything is minor.
• The Fix: The author used a technique called SMOTE. Imagine you have a tiny pile of "Critical" cards and a huge pile of "Minor" cards. SMOTE is like a photocopier that creates fake but realistic "Critical" cards to balance the deck, so the robot learns to pay attention to the emergencies.

3. The Contest: Who is the Best Detective?

The author didn't just build one robot; she built 10 different detectives (Machine Learning Models) and put them in a contest to see who could sort the bugs best.

Here are the main contestants:

• The Old School Detectives (Linear Models): Like Logistic Regression and SVM. They are simple, fast, and good at following strict rules.
• The Team Players (Ensemble Trees): Like XGBoost, LightGBM, and CatBoost. Imagine a committee of experts who vote on every decision. They are very powerful and usually very accurate.
• The Super-Reader (DistilBERT): This is a Deep Learning model. Think of it as a detective who has read every book in the library and understands the nuance of language. It doesn't just look for keywords; it understands the feeling of the sentence.

4. The Results: Who Won?

After running the contest, here is what happened:

• The Overall Champions (Accuracy): DistilBERT and XGBoost won the race for general accuracy. They were the best at getting the right answer most of the time.
  • Analogy: DistilBERT is like a genius who understands the whole story, while XGBoost is like a super-efficient committee that never misses a detail.
• The "Safety First" Champion (Recall): Logistic Regression was the best at finding the most dangerous bugs, even if it sometimes cried "Wolf!" a little too often.
  • Analogy: If you are looking for a shark in the ocean, you want a detector that screams "SHARK!" even if it's just a shadow. You don't want it to miss the real shark. Logistic Regression is that paranoid, safety-first detector.

5. The Big Takeaway

The study found that there is no single "perfect" robot. It depends on what you need:

• If you want pure accuracy (getting the most right answers overall), use the Team Players (XGBoost) or the Super-Reader (DistilBERT).
• If you want to make sure you never miss a critical disaster (even if you get a few false alarms), use the Old School Detective (Logistic Regression).

6. Why Does This Matter to You?

You might not be a software engineer, but this matters because:

• Safety: It helps prevent apps from crashing or banking systems from failing.
• Speed: It saves developers hours of reading boring reports, so they can fix the real problems faster.
• Trust: When software works better and crashes less, you have more trust in the apps you use every day.

In a Nutshell

This paper is about teaching computers to be better at prioritizing emergencies. By using smart algorithms, we can turn a chaotic pile of complaints into a clear, organized to-do list, ensuring that the most dangerous fires are put out before they burn the whole house down.

Technical Summary

1. Problem Statement

In large-scale software projects, bug repositories (such as Eclipse Bugzilla) grow rapidly, creating a bottleneck for manual triage. Developers must manually classify bug severity (e.g., Critical, Major, Minor, Trivial) to prioritize fixes. This manual process is:

• Laborious and Time-Consuming: Thousands of reports require human attention.
• Inconsistent: Severity assignment varies based on individual expertise, workload, and project context.
• Prone to Bias: Human judgment can lead to misclassification, causing critical bugs to be overlooked or low-priority bugs to consume excessive resources.
• Imbalanced: Datasets typically suffer from severe class imbalance, where "Critical" bugs are rare compared to "Trivial" ones, making it difficult for models to learn the minority class.

The core problem is the lack of an automated, consistent, and accurate system to predict bug severity levels using historical data to optimize resource allocation and software quality assurance.

2. Methodology

The study employs a supervised machine learning approach to classify bug severity using the Eclipse Bugzilla dataset (88,682 bug reports).

A. Data Preprocessing

• Cleaning: Handling missing values (mean imputation for numerical, mode for categorical).
• Feature Engineering:
  • Text Processing: Tokenization, lowercasing, and padding for transformer models. For traditional models, TF-IDF (Term Frequency-Inverse Document Frequency) and n-grams were used to vectorize the "Short_Description" text.
  • Encoding: Label encoding for ordinal categories and One-Hot encoding for nominal features (e.g., Project, Priority).
  • Scaling: Standardization (Mean=0, Std=1) for numerical features.
• Class Imbalance Correction: The SMOTE (Synthetic Minority Over-sampling Technique) was applied to balance the dataset, specifically increasing the representation of high-severity bugs to prevent model bias toward the majority class.
• Data Split: 80% for training and 20% for testing.

B. Models Evaluated

The research compared 10 supervised learning algorithms across different categories:

1. Linear Learners: Logistic Regression, Linear SVM, Passive Aggressive Classifier.
2. Stochastic Learners: SGD (Stochastic Gradient Descent).
3. Tree-Based Ensembles: XGBoost, LightGBM, CatBoost.
4. Bayesian Learners: Naive Bayes.
5. Distance-Based: K-Nearest Neighbors (KNN).
6. Deep Learning (Transformer): DistilBERT (a distilled version of BERT optimized for text classification).

C. Evaluation Metrics

Performance was assessed using:

• Accuracy: Overall correctness.
• Precision: Ability to avoid false positives (crucial for not wasting time on non-critical bugs).
• Recall: Ability to detect true positives (crucial for not missing critical bugs).
• F1-Score: Harmonic mean of Precision and Recall.
• AUC-ROC: Area under the Receiver Operating Characteristic curve.
• Confusion Matrices: To visualize True/False Positives and Negatives.
• Cross-Validation: 3-fold cross-validation was used to ensure robustness.

3. Key Results

The experiments yielded distinct performance profiles for different model families:

• Highest Overall Accuracy:

  • DistilBERT achieved the highest accuracy (90.48%), followed closely by XGBoost (90.41%) and CatBoost (90.17%).
  • This indicates that transformer-based models (understanding complex textual context) and gradient-boosted trees are superior for general bug severity prediction.

• Best for High-Severity Detection (Recall & F1-Score):

  • Logistic Regression achieved the highest Recall (0.627) and F1-Score (0.517) for the high-severity class.
  • While its overall accuracy was lower (85.07%), it was the most effective at identifying critical bugs, minimizing the risk of missing them (False Negatives).

• Best Precision:

  • Naive Bayes achieved the highest Precision (0.845), meaning when it predicted a bug was high severity, it was correct 84.5% of the time, resulting in the fewest false alarms.

• Generalization:

  • Tree-based ensembles (XGBoost, LightGBM) and DistilBERT showed stable training vs. validation accuracy, indicating good generalization and resistance to overfitting.

4. Key Contributions

1. Comparative Analysis: The study provides a comprehensive benchmark of 10 diverse algorithms, ranging from traditional linear models to modern deep learning transformers, specifically on the Eclipse Bugzilla dataset.
2. Trade-off Identification: It explicitly identifies the trade-off between Accuracy (favored by DistilBERT/XGBoost) and Recall for Critical Bugs (favored by Logistic Regression). This helps practitioners choose the right model based on project priorities (e.g., safety-critical systems vs. general maintenance).
3. Preprocessing Strategy Validation: The research validates the necessity of SMOTE for handling class imbalance and TF-IDF/Transformer embeddings for extracting semantic meaning from unstructured bug reports.
4. Actionable Insights: It offers a decision framework: use Logistic Regression if the goal is to catch every critical bug (high recall), and use DistilBERT/XGBoost if the goal is overall classification accuracy and precision.

5. Significance and Impact

• Software Quality Assurance: By automating severity prediction, teams can prioritize critical fixes immediately, reducing system downtime and security risks.
• Resource Optimization: Automated triage reduces the manual effort required for bug management, allowing developers to focus on coding and fixing rather than administrative sorting.
• Scalability: The proposed models can scale to handle the massive influx of bug reports in large open-source and enterprise projects where manual triage is impossible.
• Future Directions: The study suggests future work in Hybrid Models (combining interpretability of linear models with deep learning power), Real-time Integration with tools like JIRA/Bugzilla, and improving Model Explainability (XAI) to build trust in deep learning predictions.

In conclusion, the thesis demonstrates that while DistilBERT and XGBoost offer the best overall predictive power, Logistic Regression remains a vital tool for scenarios where missing a critical bug is unacceptable. The integration of these ML models into software development lifecycles represents a significant step toward more efficient and reliable software maintenance.