Technical Summary: A Feature-Driven Framework for Software Fault Prediction

Problem Statement

Software Fault Prediction (SFP) is a critical component of the software development life cycle (SDLC), aimed at identifying faulty modules early to reduce maintenance costs and improve quality. However, existing approaches often treat feature selection (reducing dimensionality and removing irrelevant/redundant metrics) and hyperparameter tuning (optimizing model configuration) as separate, independent tasks. This separation leaves a gap in achieving optimal model performance, as the interaction between selected features and model parameters is not systematically explored. Furthermore, software datasets are often imbalanced, with faulty modules representing a minority class, complicating the training of robust machine learning (ML) models.

Methodology

The authors propose an integrated framework that systematically combines feature selection and hyperparameter optimization. The workflow consists of seven distinct steps:

1. Data Collection: Utilization of 19 versions of open-source software projects from the PROMISE repository, comprising 21 software metrics (features) and binary fault labels.
2. Preprocessing:
   • Cleaning: Removal of duplicate modules.
   • Imputation: Handling missing values using K-Nearest Neighbors (KNN).
   • Scaling: Application of Min-Max scaling to normalize features between 0 and 1.
3. Data Partitioning: Implementation of 10-fold stratified cross-validation to ensure class distribution is preserved across training and validation sets, addressing dataset imbalance.
4. Class Imbalance Handling: Application of Adaptive Synthetic Sampling (ADASYN) to balance the dataset.
5. Feature Selection: Four distinct methods were evaluated individually:
   • Recursive Feature Elimination (RFE): A wrapper-based iterative method.
   • L1 Regularization: A sparse linear model approach.
   • Mutual Information (MI): A filter-based method capturing non-linear dependencies.
   • Correlation-based Feature Selection (CFS): A filter-based method maximizing feature relevance to the target while minimizing inter-feature redundancy.
6. Model Training & Hyperparameter Optimization: Three baseline ML models were tested: Random Forest (RF), Logistic Regression (LR), and Support Vector Machines (SVM). Each model was optimized using three search strategies:
   • Grid Search (GS): Exhaustive search.
   • Randomized Search (RS): Probabilistic sampling.
   • Genetic Algorithm (GA): Evolutionary optimization.
7. Evaluation: Models were assessed using Accuracy, Precision, Recall, F1-score, and computational efficiency (training/testing time).

Key Contributions

The study provides a comparative analysis of how different combinations of feature selection and hyperparameter tuning affect SFP performance. The primary contributions include:

• Integrated Framework: A unified approach that jointly optimizes feature subsets and model hyperparameters, rather than treating them sequentially or independently.
• Comparative Analysis of Selection Methods: A detailed evaluation of RFE, L1, MI, and CFS, demonstrating that multivariate methods (like CFS) preserve feature interactions better than univariate methods.
• Optimization Strategy Evaluation: An assessment of GS, RS, and GA, highlighting the trade-offs between computational cost and predictive accuracy.
• Model-Specific Insights: Identification of how different models (RF, LR, SVM) respond to specific feature sets and tuning strategies.

Experimental Results

The experiments were conducted on balanced data using 10-fold stratified cross-validation. Key findings include:

• Best Overall Performance: The combination of Correlation-based Feature Selection (CFS) and Genetic Algorithm (GA) optimization yielded the highest performance. Specifically, the Random Forest (RF) model achieved an accuracy of 88.40%, an F1-score of 0.88, precision of 0.89, and recall of 0.87.
• Impact of Feature Selection:
  • CFS was most effective for RF and SVM, preserving multivariate interactions and reducing redundancy.
  • RFE also performed well, particularly with RF and GA (87.41% accuracy).
  • L1 Regularization resulted in lower F1-scores and precision compared to other methods, suggesting it may over-penalize informative features in this context.
  • Mutual Information (MI) showed improvements over L1 but slightly underperformed compared to CFS and RFE.
• Impact of Hyperparameter Tuning:
  • Tuning significantly improved accuracy across all models. For instance, RF accuracy without tuning was 70.21%, rising to 86.32% with Random Search and 82.06% with GA (on L1 data) or 88.40% (on CFS data).
  • GA generally provided the strongest optimization, particularly when paired with CFS, though it incurred higher training times.
  • Randomized Search offered a cost-effective alternative, achieving competitive performance with significantly lower computational expense.
• Model Comparison:
  • Random Forest consistently outperformed Logistic Regression and SVM across all feature selection and optimization strategies.
  • Logistic Regression showed moderate gains with sparse features and lightweight optimization.
  • SVM benefited significantly from kernel tuning.
• Overfitting Mitigation: The baseline models (without feature selection or tuning) exhibited a significant gap between training and testing accuracy (e.g., RF training accuracy 81.52% vs. testing 70.21%), indicating overfitting. The proposed framework reduced cross-validation variability to approximately ±1.0%, demonstrating robustness.

Significance and Claims

The paper claims that the combined application of feature selection and hyperparameter tuning is essential for developing reliable and efficient software fault prediction models. Specifically:

• Synergy: The interaction between feature selection and parameter tuning is critical; optimizing one without the other yields suboptimal results.
• Efficiency vs. Accuracy Trade-off: While GA with CFS offers the highest accuracy, univariate methods like L1 combined with Randomized Search provide a viable, time-efficient alternative for scenarios where computational resources are constrained.
• Robustness: The proposed framework produces models with minimal cross-validation variability, ensuring that performance gains are generalizable rather than artifacts of specific data splits.
• Practical Application: The identification of critical attributes (such as Weighted Methods per Class - WMC, and Coupling Between Objects - CBO) alongside optimized models provides actionable insights for software engineers to focus testing efforts on high-risk modules.

The authors conclude that while their framework demonstrates significant improvements over baseline models, future work should focus on validating these findings across larger datasets and exploring adaptive frameworks for real-time fault prediction.