Improving Requirements Classification with SMOTE-Tomek Preprocessing

Technical Summary: Improving Requirements Classification with SMOTE-Tomek Preprocessing

Problem Statement
Requirements engineering (RE) is critical for translating stakeholder needs into system specifications, yet requirement classification often remains a manual, inconsistent process. While machine learning (ML) offers automation, its application is hindered by severe class imbalance in real-world datasets, specifically the widely used PROMISE dataset. This dataset contains 969 software requirements categorized into 12 distinct types (one functional and 11 non-functional subtypes). The distribution is highly skewed; for instance, the "Portability" category represents only 1.24% of the data, while "Security" comprises 12.9%. This imbalance causes standard ML models to bias toward majority classes, resulting in poor generalization and low performance on minority classes. Furthermore, while deep learning (DL) models have advanced significantly in Natural Language Processing (NLP), their reliance on massive datasets and computational resources makes them less practical for resource-constrained RE projects.

Methodology
This study proposes a hybrid preprocessing strategy combining SMOTE (Synthetic Minority Oversampling Technique) and Tomek Links (SMOTE-Tomek) to address both class imbalance and data noise.

• Data Representation: Unstructured text requirements are converted into numerical vectors using Term Frequency-Inverse Document Frequency (TF-IDF).
• SMOTE-Tomek Process:
  1. Oversampling: SMOTE generates synthetic samples for minority classes via linear interpolation between existing samples and their nearest neighbors, enhancing class diversity without simple duplication.
  2. Cleaning: Tomek Links identify and remove pairs of samples that are mutual nearest neighbors but belong to different classes (typically borderline or noisy samples). This step removes the majority class member of such pairs to improve class separability.
• Validation Framework: To ensure rigorous evaluation and prevent data leakage, the study employs Stratified 10-fold Cross-Validation. Crucially, SMOTE-Tomek resampling is applied only to the training folds (9 folds) in each iteration, leaving the validation fold (1 fold) untouched to preserve the original data distribution.
• Models Evaluated: The study benchmarks a diverse set of classical ML algorithms, including Logistic Regression (LR), Decision Trees, Random Forest, Support Vector Machines (Linear and RBF kernels), Naive Bayes, K-Nearest Neighbors (KNN), Gradient Boosting, AdaBoost, and CatBoost.

Key Contributions

1. Systematic Application: The paper applies SMOTE-Tomek within a stratified K-fold framework specifically designed to keep validation sets free from resampling artifacts, ensuring an unbiased assessment of generalization capabilities.
2. Comparative Evaluation: It provides a comprehensive performance analysis of classical ML models on the imbalanced PROMISE dataset, demonstrating that lightweight models can achieve high efficiency without the computational overhead of deep learning.
3. Interpretability Analysis: By utilizing Logistic Regression, the study analyzes feature coefficients to interpret the relationship between specific words and requirement types, offering insights into how preprocessing stabilizes feature learning.

Results
The integration of SMOTE-Tomek yielded significant improvements across multiple metrics, particularly for Logistic Regression:

• Logistic Regression (LR): Achieved the highest performance, improving mean accuracy from a baseline of 58.31% ± 2.05% to 76.16% ± 2.58%. The Matthews Correlation Coefficient (MCC) increased by approximately 7% (from 0.4181 to 0.6736). Hyperparameter optimization (C=10, L2 regularization, SAGA solver) further refined these results.
• Other Models: Naive Bayes and Gradient Boosting also saw substantial gains, exceeding 70% accuracy. Linear SVM improved from 71.10% to 74.20%.
• Negative Outcomes: Conversely, ensemble methods like AdaBoost and Random Forest performed poorly or degraded significantly after preprocessing (AdaBoost accuracy dropped to 16.10%), suggesting these specific algorithms are highly sensitive to the resampled distribution in this context.
• Feature Stability: Analysis of LR coefficients revealed that the baseline model exhibited extreme, variable weights for minority classes (indicating overfitting to noise). In contrast, the SMOTE-Tomek approach stabilized these coefficients, forcing the model to rely on a more robust and generalized set of lexical features.

Significance and Claims
The paper claims that SMOTE-Tomek provides a robust, scalable, and interpretable solution for requirements classification in imbalanced datasets. It demonstrates that classical ML models, when paired with appropriate preprocessing, can effectively handle the nuances of the PROMISE dataset without requiring the extensive computational resources of deep learning. The study emphasizes that while the results are promising, the small size and limited variety of the PROMISE dataset remain a constraint. The authors position this work as a practical approach for resource-constrained environments and suggest that the methodology is applicable to other domains facing similar data imbalance challenges, such as legal or healthcare document analysis. Future work is proposed to explore larger datasets and hybrid approaches combining lightweight models with deep learning.