Technical Summary: Propheticus Framework for Predictive Models in Software Dependability

Problem Statement

The increasing complexity and size of modern software systems render traditional dependability techniques (e.g., code reviews, inspections, testing) impractical due to scalability and cost constraints. Consequently, software is often deployed with faults that compromise business processes. While Machine Learning (ML) offers a promising alternative for predicting reliability and security issues (e.g., vulnerability prediction, failure prediction), its application in the dependability domain is hindered by significant complexity.

Developing robust ML models requires deep expertise in data preparation, algorithm selection, hyperparameter tuning, and statistical validation. The process is highly error-prone; minor mistakes in experimental design (e.g., data leakage, improper handling of imbalanced data) can invalidate results. Furthermore, existing tools present limitations:

• Enterprise platforms (e.g., Feedzai, DataRobot) are often inaccessible for research.
• GUI-based tools (e.g., Weka) lack flexibility for customization, have steep learning curves (Java-based), and evolve slowly.
• Low-level libraries (e.g., Scikit-learn) require extensive coding and deep ML knowledge to construct a comprehensive, reproducible experimental workflow, increasing the risk of implementation errors.

There is a lack of a commonly accepted, flexible tool within the dependability community that abstracts ML complexity while allowing for rigorous, customizable research workflows.

Methodology

The paper introduces Propheticus, a Python-based ML framework designed specifically to support research in the dependability domain. It adopts a data-centric approach, allowing researchers to focus on the problem rather than implementation details.

Architecture and Workflow

Propheticus is built on object-oriented principles and operates via a Command Line Interface (CLI). Its workflow is modular, encompassing the following stages:

1. Data Management: Loads datasets (structured as feature/target tables with JSON headers) and supports combining multiple sources.
2. Data Analysis: Performs Descriptive (statistical) and Exploratory (graphical) analysis, including heatmaps, scatter plots, and correlation matrices.
3. Data Preprocessing: Handles feature selection, dimensionality reduction (e.g., PCA, RFE), data sampling (e.g., SMOTE, undersampling), and normalization.
4. Classification/Clustering: Executes ML algorithms (e.g., SVM, Random Forest, Neural Networks) with built-in support for:
   • Cross-validation: Default 10-fold stratified cross-validation.
   • Hyperparameter Tuning: Grid-search with nested cross-validation.
   • Reproducibility: Runs experiments multiple times (default 30) with different random seeds.
5. Batch Execution: Allows users to define and run lists of configurations automatically.
6. Result Processing: Generates reports (confusion matrices, ROC curves), reduces results based on thresholds, and performs statistical comparisons (e.g., T-Test, ANOVA) to validate model differences.

Customization and Extensibility

Propheticus is designed to be configurable without modifying core code. Users can define problem-specific configurations via InstanceConfig, InstanceGUI, and InstanceBatchConfiguration files. It utilizes a plugin paradigm where new algorithms or techniques (e.g., from Scikit-learn, Imbalanced-learn) can be integrated by simply declaring them in a configuration dictionary, adhering to standard interfaces.

Case Studies and Results

The paper validates Propheticus through two distinct case studies demonstrating its application in vulnerability prediction and online failure prediction.

Case Study 1: Vulnerability Prediction

• Objective: Predict software vulnerabilities based on software metrics.
• Dataset: 604,304 files from the Mozilla project (highly imbalanced: 2,819 vulnerable vs. 601,485 non-vulnerable).
• Process: The framework facilitated an exploratory analysis revealing high feature correlation and class imbalance. The workflow involved testing various dimensionality reduction techniques (Variance, Correlation, PCA, RFE) and sampling methods (Random, SMOTE, Instance Hardness Threshold).
• Results: Initial models without preprocessing performed poorly (e.g., Decision Trees predicted only 28% of vulnerable samples). By combining Instance Hardness Threshold sampling with Variance/Correlation feature selection and fine-tuning, the Random Forest (RF) algorithm achieved 81% precision (non-vulnerable) and 77% recall (vulnerable). The framework allowed for rapid iteration and comparison of these combinations.

Case Study 2: Online Failure Prediction (OFP)

• Objective: Predict runtime system failures (Hangs and Crashes) using past data and current system state.
• Dataset: Windows XP (SP3) fault-injection data (233 system variables), limited to 50,000 non-virtualized samples.
• Process: The study assessed models across four criticality scenarios (Business-critical, Heightened-critical, Best-effort, Minimum-effort) using different metrics (F2-score, Informedness, F-measure, Markedness).
• Results: The Decision Tree (DT) algorithm performed best for critical scenarios. The framework successfully ranked models based on specific application scenarios. For instance, while undersampling improved failure detection, it increased false positives; the framework helped identify that DT with feature selection was optimal for the "Heightened-critical" scenario.
• Statistical Validation: Propheticus facilitated a paired T-Test (p-value = 0.0215) to statistically confirm that the DT algorithm was significantly different from Bagging for the most demanding scenarios, a task that would require manual implementation in lower-level libraries.

Key Contributions

1. Framework Development: The creation of Propheticus, a specialized ML framework that bridges the gap between high-level GUI tools and low-level coding libraries, specifically tailored for dependability research.
2. Workflow Abstraction: It abstracts complex ML steps (preprocessing, cross-validation, statistical testing) into a unified, configurable CLI, reducing the coding burden and the probability of experimental errors.
3. Extensibility: Unlike rigid tools, Propheticus allows researchers to easily integrate new algorithms and techniques via a plugin architecture without recompiling the core system.
4. Validation: The paper demonstrates that Propheticus can effectively handle diverse problems (binary classification with imbalance, multi-class time-series prediction) and produce results comparable to or better than manual implementations, while significantly expediting the workflow.

Significance and Claims

The authors position Propheticus not as a competitor to general-purpose ML tools, but as a flexible alternative focused on research within the dependability community. The paper claims that:

• Current tools either lack the flexibility for deep research customization (Weka) or require excessive expertise and coding effort (Scikit-learn), leading to limited or flawed experiments.
• Propheticus addresses these limitations by providing a structured workflow that enforces best practices (e.g., proper data splitting, statistical validation) while remaining adaptable to specific research needs.
• By reducing the overhead of implementation, the framework allows researchers to focus on the problem domain (reliability and security) rather than the mechanics of ML, thereby facilitating the development of more dependable systems.

The paper concludes that Propheticus successfully demonstrates its utility in easing and expediting a thorough ML workflow, as evidenced by the two case studies, and serves as a foundation for future expansion in scope and functionality.