Technical Summary: Public Machine Learning Solver Framework for Novices

Problem Statement

Machine learning (ML) is a critical tool for knowledge discovery across sectors such as finance, healthcare, and marketing. However, a significant shortage of ML experts creates a bottleneck for business professionals seeking to leverage data. The process of consulting experts is often lengthy, and even for experts, identifying the optimal solution for specific requirements can be time-consuming.

Existing tools to assist non-experts fall into three primary categories, each with limitations:

1. AutoML Systems: While fully automated, they often suffer from slow execution, limited task coverage (focusing mostly on classification/regression), narrow evaluation criteria (relying on single metrics like accuracy), lack of transparency (black-box nature), and limited flexibility.
2. Cheat Sheets: These provide static, tree-based guidelines (e.g., from Microsoft or Scikit-learn) but lack scalability, flexibility, and detailed guidance on preprocessing steps.
3. Selection Criteria Systems: These offer interactive selection but are often limited to single algorithm recommendations, rely on the knowledge of a small number of authors, and lack mechanisms to process narrative performance data from research.

The core problem addressed is the need for a system that bridges the gap between expert knowledge and non-expert users, providing a transparent, structured, and semi-automated solution that goes beyond single algorithm recommendations to suggest complete ML pipelines.

Methodology

The authors propose a novel platform, "I Solve My ML Problem" (available at isolvemymlproblem.org), which integrates expert-defined selection criteria with data-driven analysis and transfer learning. The system operates through the following technical components:

1. Data Ingestion and Characterization

Users connect to their own MySQL databases or upload Excel files. The system automatically extracts data characteristics, including:

• Attribute variance and value distribution.
• Presence of missing values.
• Number of features and dataset size.
• Attribute correlations.
• Class imbalance.
  Based on these characteristics, the system automatically infers the learning type (e.g., supervised vs. unsupervised, classification vs. regression).

2. Problem Understanding via Transfer Learning

The system accepts textual descriptions of the user's ML problem. It utilizes Sentence-BERT (a pre-trained deep neural network) to perform semantic similarity matching between the user's problem description and a portfolio of existing processing chains (pipelines). This allows the system to identify a suitable pipeline structure without requiring manual retraining of the network weights.

3. Algorithm Selection and Ranking

The core selection mechanism relies on a selection criteria matrix populated by ML experts via a companion platform (icontributetoml.org).

• Weighting: Domain experts assign weights to criteria (e.g., accuracy, interpretability, training time) and data characteristics (e.g., imbalanced data, high dimensionality).
• Scoring: The system calculates algorithm weights using a Cartesian product (dot product) of the expert weight vector and the algorithm's criterion values from the knowledge base matrix.
• Alternative Measures: The framework also supports Euclidean distance and cosine similarity for ranking, as well as a Naive Bayes approach (currently theoretical) that treats expert responses as training data to calculate posterior probabilities.
• Output: Algorithms are ranked in descending order of relevance.

4. Pipeline Construction and Refinement

The system generates a complete ML pipeline, not just a single algorithm.

• Rule-Based Refinement: A first-order logic engine (implemented using Microsoft's Z3 solver in Python) applies rules to refine the selected pipeline. For instance, if the selected algorithm is ID3 (a decision tree) and the data is continuous, the system automatically recommends a discretization step.
• Visualization: The final pipeline is represented using Business Process Model and Notation (BPMN), providing a clear, structured visualization of the workflow.
• Execution: Currently, the system outputs the pipeline as a string (code/script) for the user to execute independently, as full execution scripts for all referenced algorithms are not yet implemented.

5. Knowledge Base Management

The system is designed to be incremental and open. A companion platform allows ML experts to contribute new algorithms, selection criteria, and behavioral knowledge. A synchronization mechanism ensures the solver platform remains up-to-date with the latest expert knowledge.

Key Contributions

• Semi-Automated Pipeline Recommendation: Unlike existing tools that recommend single algorithms, this framework suggests a complete, tailored ML pipeline including preprocessing, feature engineering, and modeling steps.
• Integration of Transfer Learning and Logic: The system uniquely combines semantic similarity (via Sentence-BERT) for problem mapping with first-order logic for rule-based pipeline refinement.
• Transparency and Control: The decision-making process is explicit. Users can see the ranking of algorithms based on specific criteria and retain control to adjust weights or manually modify the pipeline.
• Dual-Platform Architecture: The separation of the "Solver" (for novices) and "Contributor" (for experts) platforms allows for continuous, community-driven enrichment of the knowledge base.
• Public Accessibility: It is presented as the first free, publicly accessible online framework that systematically operationalizes expert knowledge for structured ML problem solving.

Results

The authors report validation results based on the selection process:

• Algorithm Selection Accuracy: The system was tested against six algorithms (Naive Bayes, KNN, Bayesian Network, Random Forest, Neural Network, and SVM). When the selection criteria values provided by domain experts matched those of ML experts in the knowledge base, the system correctly recommended the intended algorithm 100% of the time across all tested similarity measures (dot product, Euclidean distance, and cosine distance).
• Pipeline Generation: The system successfully generates structured pipeline descriptions (BPMN) tailored to specific problem contexts.
• Limitations: The authors acknowledge that the current performance of pipeline refinement and component selection is constrained by the limited size of the existing expert knowledge base. The system does not yet execute the pipelines directly.

Significance and Claims

The paper claims that this work represents a significant advancement in making machine learning accessible to non-experts. By combining the strengths of cheat sheets (expert guidelines) and selection criteria systems (interactive decision support), the framework offers a structured and transparent alternative to "black box" AutoML systems.

The authors emphasize that their approach:

• Provides instant responses compared to the slow execution of optimization-heavy AutoML.
• Prioritizes explainability, allowing users to understand why a specific algorithm or pipeline was chosen.
• Is scalable, allowing for the seamless integration of new algorithms and criteria without redeployment.
• Addresses the expert shortage by operationalizing collective expert knowledge into a usable tool for business professionals.

The paper concludes by outlining future work, which includes implementing the proposed optimizer-based AutoML system, further enriching the knowledge base, enabling direct pipeline execution, and conducting broader testing with domain experts.