A Reproducible Log-Driven AutoML Framework for Interpretable Pipeline Optimization in Healthcare Risk Prediction

Technical Summary: A Reproducible Log-Driven AutoML Framework for Interpretable Pipeline Optimization in Healthcare Risk Prediction

Problem Statement
Accurate and reproducible disease risk prediction in healthcare is hindered by heterogeneous features, limited sample sizes, and severe class imbalance. While Automated Machine Learning (AutoML) has emerged to reduce manual effort, existing frameworks (e.g., Auto-sklearn, TPOT, AutoGluon) predominantly focus on model selection and hyperparameter tuning. They often treat preprocessing steps—such as feature selection, normalization, and data augmentation—as fixed or loosely optimized components. Consequently, the complex interactions between preprocessing strategies and learning algorithms remain underexplored, particularly in small, imbalanced clinical datasets. Furthermore, the stochastic nature of many AutoML search processes limits interpretability, reproducibility, and the ability to systematically analyze pipeline behavior.

Methodology: The yvsoucom-iterkit Framework
The authors propose yvsoucom-iterkit, a deterministic, log-driven AutoML framework that redefines pipeline optimization as a fully reproducible, configuration-level system. Unlike stochastic search methods, this framework employs deterministic enumeration to explore a unified combinatorial search space ($\mathcal{P}$) defined by the Cartesian product of ten configurable components:

1. Feature selection strategy ($F$) and count ($K$)
2. Normalization method ($N$) and order ($O$)
3. Data augmentation method ($A$)
4. Imbalance handling method ($B$)
5. Classification model ($M$)
6. Train/test split ratio ($S$)
7. Decision threshold ($T$)
8. Random seed ($R$)

Key Architectural Features:

• Log-Driven Execution: Every pipeline configuration is encoded as a traceable log entity. The system generates structured logs for all experimental details, including configurations, preprocessing steps, model parameters, intermediate outputs, and evaluation metrics. This ensures full traceability from configuration to result.
• Deterministic Enumeration: The framework pre-enumerates all valid pipeline branches, enabling systematic and exhaustive (or controlled) exploration of the search space without reliance on heuristic or probabilistic search.
• Multi-Level Analysis: The framework supports a hierarchical analytical approach:
  • Branch-Level: Analyzing global metric distributions and clustering.
  • Pipeline-Level: Identifying top-performing configurations.
  • Component-Level: Aggregating performance statistics (mean, variance) for individual component values to isolate their effects.
  • Cross-Component Interaction: Analyzing joint distributions to reveal dependencies between components.
  • Robustness: Evaluating stability across multiple random seeds using non-parametric tests (Friedman, Wilcoxon) and Normalized Robust Rank Scores (NRRS).
• Similarity Analysis: A value-level Root-Mean-Square (RMS) difference analysis quantifies functional redundancy between different component values (e.g., comparing "MixUp" vs. "No Augmentation").

Experimental Setup
The framework was evaluated on two representative healthcare datasets:

1. Pima Indians Diabetes: A small-scale, moderately imbalanced dataset with purely numerical features.
2. Healthcare Stroke Dataset: A more complex dataset with mixed numerical and categorical features and severe class imbalance.

The experiments involved the execution of 18,720 pipeline configurations (grouped into 3,120 data-branch collections) across multiple random seeds to assess performance and robustness.

Key Results

• Structured and Redundant Search Space: The evaluation revealed that the AutoML search space is highly structured and partially redundant. Many distinct configurations yield near-equivalent performance, suggesting a many-to-one mapping where the effective dimensionality is significantly lower than the combinatorial space.
• Component Importance:
  • On the Pima dataset, performance is primarily driven by Data Augmentation (Random Forest importance: 0.454 for Macro-F1), followed by Model Choice (0.198) and Imbalance Handling (0.101).
  • On the Stroke dataset, Imbalance Handling dominates (0.406), followed by Normalization Order and Model Choice.
• Component Similarity and Redundancy:
  • Feature selection variants (biMax vs. biMean) showed low RMS distance (0.0252), indicating interchangeability.
  • MixUp augmentation closely matched "No Augmentation" (RMS 0.0279), while Gaussian noise showed greater divergence (0.10).
  • Tomek Links for imbalance handling aligned closely with "No Imbalance Handling" (RMS 0.0325).
• Performance and Robustness Trade-off:
  • Ensemble models (Random Forest, XGBoost, Gradient Boosting) achieved the best balance between accuracy and stability. On Pima, they reached a Macro-F1 of ~0.88 with low variability ($\sigma \approx 0.023–0.026$).
  • SVM achieved high peak performance (Macro-F1 $\approx 0.689$ on Pima) but exhibited the highest variability ($\sigma \approx 0.029$), indicating sensitivity to stochastic perturbations.
  • Stroke Dataset Challenges: While Weighted-F1 was high (0.94), Macro-F1 remained low (0.67) due to severe class imbalance, highlighting the difficulty of minority class prediction in this domain.

Significance and Claims
The paper claims that yvsoucom-iterkit shifts AutoML from a "black-box" optimization problem to a transparent, analyzable experimental process. By treating pipeline configurations as deterministic, log-traceable entities, the framework enables:

1. Systematic Exploration: Exhaustive coverage of the interaction space between preprocessing and modeling, which is often overlooked in model-centric AutoML.
2. Interpretability: The ability to decompose performance drivers into specific components (e.g., identifying that augmentation strategy is more critical than feature selection count on the Pima dataset).
3. Robustness Evaluation: Moving beyond single-run optimization to statistically grounded evaluation that accounts for stochastic variation, revealing that high-performance models may lack stability.
4. Search Space Reduction: The identification of functional redundancy allows for the pruning of the search space, guiding future AutoML efforts to focus on high-impact components (model choice, imbalance handling) rather than exhaustive tuning of all parameters.

The authors conclude that effective AutoML optimization in healthcare should focus on a reduced set of interacting components, leveraging ensemble methods for stability and prioritizing imbalance handling strategies for datasets with severe class skew. The framework provides a foundation for reproducible, structure-aware pipeline optimization that bridges empirical practice with theoretical considerations of bias-variance trade-offs.