RobustModelMaker: Coupling Bootstrap Stability Selection with Leakage-Safe Nested Cross-Validation for Scientific Machine Learning

Technical Summary: ROBUSTMODELMAKER

Problem Statement
Scientific machine learning often operates in a regime of small-to-medium datasets (hundreds to tens of thousands of samples) with high-dimensional, correlated features. In this setting, standard machine learning pipelines face two compounding failure modes that threaten reproducibility and scientific validity:

1. Unstable Feature Selection: Single-run feature selection methods produce feature sets that vary substantially under small perturbations of the training data (e.g., bootstrap resampling). This instability renders selected features unreliable as scientific findings, as replication attempts often yield different subsets.
2. Optimistic Bias (Data Leakage): When the same data are used for feature selection, hyperparameter tuning, and performance evaluation, the resulting performance estimates are optimistically biased. This bias is particularly severe in small-sample, high-dimensional regimes where the search space is large relative to the data.

While stability selection and nested cross-validation are known remedies for these issues individually, they are rarely coupled in practice. Standard tooling treats them as separable steps, failing to address the interaction where unstable selection inflates the variance of an already-biased score.

Methodology
The paper introduces RobustModelMaker, a Python framework designed to couple bootstrap stability selection with strict nested cross-validation. The core design principle is that stability selection and nested cross-validation are jointly required, not optional add-ons.

• Coupled Pipeline: The framework executes bootstrap stability selection inside the training partition of each outer fold of a nested cross-validation loop.
  • Inner Loop: For each outer training set, a bootstrap aggregation is performed. Features are selected based on their frequency of selection across $B$ bootstrap resamples (using a median-importance rule).
  • Preprocessing: All preprocessing steps (imputation, scaling) and hyperparameter tuning (via inner cross-validation) are fitted strictly on the training partition of the current outer fold.
  • Evaluation: The resulting feature subset and tuned model are evaluated only on the held-out outer test partition. This ensures the test data never influences feature selection or model building, eliminating optimistic bias.
• Algorithm Agnosticism: The framework supports nine algorithms (including Random Forest, XGBoost, SVM, and linear models) across binary classification, multiclass classification, and regression. It uses a "median-importance" rule for stability selection, making it applicable to any base learner that exposes feature importance.
• Determinism and Leakage Safety: The framework enforces a "leakage safety contract" via code structure rather than convention. It guarantees reproducibility to machine precision under a fixed random seed, with specific handling for parallel execution to maintain determinism.

Key Contributions

1. Framework Implementation: A single-file, scikit-learn-compatible library that implements the coupled pipeline, including a specific procedure for determining specificity-targeted probability cutoffs in binary classification.
2. Joint Score-Stability Frontier: The paper argues that the framework's primary value is not dominating a single metric (either predictive score or stability) but occupying a position on the joint score-stability frontier that alternatives cannot match across all task types.
3. Comprehensive Validation:
   • Test Suite: A deterministic suite covering unit tests, performance budgets, and reproducibility checks.
   • Benchmarking: A statistical benchmark on three real scientific datasets (SECOM Manufacturing, Urban Land Cover, Graphene Oxide Bulk) comparing RobustModelMaker against a full-feature baseline and three alternative selectors: ANOVA F-test, Recursive Feature Elimination with Cross-Validation (RFECV), and Boruta.
   • Real-World Applications: Two case studies demonstrating practical usage: ovarian cancer biomarker discovery (PLCO Trial) and superconductor critical-temperature regression (UCI Superconductivity Data).

Results

• Benchmark Performance: Across the three benchmark datasets, RobustModelMaker was statistically indistinguishable ("preserved") from the full-feature baseline in terms of predictive performance (AUC or RMSE) while significantly reducing the number of features (47–62% reduction).
• Stability Superiority: In terms of selection stability (measured by Jaccard similarity across outer folds), RobustModelMaker consistently outperformed RFECV and Boruta, and was competitive with or superior to ANOVA F-test (which achieved perfect stability only by retaining very small, potentially suboptimal feature sets).
  • SECOM: RobustModelMaker achieved the highest stability (J=0.660) among methods retaining a substantial number of features, whereas RFECV and Boruta were highly unstable (J < 0.28).
  • Urban Land Cover: RobustModelMaker (J=0.801) selected roughly half as many features as Boruta (J=0.895) for comparable stability and score.
  • Graphene Oxide: While filter methods (ANOVA, RFECV) achieved lower RMSE by exploiting univariate dataset structures, RobustModelMaker maintained high stability (J=0.899) with a larger, more reproducible feature set.
• Application Insights:
  • In the PLCO ovarian cancer study, the framework identified a stable 16-biomarker panel (from 42 candidates) with a specific operating point (98% specificity, 30.9% sensitivity), demonstrating the ability to distinguish between redundant assay versions of the same biomarker.
  • In the Superconductivity study, the framework reduced 81 features to 36 stable features. While the full-feature baseline achieved marginally lower RMSE (~1 K difference), the framework explicitly reported this trade-off, highlighting that the stability-selected subset is more robust for future deployment and re-fitting.

Significance and Claims
The paper claims that for scientific machine learning where reproducibility and honest generalization are targets, the coupling of stability selection and nested cross-validation is necessary, not optional.

• Modest Claims on Performance: The authors do not claim RobustModelMaker produces the highest possible predictive score in all conditions. Instead, they claim it produces a feature subset and performance estimate that are "individually defensible and mutually consistent."
• Trade-off Transparency: The framework makes the trade-off between predictive performance and feature stability explicit. It reports that stability-selected models may have modestly lower performance than full-feature baselines but offers a feature set that is robust to resampling.
• Scientific Utility: The framework shifts the paradigm of feature selection from an "emergent property" of model fitting to a "first-class deliverable." It ensures that the selected features are not artifacts of a single data split but are stable findings suitable for downstream scientific interpretation and experimental validation.

The work is positioned as an implementation of established methodological principles (stability selection, nested CV) tailored to the specific constraints of scientific data, rather than a new theoretical contribution. It addresses the "reproducibility crisis" in small-sample scientific ML by providing a tool that prevents the two primary failure modes from interacting to invalidate results.