Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods

Technical Summary: Enhancing Automated Machine Learning via Homogeneous Train–Test Splitting Methods

Problem Statement
Accurate model evaluation in machine learning (ML) and Automated Machine Learning (AutoML) relies fundamentally on the assumption that training and testing subsets share the same underlying data distribution. Standard random splitting often violates this assumption in the presence of class imbalance, natural clustering, or spatial autocorrelation, leading to distributional shifts. In AutoML pipelines, where algorithms and hyperparameters are automatically selected, biased or inconsistent data partitioning can cause the system to overfit to training peculiarities, yielding deceptively strong evaluation metrics and undermining model generalization. While existing strategies like stratified sampling preserve class proportions and geometry-based methods (e.g., Kennard–Stone, SPXY) maximize feature-space coverage, none explicitly optimize for statistical similarity between the resulting partitions. This gap leaves a critical uncertainty regarding the reliability of performance estimates across diverse tabular datasets.

Methodology
The authors conducted a systematic comparative evaluation across fifteen classification datasets from the UCI Machine Learning Repository, varying in size (150 to 253,680 instances), dimensionality (4 to 169 features), and type (numerical, categorical, mixed). The study evaluated six splitting strategies:

1. Random Splitting: Uniform random assignment (baseline).
2. Stratified Sampling: Preserves target class proportions.
3. Kennard–Stone (KS): Iteratively selects training samples to maximize feature-space coverage.
4. SPXY: Extends KS by incorporating response space diversity.
5. Duplex: Alternately assigns distant sample pairs to training and test sets.
6. Optimised-Distribution (Proposed): A refinement method starting with a stratified split and iteratively swapping samples between partitions to minimize a composite objective function. This function combines Maximum Mean Discrepancy (MMD) with a penalty for deviations in empirical class proportions.

Evaluation Framework
Statistical similarity was quantified using three complementary tests converted to a unified 0–100% scale:

• Chi-Square Test: For categorical distributions.
• Kolmogorov–Smirnov (KS) Test: For continuous distributions.
• Maximum Mean Discrepancy (MMD): A kernel-based test for multivariate, mixed-type data using a combined RBF/delta kernel.

Downstream performance was assessed using the PyCaret AutoML framework. For each dataset and splitting method, models (including logistic regression, random forest, gradient boosting, SVM, etc.) were trained, tuned, and evaluated on the held-out test set. The primary metric was weighted F1, with accuracy as a secondary metric.

Key Results

• Distributional Similarity: The proposed Optimised-Distribution method achieved the highest mean MMD similarity score (89.0%), significantly outperforming all other strategies. In contrast, geometry-based methods Kennard–Stone and SPXY consistently produced near-zero MMD scores (6.9% and 9.7%, respectively). This indicates that their design to maximize feature-space coverage inherently creates training sets that are distributionally dissimilar from the test sets.
• Downstream Performance: Aggregate performance differences were modest, largely due to a "ceiling effect" on large, easily separable datasets where random splitting already yields high accuracy. However, on performance-sensitive datasets (those with lower baseline performance or complex structures), splitting strategy had a pronounced effect.
  • On difficult datasets (e.g., UCI 264, UCI 891), geometry-based methods sometimes produced highly unstable results, with Kennard–Stone yielding an F1 of 0.701 compared to 0.964 for random/stratified splits on UCI 264.
  • Conversely, on specific datasets (e.g., UCI 519), broader feature coverage from geometry-based methods did improve performance, suggesting a trade-off between distributional similarity and feature-space coverage depending on dataset characteristics.
• Stability: The study found a moderate positive association between statistical similarity and reliable performance estimates. Methods with high MMD similarity (Random, Stratified, Optimised-Distribution) produced convergent performance estimates, whereas low-similarity methods exhibited bidirectional instability (inflated estimates on easy tasks, deflated on hard ones).

Key Contributions

1. Systematic Evaluation: Provided a unified distributional similarity framework comparing five established splitting strategies across fifteen diverse UCI datasets using Chi-square, KS, and MMD tests.
2. Empirical Evidence: Demonstrated that geometry-based methods (KS, SPXY), while effective for coverage, consistently underperform random and stratified splitting regarding multivariate distributional similarity, leading to potential evaluation instability.
3. Proposed Method: Introduced the Optimised-Distribution method, which explicitly treats statistical similarity as an optimization objective via iterative sample swapping, achieving the highest similarity scores across the benchmark suite.

Significance and Claims
The paper argues that train–test splitting is a methodological choice with meaningful consequences for the reliability and reproducibility of ML evaluation. The authors claim that improved statistical similarity translates into more reliable performance estimates primarily in "performance-sensitive" regimes—specifically on structurally complex, small-sample, or low instance-to-feature ratio datasets (common in clinical and biomedical applications).

In contrast, for large or easily separable datasets, the choice of splitting strategy is less consequential because large sample sizes naturally reduce distributional differences between partitions. The authors modestly conclude that while their proposed method achieves superior similarity, its practical benefit is most pronounced where data availability is limited and sampling variability is high. They note that the work is limited to classification tasks with a single random seed and suggest future work should explore multiple seeds, regression settings, and scalable similarity measures to address computational costs.