Cross-Model Consistency of Feature Importance in Electrospinning: Separating Robust from Model-Dependent Features

Technical Summary: Cross-Model Consistency of Feature Importance in Electrospinning

Problem Statement
Electrospinning is a highly sensitive nanofabrication process where small variations in operating parameters (e.g., solution concentration, voltage, flow rate) significantly dictate fiber morphology and material performance. While machine learning (ML) is increasingly utilized to model these process–structure relationships and infer feature importance, most existing studies rely on a single model. This approach implicitly assumes that the resulting feature importance is reliable and reflective of the underlying physics. However, in the "small-data regime" typical of electrospinning research (often characterized by limited, heterogeneous datasets), different model families may fit the same data with comparable accuracy while learning fundamentally different internal representations. Consequently, feature importance conclusions drawn from a single model may reflect the model's inductive biases rather than true physical relationships, leading to potentially unreliable experimental guidance.

Methodology
To address the lack of systematic evaluation regarding the consistency of feature importance across different models, the authors conducted a cross-model analysis using a curated dataset of 96 polyvinyl alcohol (PVA) electrospinning experiments. The study employed the following methodological framework:

• Dataset: The data, sourced from Ziabari et al. and the Cogni-e-SpinDB, utilized a Response Surface Methodology (RSM) design with four input factors (solution concentration, applied voltage, flow rate, tip-to-collector distance) sampled at four levels each. The target output was the mean fiber diameter.
• Model Diversity: Twenty-one distinct models spanning five families were trained to ensure a broad coverage of inductive biases:
  • Linear: Linear Regression, Ridge, Lasso, Elastic Net.
  • Tree-based: Random Forest, Gradient Boosting, XGBoost, LightGBM, AdaBoost, Decision Tree, Histogram-based Gradient Boosting.
  • Kernel-based: Support Vector Regression (SVR) with RBF and polynomial kernels.
  • Neural Networks: Multi-Layer Perceptrons (MLP) with varying configurations.
  • Instance-based: K-Nearest Neighbors (KNN).
• Feature Importance Quantification: SHAP (SHapley Additive exPlanations) values were computed consistently across all models (using TreeSHAP for tree-based models and KernelSHAP for others) to generate global importance scores.
• Reliability Analysis: A rank-based analysis was performed to quantify inter-model agreement. Key metrics included:
  • Mean rank ($\bar{r}_j$) and standard deviation of rank ($\sigma_{rj}$) for each feature.
  • Spearman rank correlation between model pairs.
  • Top-$k$ agreement analysis to measure consensus on the most influential features.

Key Results
The study revealed a critical distinction between predictive performance and interpretive reliability:

1. Predictive Performance: Tree-based ensembles achieved the highest predictive accuracy ($R^2 \approx 0.9$), followed by kernel-based and instance-based methods. Linear models showed moderate performance, while neural networks exhibited lower and less stable results, consistent with the limitations of small training sets. Crucially, a substantial subset of models from different families achieved comparable accuracy, satisfying the condition necessary to isolate differences in feature importance from differences in predictive quality.
2. Feature Importance Consistency:
   • Solution Concentration: Identified as a fully robust parameter. It was ranked as the most influential feature (Rank 1) by every single model with zero variance ($\sigma_r = 0$). This aligns with domain knowledge regarding viscosity and polymer chain entanglement.
   • Flow Rate and Applied Voltage: These parameters exhibited high variability in their rankings ($\sigma_r > 0.9$). Their apparent importance was strongly model-dependent; for instance, flow rate ranged from moderately important to the least influential depending on the model family.
   • Tip-to-Collector Distance: Showed intermediate reliability ($\sigma_r = 0.526$), with moderate stability but lower mean rank consistency.
3. Inter-Model Agreement: The mean pairwise Spearman rank correlation across all model pairs was 0.584, indicating only moderate overall agreement. While tree-based models showed higher internal correlation, significant divergence was observed between fundamentally different families (e.g., linear vs. neural networks). Top-1 agreement was perfect (1.000), but this dropped to 0.778 for the top-3 features.

Key Contributions

• Empirical Evidence of Model Dependence: The paper provides the first systematic evaluation demonstrating that feature importance conclusions in electrospinning are not universally consistent across model families, even when predictive accuracy is comparable.
• Reliability Spectrum Framework: It introduces a method to distinguish between "robust" features (consistent across models) and "model-dependent" features (sensitive to model choice), offering a principled basis for interpreting ML results in small-data regimes.
• Validation of Domain Knowledge: The study confirms that only solution concentration is robustly identified as the dominant driver, validating physical intuition while highlighting the unreliability of inferring the importance of other parameters from single-model outputs.

Significance and Claims
The authors assert that predictive performance and interpretive reliability are distinct properties of a machine learning model. The central claim is that in small experimental datasets, feature importance derived from a single model carries an unacknowledged risk of being driven by the model's inductive biases rather than the underlying process–structure–property relationships.

The paper concludes that relying on a single model for interpretive insights in electrospinning is risky and can lead to wasted experimental effort. Instead, the authors advocate for cross-model validation as a necessary step to ensure that identified parameters are truly influential. This approach is presented as broadly applicable to other domains where ML is used to infer feature importance from limited data, offering a framework to assess the reliability of interpretive conclusions under model uncertainty. The study does not claim to solve the small-data problem itself but rather provides a method to mitigate the risk of drawing unreliable conclusions from such data.