An Interpretable Statistical Learning Framework for Binary Classification: An Application to Student Stress Prediction

Technical Summary: An Interpretable Statistical Learning Framework for Binary Classification

Problem Statement
Modern statistical learning faces a tension between predictive accuracy and model interpretability. While complex machine learning algorithms (e.g., ensemble methods, deep learning) often achieve high predictive performance, they frequently lack transparency, making it difficult to identify reliable predictors or explain the mechanisms underlying predictions. Conversely, traditional statistical models offer interpretability but may suffer from instability in variable selection due to sampling variability or multicollinearity. The paper addresses the need for a unified framework that simultaneously delivers accurate binary classification, robust feature identification, and transparent model explanations, using student stress prediction as a motivating application.

Methodology
The study proposes an integrated statistical learning framework applied to a publicly available dataset of approximately 25,500 student records. The response variable is binary (Low Stress vs. High Stress). The methodology consists of four primary components:

1. Comparative Modeling: Five classification models were developed and compared:

   • Logistic Regression: Served as the baseline.
   • Penalized Regression: Ridge Regression ($L_2$ penalty), LASSO ($L_1$ penalty), and Elastic Net (combined $L_1$ and $L_2$ penalties) were implemented to handle multicollinearity and perform variable selection.
   • Random Forest: Included as a nonlinear machine learning benchmark to capture complex interactions.
   • Implementation: All models were tuned using repeated 10-fold cross-validation on a 70% training set, with performance evaluated on an independent 30% test set.

2. Performance Evaluation: Models were assessed using Accuracy, Sensitivity, Specificity, F1-score, ROC-AUC, and Brier Score to evaluate discrimination and calibration.

3. Bootstrap Stability Selection: To address the instability of variable selection in single-model fits, the study employed bootstrap stability selection (500 resamples) using the LASSO model. Predictors were evaluated based on their selection frequency across resamples, with frequencies $\ge 0.80$ classified as stable.

4. Explainable Machine Learning (SHAP): Shapley Additive Explanations (SHAP) were applied to the Random Forest model to quantify the marginal contribution of each predictor. This provided both global feature importance and local explanations, allowing for a comparison between the interpretability of penalized regression and complex ensemble methods.

Key Results

• Predictive Performance: All five models demonstrated comparable and strong predictive performance. ROC-AUC values exceeded 0.86 for all models, and classification accuracies were above 81%. Notably, the interpretable penalized regression models (Ridge, LASSO, Elastic Net) performed nearly identically to the more complex Random Forest model, suggesting that increased model complexity did not yield substantial gains in accuracy for this dataset.
• Variable Selection Stability: Bootstrap stability selection identified six predictors as highly robust (selection frequency = 1.00): sleep hours, study hours, social media use, attendance, examination pressure, and family support. These variables were consistently selected across all 500 bootstrap samples.
• Feature Importance Agreement: SHAP analysis of the Random Forest model identified examination pressure as the most influential predictor, followed by study hours, sleep hours, and family support. There was strong agreement between the SHAP importance rankings and the stability selection frequencies, validating the robustness of these predictors across different methodological approaches.

Key Contributions
The paper contributes a reproducible workflow that integrates three complementary statistical learning techniques:

1. Penalized Regression: Provides parsimonious models through coefficient regularization.
2. Stability Selection: Evaluates the reproducibility of variable selection, distinguishing robust predictors from sample-specific artifacts.
3. SHAP Explainability: Enhances transparency by explaining predictor contributions in nonlinear models.

The study demonstrates that these methods should not be viewed as competing but as components of a single framework that enhances reliability and interpretability without sacrificing predictive accuracy.

Significance and Claims
The authors claim that their framework offers a comprehensive approach for developing predictive models that are accurate, reproducible, and interpretable. The study highlights that:

• Interpretable statistical models can achieve performance comparable to complex machine learning algorithms for binary classification problems involving correlated predictors.
• Integrating stability selection with explainable AI (SHAP) provides convergent evidence regarding feature importance, increasing confidence in scientific inference.
• While demonstrated on student stress prediction, the framework is readily applicable to a wide range of binary classification problems in fields such as medicine, public health, and finance where reliable feature identification and model transparency are critical.

The paper maintains a modest tone regarding its scope, acknowledging limitations such as the use of a simulated dataset (though designed to reflect realistic patterns), the focus on a single machine learning benchmark (Random Forest), and the lack of external validation from independent institutions. It suggests that future work could extend the framework to include additional algorithms and external validation datasets.