Comparative Evaluation of Machine Learning and Deep Learning Models for Early Prediction of Severe Acute Pancreatitis: A Multi-Model Study Using the 2012 Revised Atlanta Classification

Technical Summary: Comparative Evaluation of ML and DL Models for Early SAP Prediction

Problem Statement
Acute pancreatitis (AP) is a frequent gastrointestinal emergency where approximately 10–20% of patients progress to severe acute pancreatitis (SAP), a condition associated with 20–40% mortality and significant intensive care resource utilization. Timely identification of SAP is critical for early escalation of care. However, established clinical severity scoring systems (e.g., Ranson, APACHE II, BISAP, MCTSI) rely on data accumulated over 24–48 hours or require contrast-enhanced CT imaging, limiting their utility for immediate triage at hospital admission. There is an unmet need for prediction tools capable of stratifying severity using routine admission data available at the time of presentation.

Methodology
This study conducted a retrospective, multi-model evaluation on a Chinese cohort of 722 AP patients (585 severe, 137 mild/moderate), labeled according to the 2012 Revised Atlanta Classification. The input features consisted of 13 routine admission variables, including laboratory values (WBC, CRP, creatinine, BUN, glucose, calcium, LDH, albumin, hematocrit) and clinical measurements (age, BMI, heart rate, systolic blood pressure).

The authors evaluated 11 distinct models across three architectural families using 5-fold stratified cross-validation to address class imbalance (81% SAP prevalence):

1. Classical Machine Learning: Logistic Regression (L2 regularized), Random Forest (500 estimators, balanced weights), and Gradient Boosting (200 estimators).
2. Feedforward Deep Learning: A standard 3-layer MLP, a Residual MLP, and an Attention MLP.
3. Recurrent Deep Learning: Standard LSTM, Stacked LSTM, Bidirectional LSTM, LSTM with additive attention, and a CNN-LSTM hybrid.

Deep learning models were adapted for cross-sectional tabular data by reshaping feature vectors into sequences of length one. Performance was assessed via AUC-ROC, F1 score, sensitivity, specificity, and PPV, with decision thresholds optimized to maximize the F1 score on training folds.

Key Results

• Superiority of Classical ML: Classical ensemble methods outperformed all deep learning architectures. Random Forest (RF) achieved the highest AUC of 0.877 (F1 = 0.917, Sensitivity = 96.8%, PPV = 87.1%). Gradient Boosting followed closely with an AUC of 0.874.
• Deep Learning Performance: Among deep learning models, the standard MLP performed best (AUC = 0.836), outperforming the Residual and Attention variants. Recurrent models (LSTM family) demonstrated the lowest discriminative power, with AUCs ranging from 0.684 to 0.777.
• Recurrent Model Failure: The CNN-LSTM was the best-performing recurrent model (AUC = 0.777) but remained inferior to all classical approaches. Most LSTM variants achieved 100% sensitivity but at the cost of "threshold collapse" (decision thresholds near zero), effectively predicting the majority class (SAP) for all inputs, rendering them clinically non-discriminative.
• Sensitivity vs. Specificity: While RF achieved high sensitivity (96.8%), it maintained a PPV of 87.1%. In contrast, deep learning models often sacrificed specificity entirely to achieve maximal sensitivity, resulting in uncomputable or negligible PPV in meaningful clinical contexts.

Key Contributions

1. Systematic Comparison: The study provides a rigorous, head-to-head comparison of 11 models spanning classical ML, feedforward, and recurrent deep learning architectures for early SAP prediction.
2. Architectural Insight: It demonstrates that for cross-sectional, single-encounter tabular medical data, recurrent architectures (LSTMs) offer no advantage and often underperform compared to classical ensemble methods. The study highlights that the inductive bias of tree ensembles is better suited for independent feature vectors than sequential models forced onto non-sequential data.
3. Open-Source Implementation: The authors released the full analysis pipeline, preprocessing code, and trained model artifacts to facilitate reproducibility and future research.

Significance and Claims
The paper claims that Random Forest provides a robust, high-sensitivity tool for the early prediction of SAP severity using only routine admission data, potentially enabling earlier triage than current 24–48 hour scoring systems. The findings support the continued clinical utility of classical machine learning on structured medical data, challenging the assumption that deep learning is universally superior.

The authors explicitly state that the limited added value of recurrent architectures in this setting is likely due to the mismatch between the sequential nature of LSTMs and the cross-sectional nature of the dataset. They conclude that while the models show promise, external prospective validation is required before clinical deployment. The study acknowledges limitations including the single-cohort design (Chinese tertiary center), potential referral bias, and the lack of direct comparison with established clinical scores (BISAP, APACHE II) on the same dataset. The models are intended strictly for research purposes and are not validated for guiding patient management.