A Cost-Sensitive and Explainable Evaluation Framework for Chronic Kidney Disease Prediction Models

Technical Summary: A Cost-Sensitive and Explainable Evaluation Framework for Chronic Kidney Disease Prediction Models

Problem Statement
Chronic Kidney Disease (CKD) affects approximately 14% of adults globally, often progressing asymptomatically until advanced stages where intervention is difficult. While machine learning (ML) offers potential for early detection, a significant barrier to clinical deployment is the "black box" nature of high-performance ensemble models, which lack the interpretability of traditional statistical models. Furthermore, standard evaluation metrics (e.g., accuracy) often fail to account for the asymmetric clinical consequences of misclassification: a false negative (missed diagnosis) poses a far greater risk to patient outcomes than a false positive (unnecessary screening). Existing literature frequently prioritizes predictive accuracy over the joint evaluation of interpretability, feature contribution, and cost-sensitive trade-offs, limiting the practical utility of these models in clinical decision support systems.

Methodology
The study evaluates three supervised learning models—Logistic Regression (LR), Random Forest (RF), and Extreme Gradient Boosting (XGBoost)—using a synthetic CKD dataset containing 1,659 patient records (1,524 positive cases, 135 negative cases). The methodology employs a multi-faceted evaluation framework:

1. Data Preprocessing: Missing numeric values were imputed using the median, and categorical variables were converted to dummy variables. Highly correlated variables (e.g., specific cholesterol subtypes) were grouped or removed to address multicollinearity, particularly for the LR model. The data was split into 80% training and 20% testing sets using stratified sampling.
2. Performance Evaluation: Models were assessed using standard metrics (accuracy, precision, recall, F1-score, AUC) and five-fold cross-validation. Crucially, the study moves beyond the default classification threshold of 0.5 to perform threshold optimization.
3. Cost-Sensitive Analysis: A cost framework was applied where false negatives were assigned a penalty of $9,000 and false positives $5,000. This reflects the clinical reality that missed diagnoses lead to severe disease progression, while false positives result in additional, less harmful screening. Thresholds were varied to identify the point minimizing total cost.
4. Interpretability:
   • Logistic Regression: Analyzed via coefficient analysis and permutation importance.
   • Ensemble Models (RF, XGBoost): Evaluated using permutation importance and SHapley Additive exPlanations (SHAP) to provide both global and local explanations of feature contributions.

Key Results

• Predictive Performance: At the default threshold of 0.5, all models achieved approximately 92% accuracy. However, due to severe class imbalance, accuracy was deemed less informative than recall, precision, and F1-scores, which remained high (F1 ~96%) across all models.
• Cost-Sensitive Optimization: Threshold optimization significantly altered model outcomes.
  • False Negatives: At their respective optimal thresholds, both Random Forest and XGBoost eliminated false negative predictions entirely. Logistic Regression, while producing the fewest false positives, was the only model to retain false negatives.
  • Total Cost: Despite having fewer false positives, Logistic Regression incurred the highest overall cost due to the presence of false negatives and the higher penalty assigned to them. XGBoost and Random Forest achieved lower total costs by eliminating false negatives, even with a slight increase in false positives.
• Stability: XGBoost demonstrated the highest Area Under the Curve (AUC) and the strongest cross-validation stability, suggesting superior generalization capabilities compared to LR and RF.
• Feature Importance: All models consistently identified clinically plausible predictors.
  • Global Consensus: Glomerular Filtration Rate (GFR) and serum creatinine were identified as the most significant contributors across all models.
  • Model Specifics: XGBoost additionally highlighted urine protein, blood pressure (systolic/diastolic), and fasting blood sugar. SHAP analysis confirmed that increases in creatinine, BUN, and blood pressure, alongside decreases in GFR, drove higher risk predictions.
  • Clinical Alignment: The features identified by the "black box" models (RF and XGBoost) aligned with established clinical risk factors (e.g., diabetes and hypertension), validating their clinical plausibility.

Significance and Claims
The paper claims that extending model evaluation beyond traditional predictive accuracy to include threshold optimization, cost-sensitive analysis, and explainability techniques provides a more clinically relevant framework for assessing ML deployment suitability.

• Clinical Suitability: The study demonstrates that ensemble models, often dismissed as "black boxes," can be made clinically trustworthy when paired with explainable AI (SHAP) and cost-sensitive thresholding. Specifically, the ability of XGBoost and RF to eliminate false negatives at optimized thresholds makes them operationally superior for early disease detection compared to the more interpretable but costlier Logistic Regression in this specific context.
• Trust Building: By demonstrating that the most influential predictors in complex models align with established medical research (e.g., the critical role of GFR and creatinine), the framework helps bridge the gap between high-performance ML and clinician trust.
• Methodological Contribution: The research highlights that default thresholds are often suboptimal for healthcare applications. It argues for a shift toward evaluation frameworks that explicitly weigh the asymmetric costs of misclassification to guide model selection and deployment.

Limitations
The authors acknowledge that the study relies on a synthetic dataset generated for educational purposes, which limits the generalizability of findings to real-world clinical records. Additionally, the cost values ($9,000 vs. $5,000) are assumptions designed to stimulate clinical trade-offs rather than reflect actual healthcare economics. The lack of external validation due to the scarcity of comparable clinical datasets with GFR variables is also noted as a constraint.