Calibration, Uncertainty Communication, and Deployment Readiness in CKD Risk Prediction: A Framework Evaluation Study

Technical Summary: Calibration, Uncertainty Communication, and Deployment Readiness in CKD Risk Prediction

Problem Statement
While machine learning models for chronic kidney disease (CKD) risk prediction frequently demonstrate strong discrimination (e.g., high AUROC) on internal test sets, they often lack rigorous assessment of calibration and uncertainty quantification. This gap leaves clinicians without reliable information regarding the trustworthiness of probability outputs. Furthermore, no prior study has jointly evaluated calibration quality, uncertainty coverage via conformal prediction, and structured deployment readiness across a common model suite using external clinical validation. The field struggles not with building models, but with validating their probability outputs and generalizability under distributional shifts.

Methodology
The study evaluated five classifiers—Logistic Regression (LR), Random Forest (RF), XGBoost, Support Vector Machine (SVM) with Platt scaling, and Gaussian Naive Bayes (NB)—trained on the UCI CKD dataset (400 patients, 62.5% CKD prevalence). The evaluation framework consisted of three dimensions:

1. Calibration Assessment: Models were evaluated on the UCI test set before and after post-hoc recalibration using Platt scaling and isotonic regression. Metrics included Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and Brier Score.
2. Uncertainty Quantification: Split conformal prediction was applied using the MAPIE library to generate prediction sets with a formal 90% marginal coverage guarantee. Metrics included empirical coverage rate, average prediction set size, and singleton rate.
3. Distributional Stress-Testing: The best-calibrated variant of each model was applied to the MIMIC-IV demo cohort (97 patients, 23.7% CKD prevalence). This cohort served as a stress-test to evaluate model behavior under significant prevalence shift and feature missingness (seven features were imputed using UCI training statistics).
4. Deployment Readiness Framework: An eight-criterion checklist was developed to score models on discrimination, calibration adequacy, calibration stability, uncertainty coverage, coverage stability, prediction interpretability, subgroup equity, and transparency. Scores were binary (Pass/Fail/Marginal) with a maximum of 16 points.

Key Results

• Internal Performance: All five models achieved perfect discrimination (AUROC 1.00) on the internal UCI test set. Post-isotonic recalibration reduced internal ECE to near-zero levels (0.000–0.022).
• External Stress-Test Failure: Upon application to the MIMIC-IV demo cohort, performance deteriorated sharply across all models:
  • Discrimination: AUROC dropped to chance levels (0.48–0.58).
  • Calibration: ECE rose dramatically to 0.68–0.76, with calibration drift exceeding 0.67 for all models.
  • Uncertainty Coverage: Conformal coverage collapsed from 0.80–0.98 (internal) to 0.21–0.25 (external), far below the 90% target.
• Deployment Readiness: No model passed the deployment checklist. Total scores ranged from 2/16 (XGBoost) to 4/16 (LR, RF, SVM, NB). The only criteria met were transparency and prediction interpretability on the internal set; all criteria referencing the external stress-test (discrimination, calibration, stability, coverage, equity) failed.

Significance and Claims
The paper claims that near-perfect internal performance metrics do not guarantee reliability under distributional shift. The study demonstrates that calibration stability and conformal coverage transfer must be evaluated before any clinical ML model moves toward deployment.

Key contributions include:

• Operationalizing a Joint Evaluation: This is the first study to jointly assess calibration, conformal coverage, and a structured deployment checklist on the same model suite across an independent external cohort.
• Conformal Prediction as a Diagnostic: The collapse of conformal coverage on the external dataset serves as a direct, interpretable signal of distributional shift, indicating that the test population is not exchangeable with the training population.
• A Structured Framework: The eight-criterion deployment readiness framework provides a structured path for evaluating binary clinical prediction models, highlighting that internal metrics alone are insufficient for determining clinical utility.

The authors explicitly note that the MIMIC-IV demo cohort was used as a stress-test due to its open-access nature and specific limitations (small sample size, imputed features), rather than as a formal external validation. Consequently, the results illustrate the mechanics of distributional failure rather than estimating specific clinical deployment performance, but the conclusion remains that internal calibration quality does not ensure reliability under prevalence shift and feature missingness.