An Open, Reproducible Implementation of the CMS Hospital-Wide Readmission Measure on MIMIC-IV

Technical Summary: An Open, Reproducible Implementation of the CMS Hospital-Wide Readmission Measure on MIMIC-IV

Problem Statement
Thirty-day hospital readmission is a primary target for clinical machine learning, yet the vast majority of studies utilizing the MIMIC-IV dataset rely on a "naive" label: any readmission within 30 days. This definition fails to align with the actual metric used by the Centers for Medicare & Medicaid Services (CMS) to evaluate hospitals: the Hospital-Wide Readmission (HWR) measure. The naive label ignores critical policy distinctions, specifically the exclusion of planned readmissions (e.g., scheduled chemotherapy or staged procedures), the stratification of patients into specific specialty cohorts, and the removal of ineligible admissions (e.g., in-hospital deaths or discharges against medical advice). Consequently, models trained on naive labels answer a different, less policy-relevant question than the one hospitals are measured on.

Methodology
The study presents an end-to-end, open-source implementation of the CMS HWR measure on MIMIC-IV v3.1 (hosp and icu modules) and MIMIC-IV-Note v2.2. The methodology focuses on constructing a "faithful" cohort and evaluating a baseline model against this rigorous definition.

• Cohort Construction: Starting from 546,028 hospitalizations, the authors applied a waterfall of exclusions to reach a final cohort of 192,879 index admissions (99,285 unique patients). Exclusions included pre-ICD-10 admissions, cancer treatment, psychiatric care, rehabilitation, maternity, in-hospital death, hospice, discharge against medical advice, and transfers. The remaining admissions were assigned to one of five mutually exclusive specialty cohorts (medicine, surgery/gynecology, cardiorespiratory, cardiovascular, neurology) based on discharge service and procedures.
• Label Definition (Faithful vs. Naive): The "faithful" label strictly adheres to the CMS Planned Readmission Algorithm v4.0. An admission is only counted as a readmission event if it is unplanned. Planned readmissions are identified via a specific list of procedures and diagnoses, unless overridden by an acute, disqualifying diagnosis.
• Feature Engineering: Features were derived strictly from data available at or before discharge (point-in-time correctness) to prevent data leakage. The feature set includes 252 columns across eight groups: administrative/demographic data, laboratory panels, outpatient vitals, ICU vitals, medications, procedures, specialty cohorts, and comorbidities (encoded via a 31-flag Quan-Elixuser encoder).
• Modeling: A HistGradientBoostingClassifier (scikit-learn) served as the baseline. Probabilities were calibrated using isotonic regression. Evaluation utilized 5-fold GroupKFold (grouped by subject_id) to prevent patient-level data leakage.
• GenAI Ablation: To test the value of unstructured text, discharge summaries were embedded using Amazon Titan Text Embeddings v2, reduced via PCA, and merged with the structured baseline.

Key Results

• Label Shift: On the identical set of 192,879 index admissions, the 30-day readmission rate dropped from 17.6% (naive definition) to 15.9% (faithful HWR definition). This 1.7 percentage-point shift fundamentally alters class balance and calibration targets.
• Model Performance: The calibrated baseline achieved a ROC-AUC of 0.7305 and a PR-AUC of 0.3726. The model was well-calibrated (Brier score 0.1186) and demonstrated positive net benefit in decision-curve analysis.
• Feature Importance: SHAP analysis revealed that the model's predictive power is driven primarily by prior utilization (cumulative prior length of stay, days since last discharge, prior admission count) and acuity indicators. Demographic attributes (sex, marital status, insurance) also ranked highly.
• GenAI Findings: Adding discharge-summary embeddings to the structured baseline resulted in no meaningful uplift. The ROC-AUC increased only marginally (from 0.7304 to 0.7312 on the full cohort), a difference within fold-to-fold noise. This indicates that for this specific target, structured prior-utilization signals dominate, and discharge note embeddings are redundant.
• Fairness and Drift: Discrimination remained consistent across race, insurance, and specialty subgroups. However, temporal drift analysis (Population Stability Index) flagged significant shifts in 11 features between the 2020–2022 window and earlier years, attributed to COVID-era disruptions.

Significance and Claims
The paper asserts that its primary contribution is not a novel classifier, but a reusable, reproducible reference implementation of a clinically meaningful label. By releasing the cohort logic, the CMS Planned Readmission Algorithm v4.0 code lists, the comorbidity encoder, and the feature specification, the authors aim to lower the barrier for researchers to adopt the "faithful" label over the "naive" one.

The work highlights a recurring issue of construct validity in medical ML: optimizing for a label that does not match the policy reality. The authors claim that the "negative" GenAI result is itself informative, serving as a counterweight to the assumption that large language model embeddings automatically improve clinical baselines. The study concludes that for all-condition readmission, the signal is heavily concentrated in prior utilization and acuity, and that the cost of implementing the correct, policy-aligned label is now comparable to the cost of using the naive label.

Limitations
The authors note the study is single-center (BIDMC via MIMIC-IV) and retrospective, precluding external validity claims. The implementation is faithful to the CMS logic but is not CMS-certified or risk-standardized for hospital comparison. Additionally, the restriction to the ICD-10 era reduces the sample size, and discharge summary coverage is limited to approximately 50% of admissions.